Compare commits

..

54 Commits

Author SHA1 Message Date
Patrick Buckley 273d547f4e chore: bump version to 1.5.7 2026-05-04 03:04:09 -07:00
Patrick Buckley bbb404c363 feat(console): inline node picker replaces back-to-console banner (#475)
* feat(console): inline node picker replaces back-to-console banner

Drops the 32px banner the console proxy used to inject above proxied
server-UI pages and replaces it with an inline node-id pill in the
existing #ui-header.  Click the pill to open a dropdown that lists
healthy nodes (health dot, ws count, reachable/degraded/unreachable
text) plus a top-row link back to the console.

Reuses the .ws-tab-dropdown shell from ui/static/style.css for
animation, shadow, theme override, and item layout, so the picker
visually matches the workstream-tab chevron menu it sits next to.
Keyboard nav (ArrowDown/Up/Home/End/Tab/Escape) mirrors the chevron
menu's handler with cross-reference comments at both sites.
Lazy-fetches /v1/api/cluster/nodes against the console origin
(bypassing the prefix shim) on first open.

Reclaims 32px of vertical space, consolidates three separate
"you're on node X via console" indicators into one, and turns the
wayfinding chrome into a real cluster-nav primitive.

* fix(console): address Copilot review on node picker

- Request /v1/api/cluster/nodes?limit=1000 (collector's hard cap)
  instead of relying on the default 100 — clusters with more than
  100 nodes were silently dropping rows from the picker.
- Hand off focus to the first menu item after the async fetch
  resolves: openMenu()'s deferred focus hook ran while only the
  skeleton was in the DOM, so first-open keyboard users were
  stranded on the trigger until they pressed an arrow key.
- Tab now closes the menu without preventDefault, so focus moves
  to the next focusable element on the first press (ARIA APG menu
  pattern).  Escape still preventDefault + returns to the pill.
- Cap pill max-width at 240px and ellipsize the id span; node ids
  are accepted up to 256 chars upstream and could otherwise push
  the title and right-side controls off the appbar.  Pill carries
  a title attribute so the full id is still legible on hover.
2026-05-04 02:59:35 -07:00
Patrick Buckley 072113f7ca fix(session): properly inject queued user messages mid-loop (#474)
* fix(session): properly inject queued user messages mid-loop

Two queued-user-message bugs in ``ChatSession.send()``.

**Mid-tool-call: ``Unexpected role 'tool' after role 'user'`` on Mistral.**
The ``supports_tool_advisories`` capability flag (default False for
unknown openai-compatible models) routed cap-off providers down a
short-circuit branch in ``_collect_advisories`` that called
``_flush_queued_messages`` directly. That appended a ``user`` turn
between ``assistant(tool_calls)`` and ``tool``, which mistral-common's
``_validate_message_order`` rejects with a 400.

Drop the flag. All providers now run the unified path: queued user
messages become ``UserInterjection`` advisories that ride inside the
tool result envelope via ``wrap_tool_result``, splicing
``<system-reminder>`` text into the tool message's content. Role
sequence stays ``assistant → tool``. Live-confirmed on Mistral
medium and Qwen3 — both correctly distinguish system-reminder from
tool stdout in their reasoning.

**Mid-stream: queued message orphaned until next user send.**
After a no-tool assistant turn, ``_flush_queued_messages`` would
append the queued user message to history and the loop would
``break``, leaving the message at the tail of history with no
model response. Visible as "two sends to get one reply".

``_flush_queued_messages`` now returns ``bool``. The no-tool branch
``continue``s on drain instead of ``break``ing, so the model gets a
turn over the extended history.

Tests:
- ``test_collect_advisories_drains_text_queued_messages_to_persistent``
  pins the unified-path drain (text-only queue → ``UserInterjection``,
  no separate user turn appended to ``self.messages``).
- ``test_send_continues_when_messages_queued_during_streaming`` pins
  the loop-continue behavior (fails with 1 stream call pre-fix,
  passes with 2 post-fix).

* fix(session,ui): reject queued attachments + paperclip busy state

Copilot pointed out that the attachment-bearing branch in
``_collect_advisories`` had the same role-ordering bug as the
text-only path that 802658f fixed: an attachment-bearing queued
item would still call ``_append_user_turn`` mid-tool-call,
injecting ``user`` between ``assistant(tool_calls)`` and ``tool``.

Pragmatic fix: don't allow attachments to be queued at all.

**Backend.** ``ChatSession.queue_message`` raises a new
``AttachmentsNotQueueableError`` when called with non-empty
``attachment_ids``. The interactive ``/send`` route catches it,
releases reservations via the existing ``_release_reservation_on_fail``
hook, and surfaces ``status: "attachments_busy"`` to the caller
with the IDs in ``dropped_attachment_ids``. The coord adapter
mirrors the cleanup (releases the soft-locked reservation taken
for ``_send_id``) so the create-with-attachments path can't leak.

Now that the queue can never carry attachments, the per-item
``att_ids`` slot is gone:

- Queue tuple slimmed ``(cleaned, priority, att_ids)`` →
  ``(cleaned, priority)``.
- ``_flush_queued_messages`` collapses to a single combined-text
  user turn (no attachment branch).
- ``_collect_advisories`` queue-drain pushes ``UserInterjection``
  advisories only (no ``attachment_items`` list).
- ``dequeue_message`` no longer unreserves (queue can't reserve).
- ``_resolve_attachment_ids`` had no remaining production callers
  and is deleted along with the tests that exercised it in
  isolation.

**Frontend.** ``Composer.setBusy`` disables the paperclip whenever
busy (regardless of ``queueWhileBusy``) — text still queues,
attachments don't. ``chat.css`` gains a ``.composer-attach:disabled``
rule (mirrors the existing ``.composer-send:disabled`` treatment)
so the affordance actually looks unclickable instead of falling
through to the UA default. ``title`` and ``aria-label`` are kept in
sync for AT users (WCAG 4.1.2).

Both interactive and coordinator UIs handle the new
``attachments_busy`` response with a chat-surface error bubble:

> Attachments can't be sent while the assistant is working.
> Send a text-only message now, or wait and resend with attachments.

Chips stay in the composer so the user can retry once idle.

**Tests.** Replaced the now-impossible ``TestQueuedWithAttachments``
class with a rejection-coverage class. Rewrote the
``_queue_with_attachment`` route-test fixture to reserve directly
via ``reserve_attachments`` (the queue path no longer reaches the
reserved state). Added a route-level test for the new
``attachments_busy`` contract.
2026-05-04 02:59:35 -07:00
Patrick Buckley 7f1b0acf7a Bound search tool output against pathological inputs (#473)
* Bound search tool output against pathological inputs

Replaces the per-line truncation with a fully bounded pipeline so the
search tool can no longer overflow the LLM context — or OOM the parent —
on minified bundles, multi-GB JSONL records, or huge result sets.

Backend:
- Prefer ripgrep when on PATH; grep is the fallback. Detection is
  cached via functools.cache.
- ripgrep flags do most of the bounding natively: --max-columns 1024
  + --max-columns-preview, --max-filesize 10M, --max-count 100,
  --no-config, --no-messages, plus negative globs for the same
  noisy directories grep has been excluding.
- ripgrep added to the Dockerfile.

Streaming subprocess (_search_capture):
- subprocess.Popen with a streaming, byte-capped stdout read (4 MB).
  Defends against single-line files (training data, minified bundles)
  that would have OOM'd the previous subprocess.run capture.
- threading.Timer watchdog enforces tool_timeout even when the
  pipe read is blocked in the kernel — proc.wait(timeout=…) alone
  was insufficient because the read sat ahead of it.
- Stderr drained in a daemon thread to avoid pipe-deadlock when the
  child writes to stderr while we're still reading stdout. Cap on
  captured stderr keeps a hostile child from growing the buffer.

Tier-based formatter (_format_search_results):
- Tier 1: full path:line:content output, stream-emitted with a
  running-cost short-circuit so we never materialize past the budget.
- Tier 2: K samples per file with overflow notes; K is computed
  analytically from budget / file_count / avg-line-length so we hit
  the right ladder rung in a single pass.
- Tier 3: per-file counts only, also budget-bounded with a tail line
  reporting the omitted files. Sorted by descending count.
- Total output budget (32 KB) is well under tool_truncation, so the
  head+tail _truncate_output strategy never silently drops middle
  files in a search result.

Argument injection fix:
- The ripgrep arg list was missing the `--` separator that the grep
  branch already had. With auto_approve on the search tool, that was
  exploitable: path='--pre=COMMAND' would have made ripgrep run the
  script as a per-file preprocessor and surface its stdout. Added
  `--` and a regression test.

State-machine cleanup in _exec_search:
- rc < 0 (signal-killed by something other than us) now surfaces a
  dedicated 'killed by signal N' message instead of being parsed as
  success.
- capped + zero parsed records (e.g. one multi-MB line with no \n)
  now returns a dedicated byte-cap message instead of the malformed-
  output message that previously masked the real cause.
- _report_tool_result descriptions now match the returned payload
  (no more 'no matches' tag on a 'malformed' payload).

Defence-in-depth on env scrub:
- RIPGREP_CONFIG_PATH, GIT_CONFIG, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM
  added to _EXPLICIT_SCRUB. We pass --no-config on the rg CLI today,
  but if a future caller forgets the flag, an attacker who can set
  one of these env vars could plant a config containing --pre=… and
  recreate the same RCE shape.

Tests:
- TestSearchLineTruncation rewritten to mock _search_capture instead
  of subprocess.run (the previous tests passed ChatSession kwargs
  that no longer satisfy the constructor).
- TestSearchBackendSelection covers rg/grep detection and arg
  construction, including the --pre flag-injection regression.
- TestSearchOutputBudget exercises Tier 1/2/3 directly.
- TestSearchCaptureStreaming spawns real Python subprocess writers
  to exercise the byte-cap trim, mega-line-no-newline edge case, the
  watchdog timeout when the child writes nothing, and the stderr
  drain under load.
- test_env_scrub picks up the new tool-config keys.

* Address Copilot review on #473

- Budget the Tier 2/3 header up front so the formatter's emission stays
  strictly within _SEARCH_OUTPUT_BUDGET. Previously the fit checks only
  counted body bytes, letting the final string overflow by ~120 chars
  (header + separator) and triggering _truncate_output's head+tail
  dropout — exactly the shape this code was trying to avoid.
- Restore the (5, 3, 1) ladder in Tier 2: the analytical K from perf-2
  is kept as a starting estimate, but if that K's actual emission
  doesn't fit (the estimate ignores the header and overweights shared-
  path compression) we step down through the ladder before falling
  through to Tier 3. The previous one-shot K could collapse to counts-
  only when 3/file or 1/file would have fit.
- Only normalise rc to 0 in the capped-output path when rc < 0 (our
  SIGKILL). There's a narrow race where the child can exit naturally
  between our read and our kill; preserving a non-negative rc means
  rg's rc=2 ('matches found but some files had errors') no longer
  silently turns into a clean success when the byte cap also fires.
- Clarify _MAX_SEARCH_LINE_LENGTH doc: the cap applies to the content
  portion (after path:lineno:), not the whole emitted line.
- Add explanatory comments on the two intentional `except Exception:
  pass` blocks in _search_capture (stderr drain, pipe close in the
  cleanup finally) so static analysis and future readers can see the
  silence is deliberate.
- Tighten the budget tests: now assert strict `<= _SEARCH_OUTPUT_BUDGET`
  instead of the +512-char slack that was masking the header overflow.
- New regression tests:
  - Tier 2 ladder step-down (K=5 over budget, K=3 fits, no Tier 3 fall-through)
  - capped + rc=2 surfaces stderr instead of being normalised to success
  - capped + rc<0 (our SIGKILL) flows through as a partial-result success

* chore(search): post-review cleanup

Follow-up to the Copilot-review fixes in 39d2aa2 — these are all small
quality items (no behaviour change, no new tests).

- q-1: collapse the Tier 2 candidates filter to a single expression.
  Drops the redundant inner ``max(estimated_k, 1)`` and the unreachable
  ``if not candidates`` branch (the ladder ends in 1 and ``estimated_k``
  is already floored at 1, so the comprehension always yields ≥ ``[1]``).
  ``or [...]`` is kept as defence against future ladder changes.
- q-2: update _format_search_results docstring to match the new ladder
  semantics (analytical seed → step down through (5, 3, 1) from the
  highest rung ≤ the estimate). The previous wording suggested every
  Tier 2 attempt started at 5.
- q-3: combine the two ``from turnstone.core.session import ...``
  statements in test_tier2_steps_down_ladder_before_falling_to_tier3
  into a single top-of-function import (matches the surrounding tests).
- q-4: shorten the explanatory comments on the two best-effort cleanup
  paths in _search_capture to one line each. Both sites now read with
  the same shape ("# best-effort: pipe may be torn down by ...").
- q-5: trim the _MAX_SEARCH_LINE_LENGTH comment from 7 lines back to 3.
  Keeps the load-bearing semantic (cap is on the content portion only)
  and the pathological-line defence; drops the paths-aren't-bounded
  parenthetical, which was background reading rather than WHY.
2026-05-04 02:59:35 -07:00
renovate[bot] 6904bd8f39 chore(deps): lock file maintenance 2026-05-04 02:59:35 -07:00
renovate[bot] 4d677d1ebf chore(deps): update github actions 2026-05-04 02:59:35 -07:00
Patrick Buckley 35f462a46d chore: bump version to 1.5.6 2026-05-03 13:44:50 -07:00
Patrick Buckley ec74334e74 feat(providers): api_surface toggle + mistral medium reasoning fix (#469)
* feat(providers): api_surface toggle + mistral medium reasoning fix

Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions.  The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).

Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
  and thread it through `create_provider` / `model_registry.get_provider`.
  `openai-compatible` defaults to Chat Completions; operators can flip
  individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
  on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
  `chat_template_kwargs`.  Operators running gpt-oss-style local
  templates that consume `reasoning_effort` from the chat template now
  opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
  server-side at create/update time; pre-filled by Detect via the
  profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
  api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
  for capability and server_compat resolution; previously the fallback
  passed `alias=None`, which silently dropped per-model caps on the
  agent path.

Tests: 5117 passed (-m "not live"); ruff + mypy clean.

* fix(providers): don't auto-suggest Responses for Mistral medium

vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls.  Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.

Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile.  Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.

* fix(providers): address Copilot review on PR #469

- providers/__init__.py: drop the redundant *_responses_provider /
  *_chat_provider names; have create_provider use _openai_provider and
  _openai_compat_provider directly so they're not flagged as unused
  globals.
- console/server.py: tighten _validate_api_surface to a strict equality
  match against the canonical {"chat", "responses"} set.  The previous
  strip().lower() membership check accepted ' Responses '/'CHAT' but
  stored the raw string verbatim, which then failed to round-trip
  through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
  type, api_surface, extra_body) on provider == "openai-compatible" at
  save time so toggling provider away can't leave a stale hidden surface
  selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
  longer flags the call as a wrong-name keyword (the point of the test
  is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
  for the api_surface validation on both create and update — covers the
  bogus-value rejection, non-canonical-string rejection, and the happy
  path persisting through to the refreshed registry.
2026-05-03 13:40:30 -07:00
Patrick Buckley 2fd0c29a92 fix(memory): query-aware candidate selection + OR-of-terms search (#468)
* fix(memory): query-aware candidate selection + OR-of-terms search

The system-message memory composition path used a recency-ordered
candidate set (`_list_visible_memories(limit=fetch_limit)`).  On
deployments with more than `fetch_limit` (default 50) visible
memories, BM25 only ever ranked the 50 most-recently-touched memories
— a relevant memory written months ago was silently invisible
regardless of how well it matched the recent context.  Multi-word
search at the SQL layer used AND-of-terms, killing recall on any
multi-word query without an exact field overlap.

## Functional changes

- `_init_system_messages` (`turnstone/core/session.py`): extract
  recent context first, then `_search_visible_memories(context)` to
  pull query-aware candidates.  Search hits below `fetch_limit` union
  with the recency list (deduped by memory_id) so the BM25 candidate
  pool is always a SUPERSET of the prior recency-only pool — even on
  noisy queries where the cap fills with stopwords, the recency-50
  the original bug surfaced still reaches BM25.  Empty context skips
  search entirely.  Candidate-selection logic extracted into
  `_select_memory_candidates`.

- `search_structured_memories` (PostgreSQL + SQLite): per-term
  clauses join with OR instead of AND.  A row matches if ANY term
  matches ANY of name/description/content.  Downstream BM25 narrows
  back down by relevance.

## Perf hardening

- Collapse the 1-3 fanned scope queries into a single SQL.  New
  backend methods `list_visible_structured_memories` /
  `search_visible_structured_memories` union the visibility scopes
  into one WHERE OR-group, so a composition rebuild now hits the DB
  at most twice (search + recency) instead of up to six times.

- Cap and normalize search terms.  Composition can hand a multi-KB
  pasted message to ILIKE-based search; without a cap, every distinct
  token would emit one unindexable predicate per scope-fanned query.
  `normalize_search_terms` (`storage/_utils.py`) de-dupes
  case-insensitively, drops <2-char tokens, and hard-caps at 16.

- Per-turn search cache.  `_init_system_messages` fires from many
  call sites within one turn (state transitions, MCP refresh, tool
  results) and the recent-context query is identical across them.
  Session-instance cache keyed by (query, mem_type, limit) absorbs
  the duplicates; invalidated in `_append_user_turn` and after
  memory save/delete tool actions.

- Stable secondary sort by `memory_id`.  `updated` is second-precision
  and `touch_structured_memories` can land a batch on identical
  timestamps; without a tie-breaker SQL returns rows in
  implementation-defined order, BM25 input shuffles, and the
  LLM-side prompt cache misses across calls.  All four backend ORDER
  BYs now break ties on `memory_id ASC`.

## Quality cleanups

- Coalesce `memory.search.term_count` + `memory.search.zero_results`
  into a single `memory.search` log carrying both `term_count` and
  `result_count`.
- New `memory.composition` log: source / candidates / injected.
- Promote a shared `make_chat_session` factory to `tests/_helpers.py`.
- Rename SQL builder local `extra` -> `scope_filters` for clarity.
- Add docstrings on `search_structured_memories` so the AND->OR flip
  survives future readers.

## Tests

Adds 20 tests across `tests/test_structured_memory.py`,
`tests/test_structured_memory_storage.py`, and
`tests/test_memory_relevance.py`: recency-ceiling regression,
empty-query fallback, sparse-match union, recency-preserved-when-
search-returns-noise (locks in the pool-superset invariant),
OR-of-terms on both backends, scope filtering preserved,
search-facade multi-word behavior, term-cap normalization, the new
visible-scope helpers (list + search + empty-scopes guard),
coord-scope composition isolation, end-to-end
`memory(action='search')` tool execution, per-turn cache hit +
invalidation, and stable ordering under tied `updated` timestamps.

Memory test sweep: 102/102.  Broader regression
(session, storage, coordinator, load_skill): 411/411.

* fix(memory): address Copilot review on PR #468

Three follow-ups from Copilot's inline review:

1. SUPERSET invariant violation (Copilot, session.py:5510).
   `(search_hits + extra)[:fetch_limit]` capped the union back down to
   fetch_limit, evicting the recency tail when search added distinct
   hits.  Recency tail is exactly where ancient-but-recently-touched
   memories live — the recall this PR is supposed to improve — so
   tail eviction recreated the bug for the narrow case where a query
   term fell off the 16-cap and the matching memory sat in
   recency[40-49].  Drop the cap; both halves are already SQL-capped
   at fetch_limit, so the union is at most 2 × fetch_limit (~100 with
   defaults).  BM25 over 100 candidates in pure Python is sub-ms;
   irrelevant recency fillers get score=0 and don't pollute ranking.
   Updates the docstring to actually be honest about the invariant.
   Adds `test_recency_tail_preserved_when_search_adds_distinct_hits`
   that locks the behavior in: 5 search hits + 10 recency = 15-item
   pool, every recency item present, source="union".

2. Unbounded `query.split()` in normalize_search_terms (Copilot,
   _utils.py:74).  `str.split()` allocates the full token list before
   the cap-after-16 break, so a 100KB pasted query did MB of throwaway
   work even though only 16 tokens entered SQL.  Switch to
   `re.finditer(r'\S+', query)` — streaming iterator, stops scanning
   at the first 16 normalized terms regardless of input size.

3. Misleading + unbounded log term_count (Copilot, session.py:8571).
   `len(item["query"].split())` had two problems: same unbounded
   split as #2, and the value reported the raw input token count
   rather than the normalized term count that actually hit the SQL
   WHERE clause — misleading metric for an operator trying to
   understand storage-side behavior.  Switch to
   `len(normalize_search_terms(item["query"]))` — accurate count, and
   bounded for free via #2.

Refuted: github-code-quality flagged `...` bodies in the new Protocol
methods as "statement has no effect."  False positive — `...` is the
canonical Protocol body convention, used 213 other times in the same
file.

Memory test sweep: 103/103.  Broader regression: 411/411.
2026-05-03 13:40:30 -07:00
Patrick Buckley 1207d27363 fix(tests): isolate metrics-singleton swaps so they don't leak across files
CI failure on main: test_publish_records_metric_outcome saw an empty
calls list — its monkeypatch was patching a different metrics
instance from the one `_publish_models_metadata` reads.

Two changes:

- test_close_reason_persistence.py: replace the bare
  `srv_mod._metrics = MetricsCollector()` assignment in `_make_app`
  with an autouse `monkeypatch.setattr(srv_mod, "_metrics", ...)`
  fixture so the test's metrics swap auto-restores. Other test
  files (test_auth.py, test_server_attachments_endpoints.py) carry
  the same anti-pattern; left for a follow-up since they're not on
  the critical path here.

- test_server_node_models_metadata.py: switch the publish-helper
  metric test to a string-form `monkeypatch.setattr("turnstone.
  server._metrics", FakeMetrics())` so it replaces whatever binding
  the live module currently holds, regardless of what other tests
  did to it. Robust against future leaks of the same shape.
2026-05-03 13:40:29 -07:00
Patrick Buckley 733c9818d4 feat(coord): expose healthy model aliases per node on list_nodes (#466)
* feat(coord): expose healthy model aliases per node on list_nodes

Surfaces a `model_aliases` field on each `list_nodes` row so a
coordinator can discover which model aliases each cluster node will
accept on `spawn_workstream(model=...)` without an HTTP fan-out.

Each server projects its registry into a `models` entry on
`node_metadata` (`{alias, provider, healthy}` per alias) at lifespan
startup, on every 30s heartbeat tick, and after `internal_model_reload`.
The publish helper short-circuits on a payload-equality cache so a
stable cluster doesn't pay UPSERT churn — exposed via the new
`turnstone_node_models_publish_total{outcome="written|skipped"}`
Prometheus counter so operators can graph cache hit-rate.

Coord client filters the per-alias rows to healthy aliases only and
drops the provider-side model identifier (`cfg.model`) — coords kept
reaching for it when they should pass the local alias.

* fix(coord): address Copilot+CodeQL feedback on list_nodes models work

- internal_model_reload: reuse a single get_storage() local across the
  registry load and the metadata publish (Copilot:3047)
- _collect_node_models_metadata: iterate sorted aliases so two
  structurally identical registries built in different insertion orders
  serialize to the same JSON — directly improves the publish-cache hit
  rate exposed via turnstone_node_models_publish_total (Copilot:3105)
- tests: drop mixed turnstone.server import style flagged by CodeQL —
  hoist _metrics into the from-import block, and use sys.modules in
  the shutdown-race regression test instead of `import as srv`
2026-05-03 13:40:29 -07:00
Patrick Buckley 28a2779c10 fix(core): scope rehydrate fallback to manager, fix resume orphan
Address Copilot feedback on PR #465:

1. The has_alias fallback in both session_factories silently rewrote
   any unknown caller-supplied alias to the default, including on the
   fresh-create path where the create handler maps the factory's
   ValueError to a 503 with operator-friendly text. A typo in
   body.model would now silently start a workstream on the default
   instead of telling the caller their requested model could not be
   resolved. Move the fallback out of the factories: each factory
   raises again on unknown aliases, and SessionManager filters stale
   aliases out of the rehydrate path via a new ``model_validator``
   constructor kwarg (production wiring passes ``registry.has_alias``
   on both interactive and coordinator).

2. ChatSession.resume()'s elif branch flipped self.model to the
   persisted model name even when the alias was unresolvable, leaving
   the session paired with the constructor's default provider/client
   but a removed model name — a broken state whose next API call
   fails. Drop the model copy: keep the constructor's coherent
   default (provider + model + capabilities) and just log the
   unreachable saved values so the missing alias is auditable.

Tests:
- Move stale-alias coverage from the factory level into
  SessionManager (tests/test_session_manager.py): validator drops
  stale aliases before reaching build_session; live aliases pass
  through unchanged.
- tests/test_sessions.py renamed test_resume_restores_model →
  test_resume_keeps_defaults_when_alias_unresolvable to match the new
  contract.
2026-05-03 13:40:29 -07:00
Patrick Buckley 56364b0b5b fix(core): preserve workstream model + config on rehydrate
SessionManager.open() was calling build_session(ws) without a model
arg on the rehydrate path. The session_factory then resolved the
*current* default alias, ChatSession.__init__'s _save_config() (INSERT
OR REPLACE per-key) clobbered the persisted workstream_config with
those defaults, and the subsequent resume() "restored" what was now
the default — silently resetting model_alias, model, temperature,
reasoning_effort, max_tokens, skill, creative_mode, instructions,
token_budget, and notify_on_complete on every reopen and every
service restart, for both interactive and coordinator workstreams.

Three layers:

1. SessionManager.open() now reads workstream_config via
   self._storage.load_workstream_config(ws_id) and threads the saved
   model_alias into build_session(ws, model=saved_alias).

2. ChatSession.__init__ now skips its initial _save_config() when a
   workstream_config row already exists for self._ws_id — protects
   every other persisted knob without having to plumb each one
   through the adapter signature, and catches any future construction
   path that forgets to thread model through build_session.

3. Both session_factories (server.py interactive, console
   session_factory.py coordinator) now treat an unknown caller-
   supplied alias the same as an unset alias: fall back to the
   runtime default rather than raising. Without this, a workstream
   pinned to an alias an operator has since removed from the registry
   would 500 on every reopen — defeating the "best effort restore,
   default if the original is gone" contract this fix is meant to
   deliver. Mirrors _effective_default_alias's existing has_alias
   guard against a stale ConfigStore default.
2026-05-03 13:40:29 -07:00
Patrick Buckley afb5804a7c fix(console): address Copilot feedback on Models → Roles sub-tab
Three changes from PR review:

- Permission gating: hide the Roles sub-tab button when the user
  lacks ``admin.settings``.  The sub-tab loads/saves through
  ``/v1/api/admin/settings``, so an admin with ``admin.models`` but
  no ``admin.settings`` would otherwise see a perpetual 403 loader.
  When Roles is the active sub-tab and the permission check fails,
  snap the panel back to Definitions so the user lands somewhere
  usable.

- Drop the redundant ``/v1/api/admin/model-definitions`` fetch from
  ``loadAdminModelRoles``.  Both entry points (initial Models-tab
  open + ``models_changed`` SSE refresh) flow through
  ``loadAdminModels`` first, which already populates ``_modelDefs``
  + ``_modelDefaultAlias``; ``_saveModelRole`` doesn't touch model
  definitions, so the cached snapshot stays accurate when the save
  chains back here.  Halves the per-render request count and
  removes a wasted round-trip on every cluster-wide model edit.

- Add ``test_models_changed_event.py`` covering the SSE fanout the
  prior commit introduced: each model-definition CRUD endpoint
  emits exactly one ``models_changed``, settings PUT/DELETE only
  emit for keys in ``_MODEL_AFFECTING_SETTING_KEYS`` (parametrised
  over all eight), and unrelated settings (e.g.
  ``session.retention_days``) don't trigger spurious refreshes.
  The expected key set is pinned in the test so a stray addition
  to the allowlist doesn't silently bypass coverage.
2026-05-03 13:40:29 -07:00
Patrick Buckley 4b508a1319 feat(console): add plan_agent + task_agent to Models → Roles
Same shape as the coordinator/judge rows already there: alias dropdown
+ reasoning_effort dropdown sourced from the existing
``model.plan_alias`` / ``model.plan_effort`` and
``model.task_alias`` / ``model.task_effort`` settings.  Adds the four
keys to the SSE ``models_changed`` allowlist so changes from the
Settings API also trigger a live dropdown refresh, and filters them
out of the Settings tab so they only render in one place.
2026-05-03 13:40:29 -07:00
Patrick Buckley 9c2cb185e1 feat(console): consolidate role-model settings + live-refresh dropdowns
Lifts judge and coordinator model assignments out of their respective
admin tabs and into a new Models → Roles sub-tab so role overrides live
next to the model definitions they reference. Forward-looking shape for
the upcoming perception.{audio,image,video} model settings — adding a
new role is one entry in the declarative MODEL_ROLES array.

Also drops the misleading "Coordinator subsystem not configured" home
banner. The session factory already falls back to the registry's
default model when coordinator.model_alias is unset, so the banner was
nagging on fresh installs where the system was actually working. The
related _probeCoordSubsystem / _homeCoordReady plumbing went with it.

Wires SSE-driven live refresh: the console now emits a models_changed
event when a model definition is created/updated/deleted/reloaded, or
when a model-affecting setting (model.default_alias, judge.model,
coordinator.model_alias, coordinator.reasoning_effort) changes.
Connected browsers refetch /v1/api/models on receipt so the home
composer's model dropdown and the Roles sub-tab stay accurate without
a manual reload — fixes the case where editing the underlying model
for an existing alias left the dropdown showing the old model id.

Companion cleanups:
- Renamed .judge-section-* CSS classes to .admin-subtab-* and shared
  them with the Models sub-tab switcher (same a11y attrs, arrow-key
  nav). Old names had no other callers.
- Filtered judge.model out of the Judge Settings sub-tab and
  coordinator.model_alias / coordinator.reasoning_effort out of the
  Settings tab — they live exclusively under Models → Roles now.
- Reworded the _require_coord_mgr 503 messages to point operators at
  the Models tab instead of suggesting they set coordinator.model_alias.
2026-05-03 13:40:29 -07:00
Patrick Buckley 0519b847bd docs(skills): add import-conversation-history SKILL.md
Source-agnostic guide that teaches an agent Turnstone's destination
contracts (workstream + conversations schema, ws_id routing, OpenAI
message shape, tool-call/result pairing, provider_data fidelity blob,
attachment lifecycle) so it can map any external chat export onto them.
Validated against turnstone.core.skill_parser.
2026-05-03 13:40:29 -07:00
Patrick Buckley bbc8b99a9f fix(console): home composer attachments + coord chat user-message pills (#462)
* fix(console): home composer attachments + coord chat user-message pills

Two parity gaps in the console's coordinator surface:

- The embedded creator on the home page accepted only text — the
  paperclip / paste / drop pipeline that the in-coord composer and the
  interactive new-ws modal both expose was missing, so a user couldn't
  attach files at create time. Stage Files in memory (no ws_id yet) and
  ship them multipart on Start; the coord create endpoint already accepts
  multipart via create_supports_attachments=True.

- User messages with attachments rendered as plain text on both live
  send and history replay — no chip cluster like the interactive pane.
  Added appendUserMessageWithAttachments and a structured userAttachments
  list built from _attachments_meta (preferred) or the multipart parts
  themselves, then rendered the same .msg-user-attach pill strip the
  interactive pane uses.

Polish from a designer pass:

- Pill background was --panel-2, equal to the .msg bubble background in
  both themes (border contrast ≈1.4:1, below WCAG 1.4.11). Switched to
  --panel so the pill sits on a different surface than the bubble.
- Capped chip filename width inside the home composer (max-width 200px +
  ellipsis) so a long filename doesn't push the strip past the textarea.
- aria-live="assertive" → "polite" on #home-coord-error; client-side
  validation isn't an interrupt-level event.
- Reserved min-height on .home-composer-error and dropped the
  display: none/block toggling so validation messages no longer reflow
  the active-coordinators list below.

* fix(console): address PR #462 review feedback

- Block home-composer submit when files are staged but the task field is
  empty.  Server's _coord_create_post_install short-circuits on an empty
  initial_message, so the multipart upload would create pending
  attachment rows that never reserve onto a turn — orphaned until the
  GC sweep.  Fail in the browser instead.
- Drop the redundant `part &&` guard in coordinator.js's history-replay
  multipart loop; the earlier `if (!part || ...) continue` already
  filtered.
- Rewrite the home-mount .composer-chip-name CSS comment.  shared/chat.css
  defines .composer-chip{,-size,-remove} but no .composer-chip-name rule
  — the span inherits the parent chip font with no width cap.
- Add smoke-guard string assertions in test_coordinator_page.py for
  appendUserMessageWithAttachments and msg-user-attach so a future
  rename can't silently regress the attachment affordance.
2026-05-03 13:40:29 -07:00
Patrick Buckley bd9f780b21 chore: bump version to 1.5.5 2026-05-01 14:09:38 -07:00
Patrick Buckley 5d14b5f675 fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration (#461)
* fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration

Loading a saved workstream silently dropped tool results and missed
verdict / output-guard / truncation signals on replay. Root cause was
in `Pane.prototype.replayHistory`: an assistant message carrying both
content and tool_calls cleared the `lastToolBlock` anchor before the
following tool-result iteration could attach. The fix reorders content
to render before the tool block (matching live SSE order) and
restructures the tool-result branch to anchor by `data-call-id` so
multi-tool batches render `[hdr A][out A][hdr B][out B]` rather than
bunching outputs at the bottom.

Beyond the bug, replay now reaches near-parity with the live UX:

- Persisted intent verdicts and output_assessments flow through both
  the SSE replay (`_build_history`) and the `/history` REST endpoint
  used by coord. Single shared helper module owns the wire shape.
- Memory/recall calls persist instead of being filtered at storage
  time — full audit trail; UI dims them by default with hover-reveal
  so heavy memory usage doesn't crowd the narrative.
- Truncation indicator surfaces as a sibling pill (consistent across
  interactive + coord) when a tool result hit the 2000-char cap.
- `replayHistory` wraps DOM work in `aria-busy` so screen readers
  don't get a chatty announce-flood on long replays.
- `_build_history`'s storage I/O moves off the event loop via a new
  `events_replay_prepare` async hook for the SSE path; other async
  callers wrap in `asyncio.to_thread`.

Coord parity:

- `/history` REST endpoint decorates tool_calls with verdict +
  output_assessment + truncation flag (was previously raw
  `load_messages` output).
- Coord JS stamps `judge_verdict` / `heuristic_verdict` from
  history-loaded `tc.verdict` so the existing batch render paints
  the persisted pill, seeds the verdict cache to dedupe later live
  SSE events, and emits an inline `.coord-tool-row-warning` chip
  per call instead of a generic chat line.
- Memory/recall dim rule mirrored on `.coord-tool-row[data-tool-name=...]`.

* fix(replay): address PR #461 review feedback + raise tool-result storage cap

Copilot review feedback:

- Sibling-chain dim rule (memory/recall) now adds :focus-within
  alongside :hover for .tool-output / .media-embed / .output-warning
  / .tool-output-truncated — keyboard users tabbing into a faded
  subtree now get full opacity.
- ``cfg.open_post_load`` is now invoked via ``await asyncio.to_thread``
  so its sync ``_build_history`` call (storage I/O for verdict
  indexes + message reconstruction) doesn't block the event loop on
  every workstream open. Mirrors the SSE replay path that's already
  protected via ``events_replay_prepare``.
- Replaced the hardcoded ``2000`` literal in server.py and session.py
  with ``TOOL_RESULT_STORAGE_CAP`` from the shared decoration module
  so the UI truncation-pill detection can't silently desync from the
  storage write side.

While here:

- Raised ``TOOL_RESULT_STORAGE_CAP`` from 2000 → 10000. A 2000-char
  clip routinely cut grep / file-read bodies mid-line, leaving the
  audit trail useless for retrospective debugging. FTS5 + row size
  grow proportionally; the per-tool upper bound is still bounded
  upstream by ``_truncate_output``'s context-budget clamp.
- Updated the user-visible truncation-pill tooltip on both
  interactive and coord to reflect the new cap.
- ``test_decorates_tool_calls_and_marks_truncated`` now references
  the constant instead of a literal so it stays correct on future
  cap changes.
2026-05-01 14:06:08 -07:00
Patrick Buckley 4693fa95f1 chore: bump version to 1.5.4 2026-04-30 23:51:06 -07:00
renovate[bot] c3423d6606 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.8 2026-04-30 23:48:53 -07:00
Patrick Buckley 53f1222c22 refactor(coord): remove priority queue + queue depth indicator + broken CSS
Speculative reliability machinery from the Stage 3 push that turned
out not to address any user-visible bug. The actual fixes (state /
activity disjunction in handleChildState, bulk-fetch race fix in
_fetch_live_block, push approve_request via cluster bus) are what
resolved the wedged-row issues. Manual testing showed the per-tab
SSE listener queue depth never climbed past single digits even when
rows were stuck — overflow was never the cause.

Removed
- ``_CRITICAL_EVENT_TYPES`` + ``_put_with_priority`` helper.
- Per-tab listener queue selective drop (back to plain
  ``contextlib.suppress(queue.Full)`` everywhere).
- ``ClusterCollector._fanout`` reverts to the same.
- WebUI ``_broadcast_intent_verdict`` / ``_broadcast_approval_resolved``
  / ``_broadcast_approve_request`` revert to plain ``put_nowait``.
- ``_queue_stats`` periodic SSE emit + frontend status-bar indicator
  + the supporting CSS rules.
- Broken ``.approval-block`` ``transition: max-height`` /
  ``max-height: 80vh`` / ``overflow: hidden`` rules — the transition
  never fired (nothing toggled max-height) and ``overflow: hidden``
  clipped long verdict reasoning. Layout-shift on auto-expand jumps
  again, which is preferable to clipped content (Copilot review).

Tidied
- ``_CollectorProtocol`` / ``_ManagerProtocol`` method bodies switch
  from ``...`` ellipsis to docstring-only bodies, silencing four
  CodeQL "statement has no effect" warnings without changing the
  Protocol contract.

5024 passed, ruff + mypy clean.
2026-04-30 23:48:53 -07:00
Patrick Buckley 802d87a57f feat(coord): Stage 3 SessionManager Children primitive lift + cluster bus push paths
Lift the Children primitive out of CoordinatorAdapter into universal
SessionManager core primitives, replace the fragile poll + state-event
piggyback paths with first-class cluster bus event types for inline
approval delivery, and clean up the resulting frontend reducer.

Architecture
- New `turnstone/core/children_registry.py` — universal parent → children
  + reverse-lookup primitive with atomic `add_child` (returns parent UI
  for race-free dispatch). Lifted from `CoordinatorAdapter`.
- New `turnstone/core/child_source.py` — `ChildSource` Protocol with
  `SameNodeChildSource` (in-process via SessionManager state observer)
  and `ClusterChildSource` (cross-node via ClusterCollector listener).
- `SessionManager._on_state_change` upgraded to multi-subscriber
  (`subscribe_to_state` / `unsubscribe_from_state`) under a dedicated
  lock; CLI consumer migrated.
- `CoordinatorAdapter` shrunk: 731 → ~640 LOC. Children data lives in
  the registry; fan-out lives in ClusterChildSource. Backward-compat
  property facades dropped; tests updated to use the registry surface.

Cluster bus event vocabulary
- New event types `intent_verdict`, `approval_resolved`,
  `approve_request` flow through both `ClusterCollector._apply_delta`
  (translation from node SSE) and `emit_console_ws_*` (synthesis on
  console pseudo-node).
- `CoordinatorAdapter._dispatch_child_event` re-emits as
  `child_ws_intent_verdict` / `child_ws_approval_resolved` /
  `child_ws_approve_request` on the parent coord's SSE stream.
- New `_broadcast_intent_verdict` / `_broadcast_approval_resolved` /
  `_broadcast_approve_request` no-op hooks on `SessionUIBase`. WebUI
  pushes to the global queue; ConsoleCoordinatorUI pushes to the
  collector. `approve_tools` calls `_broadcast_approve_request` right
  after setting `_pending_approval` so the items reach the coord tree
  immediately, eliminating the bulk-fetch race.

Cleanups
- `pending_approval_detail` piggyback on `ws_state` / `cluster_state`
  removed end-to-end. Bulk fetch + explicit verdict / approve-request
  push are the canonical carriers.
- Browser `_judgePollTick` 90-second poll loop deleted; push path is
  authoritative.
- `urgent` flag on `scheduleLiveFetch` deleted (only caller was 409
  retry; replaced with `invalidateLiveBadge` + standard schedule).
- Console `_fetch_live_block` derives `pending_approval` from a
  disjunction (`activity_state="approval"` OR `state="attention"`
  OR detail present) so the bulk fetch can't return false during the
  state-transition race window.
- Coord-side merge guard in `flushLiveFetches` no longer clobbered:
  `handleChildState` only stamps `sseUpdatedAt` when authoritatively
  clearing detail.
- `child_locality` capability flag removed (was inert dead code).

Reliability
- Selective drop on listener queue overflow: critical event types
  (verdicts, approvals, ws_closed, child_ws_*) evict one oldest item
  to make room rather than dropping themselves on a full queue.
  Best-effort events (state ticks, content tokens, status, activity)
  drop as before. Applied to `SessionUIBase._enqueue`,
  `ClusterCollector._fanout`, and the `WebUI._global_queue` puts in
  the new broadcast hooks.
- `_state_subscribers` snapshot under a dedicated lock so concurrent
  subscribe / unsubscribe during dispatch can't shift the iterator.

UX / a11y
- Loading placeholder in renderChildRow keeps row height stable while
  the bulk fetch is in-flight (sr-friendly aria-label).
- Focus preservation across `_renderChildrenNow` (capture +
  restore by row + marker) and across targeted `_updateChildRow` swaps.
- Layout-shift transition on the approval block max-height; respects
  `prefers-reduced-motion`.
- Sidebar pending count: `(N children · M pending)`.
- Risk pill `aria-label` spells out level + confidence for SR users.
- Per-coord SSE listener queue depth surfaced in the status bar
  (`queue N/500`) with color escalation (warn at >50%, danger at >80%).

Tests
- 305+ test changes across 8 files. New unit tests for
  `ChildrenRegistry`, `ChildSource` (both impls + multi-subscriber
  observer), the new collector emit + apply_delta cases, the dispatch
  cases for new event types, the broadcast hook overrides on both
  WebUI and ConsoleCoordinatorUI, and the focus / placeholder /
  pending-count frontend assertions in `test_coordinator_page.py`.

5024 passed, ruff + mypy clean.
2026-04-30 23:48:53 -07:00
Patrick Buckley 8349d9994d feat(console): multi-select delete UX for Saved Coordinators (#458)
* feat(console): multi-select delete UX for Saved Coordinators

Mirror the per-server "Saved Workstreams" multi-select delete onto the
console's "Saved Coordinators" section.  Coordinator deletes go through
the existing routing proxy at POST /v1/api/route/workstreams/delete
(body-keyed by ws_id, since coordinators live on the node that owns
them) — no backend change required.

Pagination caps the visible page (and therefore the Select-All fan-out)
at 24.  Without it, a Select-All on a busy cluster would pin the
console proxy pool with hundreds of parallel deletes through the
fan-out router.  While in delete mode the saved-coordinators list is
frozen against SSE re-renders so visible cards don't shuffle out from
under the user's selections (drained on cancel / post-delete close).

Refactor: shared logic now lives in turnstone/shared_static/cards.{css,js}.

  * .ws-delete-* CSS moved out of ui/static/style.css into the shared
    sheet alongside .dashboard-card; the existing ui/static modal
    markup picks up class hooks instead of id-scoped rules.
  * createSavedCardsController() owns mode state, checkbox decoration,
    toolbar wiring, focus trap, modal lifecycle, and batch fan-out.
    Both ui/static (Saved Workstreams) and console/static (Saved
    Coordinators) instantiate one controller; ui/static is now ~300
    LOC lighter as a result.
  * Internalises stale-selection prune across SSE re-renders, the
    wsId->item lookup map (was O(selected x N)), and the aria-hidden
    wrap on the toggle button's emoji glyph.

Designer review tightened the affordance:

  * Modal close restores focus to the toggle button (was landing on
    <body>) — WCAG 2.4.3.
  * Modal [role="alert"] gets a red-chip treatment when populated,
    stays invisible at rest via :not(:empty).
  * Pagination consolidated onto the existing .pagination control
    (terse "X / Y" label + arrow-glyph buttons) instead of a parallel
    .coord-pagination treatment.
  * Filled destructive buttons darkened to #dc2626 in dark theme so
    the white label clears WCAG AA contrast (was 3.0:1 on --red).
    Light theme keeps --red unchanged (5.9:1 already passes).
  * Toolbar wraps below 700px viewport — Delete Selected drops to its
    own full-width row underneath count + Cancel + Select All for
    thumb-target separation.
  * .ws-card-check:focus-visible outline + word-break on
    .ws-delete-item for narrow-modal long aliases.

* fix(cards): address Copilot review feedback on PR #458

* closeModal focus restore now falls back to the section toggle button
  (opts.buttonId) when prevFocus is hidden or detached.  The post-delete
  Close path runs cancel() before closeModal(), which puts the bar at
  display:none — so the captured prevFocus (the bar's "Delete Selected"
  button) is no longer focusable and focus would land on <body>,
  defeating the WCAG 2.4.3 fix.  Esc / Cancel paths still land on the
  original focus owner because the bar stays visible in those flows.

* Saved Coordinators onClose drains _savedCoordsRetry before reloading.
  Without it, SSE events that arrived during the delete-mode freeze
  leave the retry flag true, so loadSavedCoordinators's .finally()
  re-fires a second fetch immediately after the first resolves.  Mirrors
  the same idiom in cancelCoordDeleteMode.
2026-04-30 23:48:53 -07:00
Patrick Buckley ac1fd67137 chore: bump version to 1.5.3 2026-04-30 13:34:35 -07:00
Patrick Buckley 1b40ae79f9 fix(storage): address PR #457 review feedback
Three issues from the Copilot review on PR #457:

1. SQLite race in bulk_close_stale_orphans (Copilot): the SELECT-then-
   UPDATE flow doesn't re-apply the eligibility predicates on the
   UPDATE, so a row that gets touch_workstream-bumped (or set_state-
   transitioned) between the two statements would still be flipped
   to closed.  Postgres dodges this via UPDATE...RETURNING (one atomic
   statement); SQLite needs the explicit re-application.  Fix: rebuild
   the WHERE conditions list once, apply on both SELECT and UPDATE,
   then SELECT-back by ``state='closed' AND updated=now`` to get the
   accurate closed-id list.  A row that became fresh between the two
   statements skips the UPDATE entirely.

2. SQLite IN-clause bind-parameter limit (Copilot): default 999 cap
   could be exceeded on a backlog reap (e.g. after a long outage).
   Chunked the candidate id list at 500 — same chunk size
   prune_workstreams (line 453) uses for the same reason.

3. Wall-clock-dependent test asserts (Copilot, two locations): the
   tests asserted ``updated > '2024-01-01T00:00:00'`` which is fragile
   on systems with skewed clocks or pre-2024 dates.  Replaced with
   ``updated != stale_seed`` — captures the same intent (the value
   was bumped) without depending on wall-clock date.

Two ``...``-as-no-op flags from github-code-quality were false
positives — ``...`` is the standard Python idiom for Protocol method
bodies and matches every other method in _protocol.py.  No code change.
2026-04-30 13:34:15 -07:00
Patrick Buckley b078ddccf0 fix(session_manager): scope orphan reaper by services.last_heartbeat
Replaces the ``node_id == self_node_id`` orphan-scoping heuristic from
earlier on this branch with liveness-based scoping using
``services.last_heartbeat``.  The heuristic was wrong for the post-#384
world: PR #384 (refactor: replace hash-ring rebalancer with rendezvous
hashing) deleted the rebalancer that used to keep workstreams.node_id
pointing at a live node.  Without it, ``workstreams.node_id`` is now
stamped at create time and never updated, so in containerized
deployments with dynamic hostnames a dead pod's rows have ``node_id``
matching no surviving service — they'd accumulate forever under the old
heuristic.

services.last_heartbeat is the same primitive the rendezvous router
uses for routing.  Reusing it here keeps reap scoping aligned with
routing: dead pods' rows fall out of the live set after the heartbeat
window and become reapable; alive pods' rows stay protected as long as
they heartbeat.

Mechanics:

- ``bulk_close_stale_orphans`` parameter renamed
  ``node_id: str | None`` → ``live_node_ids: list[str] | None``.  The
  WHERE clause becomes ``(node_id IS NULL OR node_id NOT IN
  live_node_ids)``.  ``None`` skips the filter entirely (single-process
  / tests / operator backfill).  ``[]`` treats every row as
  unprotected.
- ``SessionManager.close_idle`` pass 2 calls
  ``storage.list_services(self._service_type)`` to enumerate live
  peers, passes their service_ids as ``live_node_ids``.  ``_service_type``
  is derived from ``self.kind`` (INTERACTIVE→"server",
  COORDINATOR→"console") via a module-level mapping — no constructor
  param, so production wiring can't miswire the kind/service_type
  pairing.
- list_services failure → pass 2 is skipped this tick (conservative;
  never reap when liveness state is unknown).  Pass 1 still runs.
- ``workstreams.node_id`` with NULL value is always eligible — defends
  against ANSI ``NULL NOT IN (...)`` evaluating to NULL (not TRUE) and
  silently protecting orphans forever.
- Migration 048 simplified to ``(kind, updated)``; the new query's
  ``NOT IN (small list)`` predicate against an unbounded-cardinality
  column doesn't index well, so leading ``node_id`` would just add
  write cost.

Tests cover the live-services protection (own/dead/null cases), the
empty-peers reap-all case, the list_services-failure conservative
fallback, both kind/service_type pairings (interactive→"server",
coordinator→"console"), and the combined live_node_ids +
exclude_ws_ids filter matrix.
2026-04-30 13:34:15 -07:00
Patrick Buckley 4b6c93a0e9 perf(storage): partial composite index for the orphan reaper query
bulk_close_stale_orphans runs every min(300s, idle_timeout/4) on
every server and console process.  Its WHERE shape is:

    WHERE kind = ?
      AND state IN ('idle','thinking','attention','running')
      AND updated < ?
      AND node_id = ?           -- multi-node interactive only

At current scale the existing single-column indexes are sufficient —
idx_workstreams_state prunes to non-closed and the planner filters the
rest sequentially.  At 100k+ rows that filter becomes a tablescan-
shaped cost.

A partial index covering only BULK_CLOSE_STATE_VALUES rows matches the
reaper's query exactly while staying tiny — closed rows (typically
95%+ of the table) and error rows are excluded, so the index is
roughly 5% the size a full multi-column index would be.  Write
amplification only kicks in for transitions touching one of the four
covered states.

Column order (node_id, kind, updated): node_id is the most selective
filter for multi-node interactive (each server prunes to its own
node's rows), kind second so coord-only and interactive-only queries
within a node still get index-only scans, updated last so the range
comparison rides the trailing column.

Postgres uses CREATE INDEX CONCURRENTLY so the build is non-blocking
on a live system; SQLite has no concurrent concept and the table-
level write lock already serializes, so a plain CREATE INDEX is fine.
2026-04-30 13:34:15 -07:00
Patrick Buckley 9d283e951f fix(console): periodic idle cleanup for the coordinator pool
The console's coord SessionManager had no idle thread — close_idle was
never called for coordinator workstreams.  This is the worse half of
the lifecycle leak: the dashboard filters via the in-memory pool, so
DB-only orphan coords were invisible.  At empirical diagnosis,
coord closure was 16% (10 closed / 64 total) vs interactive 63%.

Adds _coord_idle_cleanup_thread mirroring turnstone/server.py's
_idle_cleanup_thread but skipping the rate-limiter / global-queue arms
the console doesn't have.  Started from the lifespan when coord_mgr is
constructed and server.workstream_idle_timeout > 0 (reuses the
existing setting — same cadence works for both kinds).

Initial sweep runs INSIDE the thread before the first sleep, not
synchronously in the lifespan: cold-start orphans are reaped without
blocking Starlette boot.  Important because cold start with many DB
orphans (the precise condition this code targets) is exactly when the
UPDATE is most likely to be slow.

Helper takes an optional stop_event parameter purely for tests —
production callers pass None and the daemon runs for process lifetime.
This avoids the SystemExit-from-stub + module-wide filterwarnings
fragility a previous iteration relied on.

Four tests: initial sweep runs before first sleep, ticks fire each
loop, exceptions don't kill the thread, stop_event exits cleanly.
2026-04-30 13:34:15 -07:00
Patrick Buckley 4e407e7d4f fix(session_manager): close DB-orphan workstreams in close_idle
Real bug: workstream rows accumulate in non-closed states (idle,
thinking, attention, running) when their owning process restarts or
crashes.  Empirical diagnosis on a live deployment found ~60 stuck
coord rows in DB invisible to the in-memory-keyed dashboard, plus
100+ interactive rows older than the 2h timeout (one stuck "thinking"
for 2 weeks — impossible across a process restart).

Root cause: close_idle iterates self._workstreams.values() — only the
loaded subset.  Anything left behind by a prior process incarnation
sits in DB forever because nothing ever re-loads it.

This commit gives close_idle a second pass.

Pass 1 (existing, unchanged): close loaded IDLE rows whose
ws.last_active (monotonic) is past timeout.  IDLE-only so legitimately-
attentive rows (waiting for user response) stay live.

Pass 2 (new): bulk-close DB rows of this manager's kind whose updated
is past the wall-clock cutoff and which aren't currently loaded.
Closes the broader BULK_CLOSE_STATE_VALUES set — any matching row is
by definition not loaded by any process and cannot be in a live
interaction.  Scoped by self._node_id so a sibling node can't reap
rows we own (multi-node interactive correctness).  No emit_closed —
never-loaded rows have no SSE listeners expecting them.

Lock invariant: pass 1 holds self._lock briefly to snapshot victims
and pop them (existing behavior).  Pass 2 holds self._lock briefly to
snapshot the loaded keys, then releases before the DB UPDATE so a slow
reaper query can't block create/get/set_state.

Also fixes a same-process race in open(): the rehydrate path read DB,
released the manager lock, then re-acquired to install — a concurrent
pass 2 between the two acquisitions snapshots loaded keys without the
in-flight ws_id, and could clobber its DB row to closed.  open() now
calls touch_workstream(ws_id) on rehydrate so the row's updated is
fresh against any pass-2 cutoff.  Pure timestamp write is safe against
concurrent close() (close still wins on the state column).

Three new tests cover the DB orphan pass (basic, exclude-loaded, kind
filter) plus node_id scoping (own/foreign rows, None-skips-filter) and
the open() rehydrate touch.
2026-04-30 13:34:15 -07:00
Patrick Buckley 7ab24e500b fix(storage): add bulk_close_stale_orphans + touch_workstream primitives
Two new methods on the StorageBackend Protocol, with implementations on
both Postgres (UPDATE ... RETURNING) and SQLite (SELECT-then-UPDATE in
one transaction).  No callers yet — wiring lands in subsequent commits.

bulk_close_stale_orphans(kind, cutoff, exclude_ws_ids, node_id=None)
flips rows in BULK_CLOSE_STATE_VALUES (idle/thinking/attention/running)
to closed when their updated timestamp is lex-older than cutoff.  The
node_id filter scopes the reap to a single node's partition — required
for multi-node interactive deployments where each node only has
authority over its own workstreams.node_id rows.  Excludes loaded ids
so the in-memory pass owns those.

touch_workstream(ws_id) bumps updated without changing state.  Used by
the open() rehydrate path to defend against the orphan reaper clobbering
a freshly-loaded row whose DB updated is older than the cutoff.  Pure
timestamp write is safe against concurrent close() because close still
wins on the state column.

BULK_CLOSE_STATE_VALUES is centralized in workstream.py so the two
backend implementations and FakeStorage all agree; if a new transient
state is added to WorkstreamState, deciding whether it joins this set
is part of the change rather than an after-the-fact audit across three
files.

Storage tests (run against both backends via the conftest fixture) cover
the kind/state/cutoff/exclude/node_id matrix plus touch_workstream.
2026-04-30 13:34:15 -07:00
Patrick Buckley 5bcbcb73b9 chore: bump version to 1.5.2 2026-04-30 03:15:59 -07:00
Patrick Buckley af6749421a fix(metacog): drop duplicate [repeat: tool()] info line
The themed ``tool_reminder`` bubble below the tool block already
shows the metacog text, and the tool block immediately above it
carries the tool name — so a separate gray ``[repeat: list_workstreams()
called with same arguments]`` info line was just duplicate visual
noise (operator-visible in the screenshot below the bubble).

Drop the ``ui.on_info`` call inside ``_apply_post_execute_advisories``
that emitted the diagnostic line.  Update the docstring to reflect
that the bubble is the canonical signal.  Rename
``test_emit_repeat_ui_line_on_streak_fire`` →
``test_no_legacy_repeat_info_line_on_streak_fire`` and invert the
assertion.
2026-04-30 03:15:22 -07:00
Patrick Buckley 4d6cb77075 fix(cli): add on_user_reminder + on_tool_reminder to TerminalUI
CI typecheck failed because ``WorkstreamTerminalUI(TerminalUI)``
inherits from ``SessionUI`` (the Protocol), and the Protocol's
``on_user_reminder`` / ``on_tool_reminder`` declarations have empty
bodies — mypy treats those as implicitly abstract, so the subclass
became un-instantiable.

Add real implementations on ``TerminalUI`` that render reminders as
``[metacognition · type] text`` lines in yellow.  This also restores
the metacog signal on the CLI surface (the legacy
``[metacognition: nudge injected — …]`` info-line went away with
``_emit_nudge_ping``; without this commit the CLI showed no signal
at all for metacog nudges).  Tool-channel and user-channel render
identically because terminal output is anchored by stdout flow
rather than by DOM anchor — the line lands directly after the
message it advises.
2026-04-30 03:15:22 -07:00
Patrick Buckley 5c225ef39b docs(metacog): align comments with side-channel + tool-channel scope
Address Copilot's review feedback on PR #456 — the docstrings and
inline comments hadn't all caught up with the architectural shift
across the branch:

  - ``_apply_reminders_for_provider`` docstring: "every user message"
    → role-agnostic, since tool messages also carry ``_reminders``
    (tool_error / repeat).
  - ``_mark_reminders_delivered`` docstring: same role-agnostic
    update; explicitly note both channels.
  - ``_append_user_turn`` callsite comment near
    ``_attach_pending_user_reminders``: still described splicing
    ``<system-reminder>`` blocks into user content; updated to
    reflect the side-channel attach + transient-copy splice at the
    provider boundary.
  - ``_build_history`` block comment: was user-message-only; now
    mentions tool messages and both ``user_reminder`` /
    ``tool_reminder`` SSE events.
  - ``_build_history`` propagation comment: same role-agnostic note
    on the per-entry surface.
  - ``app.js`` ``user_reminder`` SSE handler comment: said the
    bubble renders "above" the user message, but
    ``insertAdjacentElement('afterend', el)`` drops it BELOW.
  - ``app.js`` ``replayHistory`` comment: said "insertBefore drops
    the reminder directly above the just-rendered user bubble";
    same fix — bubble lands BELOW.

No behaviour change.
2026-04-30 03:15:22 -07:00
Patrick Buckley f5a843f44a fix(metacog): drop write-success-clear so sequential same-call streaks fire
The repeat-detection block in ``_apply_post_execute_advisories`` had
a leftover "clear streak when a write tool succeeded" branch from
when ``RepeatDetector`` tracked cumulative counts.  With the
consecutive-streak semantics introduced earlier in the branch the
branch became:

  1. Redundant — any different (name, args) signature already resets
     the streak via ``RepeatDetector.record``, so an intervening
     read/write naturally breaks the streak.
  2. Actively wrong — the clear runs ONCE at the top of each
     ``_apply_post_execute_advisories`` call, before the per-result
     loop records sigs.  In a single parallel batch
     ``[bash, bash, bash]`` the clear runs once and then three
     ``record`` calls accumulate to count=3 in the same call → fires.
     But across three sequential turns, each turn calls
     ``_apply_post_execute_advisories`` fresh, the clear runs at the
     top of each call, and only one ``record`` per call follows — so
     the count never gets above 1 and the canonical
     "small local model stuck on ``bash('echo test')``" pattern
     never triggered the nudge.

The asymmetry only existed for successful calls — failures don't
satisfy the ``not _tool_error_flags.get(tc["id"])`` predicate, so
the clear didn't fire and sequential failures already worked.  The
fix is to drop the clear entirely; ``RepeatDetector``'s
consecutive-streak semantics handle every case uniformly.

Tests:

  - ``test_successful_write_clears_streak`` →
    ``test_intervening_different_call_resets_streak`` —
    rewords the assertion to reflect the actual mechanism (any
    different sig resets, write-or-otherwise) since "writes clear"
    was the bug, not the contract.
  - ``test_failed_write_does_not_clear_streak`` →
    ``test_sequential_bash_failures_fire_repeat`` — same shape, just
    framing fixed.
  - New ``test_sequential_bash_same_command_fires_repeat`` —
    regression for the bug user hit (three sequential successful
    ``bash('echo test')`` calls now correctly fire the nudge).
2026-04-30 03:15:22 -07:00
Patrick Buckley cf44841624 feat(metacog): themed reminder bubble unifies user + tool channels
The yellow themed reminder card introduced for user-channel nudges
(correction / denial / resume / start / completion) now also fronts
tool-channel nudges (tool_error / repeat).  Pre-fix the tool channel
shipped its reminders inside the tool-result envelope via
``wrap_tool_result``, leaking the ``<system-reminder>`` block into
``self.messages`` content (same problem the user channel had before
the side-channel refactor) and surfacing the legacy gray
``[metacognition: nudge injected — …]`` info line as the only
operator-visible signal — duplicated alongside the new themed bubble
for user-channel nudges.

Tool-channel parity:

  - ``_collect_advisories`` now returns
    ``(persistent_advisories, metacog_reminders)``.  Persistent
    advisories (``GuardAdvisory`` / ``UserInterjection``) keep
    riding ``wrap_tool_result`` because they ARE conversation
    history.  Metacognitive reminders extract to the second tuple
    element; the caller attaches them to the tool message dict's
    ``_reminders`` side-channel and emits ``on_tool_reminder``.
  - ``_apply_reminders_for_provider`` already handles ``_reminders``
    on any role, so the tool-channel splice into wire content is
    free.  ``_build_history`` also already propagates
    ``entry["reminders"]`` regardless of role, so reload renders the
    bubble too.
  - ``SessionUI`` Protocol gains ``on_tool_reminder(reminders,
    tool_call_id)``; ``SessionUIBase`` enqueues a ``tool_reminder``
    SSE event with the ``tool_call_id`` anchor.
  - ``_emit_nudge_ping`` had no remaining callers and was removed —
    the themed bubble (live SSE + ``/history`` reload) is the
    canonical operator signal for both channels now.

UI polish (the four fixes the screenshot caught for the user
channel + their tool-channel mirror):

  - Bubble renders BELOW the message it advises (semantically: a
    hint to the model right before its turn).  ``addUserReminder``
    swaps ``insertBefore`` for ``insertAdjacentElement('afterend',
    el)``; ``addToolReminder`` anchors below the ``.ts-approval``
    block whose tool result triggered the batch's reminder.
  - Label uses the full feature name ``metacognition`` (was the
    ``metacog`` shorthand).
  - Card width / alignment inherits from the base ``.msg`` rule —
    ``align-self: flex-end`` and the explicit ``max-width`` are
    gone, so the card matches the user / assistant column instead
    of pinning right-aligned narrow.
  - The legacy ``[metacognition: nudge injected — …]`` gray info
    line is gone for both channels.

Frontend additions:

  - ``Pane.prototype.addToolReminder(reminders, toolCallId)``
    anchors below the ``.ts-approval`` block (live: by
    ``data-call-id``; replay: by "last block in messagesEl"
    fallback, which is correct because messages render in order).
  - SSE switch case ``"tool_reminder"`` calls ``addToolReminder``.
  - ``replayHistory``'s tool-message branch now calls
    ``addToolReminder`` when ``msg.reminders`` is present.
  - ``addUserReminder`` advances its anchor on each loop iteration
    so multiple reminders stack in queued order rather than
    reversed.

Coord console parity:

  - ``coordinator.js`` gains ``appendReminderBubble`` /
    ``appendUserReminderLive`` / ``appendToolReminderLive`` mirroring
    the interactive UI.  The tool-channel anchor walks
    ``toolRows[callId].batch`` to attach below the
    ``.coord-tool-batch`` construct (one bubble per dispatch turn,
    matching the "one nudge per batch even with many failing tools"
    drain).
  - SSE switch handles ``user_reminder`` and ``tool_reminder`` on
    the coord conversation surface.
  - ``/history`` replay propagates ``msg.reminders`` for user and
    tool messages — same wire shape as the interactive pane.
  - ``.msg.user-reminder`` styles moved to
    ``shared_static/chat.css`` so both surfaces inherit the same
    yellow themed bubble from the shared base.

Defensive read on ``_apply_reminders_for_provider`` (per Copilot
review on the closed PR): a malformed ``_reminders`` entry (string,
None, etc. — corruption / partial state) used to abort ``send`` via
AttributeError on the ``.get("text", "")`` call.  Filter to dicts
before building the block, mirroring the same filter
``_build_history`` already applies on the wire-out side; an
all-malformed list passes through as no-reminders.

Tests:

  - ``test_collect_advisories_drains_tool_buffer_on_last_result``
    rewritten to assert the ``(persistent, metacog)`` tuple shape
    and that ``MetacognitiveAdvisory`` no longer appears in the
    persistent list.
  - ``test_collect_advisories_holds_*`` and ``_drops_*`` updated for
    tuple return.
  - ``test_attach_emits_visibility_ping`` /
    ``test_collect_advisories_emits_visibility_ping`` inverted to
    assert the legacy gray line is gone on both channels.
  - ``TestSessionUIBaseToolReminderHook`` covers the new SSE event
    shape with the ``tool_call_id`` anchor.
  - ``test_malformed_reminders_filtered_out`` and
    ``test_all_malformed_reminders_passes_through`` cover the
    Copilot-flagged defensive filter.
2026-04-30 03:15:22 -07:00
Patrick Buckley 7ffab6a272 fix(session): metacog reminders ride a side-channel, not user content
User-channel metacognitive nudges (correction, denial, resume, start,
completion) used to be spliced into ``user_msg["content"]`` permanently,
which leaked the ``<system-reminder>`` envelope into every consumer of
``self.messages`` — UI replay (mitigated by a regex strip in /history),
compaction, title generation, and any future channel adapter that
echoes conversation context.  The /history strip was a band-aid;
compaction and title-gen still saw the raw spliced text.

Switch to a side-channel: ``_attach_pending_user_reminders`` writes the
rendered reminder list to ``user_msg["_reminders"]`` (sibling key,
leading-underscore convention shared with ``_attachments_meta`` /
``_provider_content``).  At the provider boundary, a new
``_apply_reminders_for_provider`` builds a transient shallow-copy with
the reminder spliced into ``content``; the original message dict
stays clean.  ``sanitize_messages`` drops the sibling key on the wire.

Once-per-session-not-per-turn semantics for the wire: after stream
success the loop calls ``_mark_reminders_delivered``, which flips a
``_reminders_delivered`` flag on every user message that carried
reminders into that call.  ``_apply_reminders_for_provider`` skips
already-delivered messages so the model sees each reminder exactly
once (the turn it advised).  ``_build_history`` ignores the delivered
flag entirely, so reconnecting tabs render the same nudge bubble the
originating tab saw via the live ``user_reminder`` SSE event.

UI surface:

  - ``SessionUIBase.on_user_reminder`` enqueues a
    ``{type: "user_reminder", reminders: [...]}`` SSE event with the
    same shape ``_build_history`` surfaces.
  - ``app.js`` renders a ``.msg.user-reminder`` bubble (yellow accent,
    pill-styled) anchored above the user message it advises, both
    live and on history replay.
  - ``replayHistory`` renders ``addUserMessage`` before
    ``addUserReminder`` so the anchor lookup finds the just-rendered
    turn (not a prior one).
  - Multi-tab caveat documented inline: non-originating tabs receive
    no ``user_message`` SSE event today, so a reminder may anchor to
    a stale prior bubble until ``/history`` reload corrects it.

Pre-existing bug surfaced by the audit: cancel handlers
(``GenerationCancelled`` / ``KeyboardInterrupt`` / generic
``Exception``) in ``ChatSession.send`` cleared
``_pending_tool_advisories`` but not the user-channel buffer.  Both
now drain through a shared ``_drain_pending_advisories`` helper.

Removed the ``/history`` regex strip — the side-channel approach
makes it redundant.  Hoisted ``escape_wrapper_tags`` +
``render_system_reminder`` imports to module top (called 2-3× per
turn).

Tests:

  - ``TestApplyRemindersForProvider`` — pass-through-by-reference,
    string + list content splice, escape on user-typed wrapper tags,
    multi-reminder ordering, source-untouched invariant, delivered
    flag skip path, fallback for unexpected content shape.
  - ``TestMarkRemindersDelivered`` — flag idempotency, no-reminders
    no-flag, only marks user messages with reminders.
  - ``TestUpdateTokenTableMsgsParam`` — calibration uses pre-built
    msgs when provided, falls back when not.
  - ``TestUserAdvisoryCancelClear`` — all three cancel branches drain
    the user buffer.
  - ``TestReminderSidechannelIsolation`` — compaction's
    ``_format_messages_for_summary`` and the title-gen extraction
    loop cannot see reminders by construction.
  - ``TestSessionUIBaseUserReminderHook`` — ``on_user_reminder``
    enqueues the right SSE shape.
  - ``TestBuildHistoryReminderPropagation`` — ``entry["reminders"]``
    propagation, absent / empty / multi / coexist-with-attachments
    cases, malformed input filtering, all-malformed elision.
  - ``test_sanitize_messages_strips_underscore_sibling_keys`` covers
    ``_reminders`` and ``_reminders_delivered``.
2026-04-30 03:15:22 -07:00
Patrick Buckley ba3bc9d989 fix(metacog): N>=3 streak detector + drop redundant error-prefix list
Cleanup pass on the metacognitive nudge stack — restores pre-split
errored-counts-toward-repeat behaviour and tightens the is_error
plumbing through the per-batch advisory hook.

The per-batch hook in ``_run_loop`` was duplicating the is_error
signal: ``self._tool_error_flags`` (set by ``_report_tool_result``)
and a string-prefix tuple (``Error`` / ``JSON parse error`` / …).
Two truth sources is what got us here — bash commands that exit
non-zero with normal stdout matched the flag but not the prefix,
the deny path matched the prefix but not the flag, and the result
was that stuck-loop detection silently broke for the most common
failure mode (the model bashing the same broken command).

Single source of truth now:

- ``_execute_tools.run_one`` deny branch routes through
  ``_report_tool_result(is_error=True)`` so denied calls populate
  ``_tool_error_flags`` like every other error path.
- The error-prefix tuple is gone; the write-success-clear gate and
  the tool-error-nudge gate both read ``_tool_error_flags`` only.

Repeat-detection state moves from a ``set[str]`` (fired on the second
identical call, ignored errors entirely) to a ``RepeatDetector``
helper in ``metacognition.py`` with consecutive-streak semantics:

- Threshold raised from 2 to 3 — two-in-a-row was noisy on
  legitimate transient retries; three is the cheapest stuck-loop
  signal.
- Recording a different signature resets the count, so [A, A, B, A]
  is two short streaks of 2 and not a streak of 4. Bounded by O(1)
  state regardless of session length.
- Errored calls now count toward the streak (the split into a
  separate metacog module unintentionally introduced a "skip errors"
  branch — restored).

While there:

- ``metacognition._COOLDOWN_SECS`` default aligned to 300s (matches
  ``MemoryConfig.nudge_cooldown`` and the ``memory.nudge_cooldown``
  config-store default; was set to 30 by an earlier investigation).
- The per-batch advisory block (~80 lines of mixed orchestration
  inside ``_run_loop``) is extracted to
  ``ChatSession._apply_post_execute_advisories`` so the wired
  behaviour is testable without driving ``_run_loop`` end-to-end.
  Producer extraction to a dedicated module is deferred to a
  follow-up; advisory producers all live on ``ChatSession`` for
  now per existing convention.
- Frontend ``appendToolOutput`` (turnstone/ui/static/app.js) now
  skips rendering when the parent approval block is denied or
  the output starts with ``Denied by user`` / ``Blocked``,
  mirroring the history-replay guard at ``_build_history``.
  Previously the live SSE path didn't need this guard because
  the deny path never emitted a ``tool_result`` event; the
  is_error routing change above means it does now, so without
  this guard the badge from ``resolveApproval`` and the SSE
  output would both render.

Tests: 8 unit tests for ``RepeatDetector`` covering streak,
threshold, clear, and intervening-sig reset; 9 integration tests
for ``_apply_post_execute_advisories`` covering the wired
behaviour (3-identical fires warning + advisory + UI line, errored
calls count toward streak as a regression guard, intervening sig
resets streak, successful write clears, failed write does not,
JSON outputs tracked but not inline-warned, tool_error nudge gates
on memory_count, repeat UI line emitted on streak fire).
2026-04-30 03:15:22 -07:00
Patrick Buckley dbe023b4dd chore: bump version to 1.5.1 2026-04-29 20:21:12 -07:00
Patrick Buckley 3d3a8b7367 docs(coord): tighten handleChildState comment per Copilot review
The pre-existing comment said pending_approval_detail "rides on
every ws_state event" — that overstated the case.  The node-side
emit is gated on ``_pending_approval is not None`` so the field is
absent on the steady-state broadcast and possibly null on a node
mid-rolling-upgrade.  The handleChildState fallback already
handles both cases; only the comment was wrong.
2026-04-29 20:20:38 -07:00
Patrick Buckley 99eff73a97 feat(coord): pass pending_approval_detail on child_ws_state SSE events
Inline child approve/deny in the coord tree UI was rendering downstream
of the bulk-live cache (``GET /v1/api/cluster/ws/live``), not the SSE
stream. ``child_ws_state`` events were tiny notifications that fired
an urgent live-bulk fetch on every activity_state transition into/out
of "approval", just to pick up the rich ``pending_approval_detail``
payload. With multiple coord tabs and multi-child workstreams, that
urgent-fetch pattern compounded the SSE-executor pressure Shape A
is unwinding.

Thread the field through every layer so the SSE event itself carries
the rich payload — browser mutates ``liveBadgeCache`` directly,
no urgent fetch:

  1. Node ``WebUI._broadcast_state`` emits ``pending_approval_detail``
     on ``ws_state`` events. Gated on ``_pending_approval is not None``
     so the per-broadcast verdict-cache deepcopy only runs when there
     is actually an approval pending. ``_build_node_snapshot`` also
     projects the field so the console's reconnect-via-snapshot
     resync path delivers it (without this the new collector
     forwarding would never see the field on a snapshot row).

  2. Console ``ClusterCollector._apply_delta`` (live ``ws_state``
     forwarding) and ``_reconcile_node`` (snapshot resync diff) both
     forward the field on the emitted ``cluster_state`` event, AND
     ``_apply_delta`` persists it on the cached ``ws`` dict so the
     ``get_node_detail`` / ``get_snapshot`` endpoints between
     reconciliations don't render stale approve/deny buttons.

  3. ``CoordinatorAdapter._dispatch_child_event`` re-emits the field
     on the ``child_ws_state`` event sent to coord listener queues.

  4. Frontend ``handleChildState`` reads ``ev.pending_approval_detail``
     and writes it directly into ``liveBadgeCache``, tagging the
     entry with ``sseUpdatedAt``. ``flushLiveFetches`` honors that
     tag for ``SSE_AUTHORITATIVE_MS`` (3s) — the upstream
     ``/dashboard`` cache has its own ~2s TTL, so a bulk-poll
     landing right after a transition can otherwise clobber the
     fresh SSE-set state with pre-transition data.

The pre-fix ``enteredApproval`` / ``leftApproval`` urgent-fetch
branch is removed. The 409 stale-call_id retry path keeps its own
urgent fetch — that's a different scenario.

Tests cover the forwarding contract at every layer, the broadcast
gate (event includes the field when an approval is pending,
omits it otherwise, and clears after resolution), and the
``flushLiveFetches`` merge-guard structural shape so a refactor
that keeps the symbols but inverts the comparison or drops the
``prev.live`` check can't pass silently.
2026-04-29 20:20:38 -07:00
Patrick Buckley 99fcd30299 fix(console): offload sync DB calls in coord children/tasks handlers
``coordinator_children`` was calling ``storage.list_workstreams``
directly on the event loop, ``coordinator_tasks`` did the same with
``load_task_envelope``, and ``_resolve_coordinator_or_404`` (called
from both handlers, plus ``coordinator_history`` and
``_resolve_coord_session``) did the same with
``storage.get_workstream`` on its cold-cache path.

The cold-cache resolver path is hit on every console restart,
coordinator eviction, and console proxy hop — exactly when the
event loop is most contended. Three coord tabs reconnecting after a
brief network blip = three serial event-loop blocks per call site.
Other lifted handlers in this file already use
``asyncio.to_thread``; bring all four call sites onto the same
pattern.

Convert ``_resolve_coordinator_or_404`` to ``async def`` and update
its four call sites to ``await``. Exception flow is unchanged.
2026-04-29 20:20:38 -07:00
Patrick Buckley 423c2e80b7 fix(console): isolate coord SSE polling on a dedicated 200-thread pool
Each coord ``events`` SSE listener parks a thread on
``client_queue.get(timeout=5)`` for the connection lifetime. The
console's coord endpoint was wiring no ``sse_executor_lookup`` on
``coord_endpoint_config``, so those parks landed on Python's default
ThreadPoolExecutor (~min(32, cpu_count+4)) and competed with every
other ``asyncio.to_thread`` caller (storage, router, audit). A few
coord tabs against a multi-child workstream would stall new request
handlers waiting for a worker thread.

Mirror the interactive-side precedent (the ``sse_executor`` /
``sse_executor_lookup`` pattern in ``turnstone/server.py``) — build a
dedicated 200-thread ``coord_sse_executor`` in the console lifespan
and wire ``sse_executor_lookup`` onto ``coord_endpoint_config``.
Drain order matters: shut the pool down AFTER ``coord_adapter.shutdown()``
so no new listeners arrive at a dying pool. ``cancel_futures=True``
discards queued-but-not-started futures during teardown.

Update the stale comment on the interactive-side wiring that claimed
"coord wires None and falls back to the default executor" — it now
points at the console's matching wire.
2026-04-29 20:20:38 -07:00
Patrick Buckley a0eb77360d fix(coord): tighten coord_registry refresh logging + comments per round-2 review
Three follow-ups from Copilot's round-2 review on #453.

ValueError logging surfaced the wrong reason
The catch-all ``except ValueError:`` logged ``reason=no_enabled_rows``
unconditionally, but ``ModelRegistry.__init__`` raises ValueError for
five distinct config issues (empty models, default / fallback / agent /
plan / task alias not present).  Operator looking at logs for a
config.toml typo would see the wrong cause.  Switch to
``log.warning("...reason=%s", exc)`` so the actual error message
threads through.  Behavior unchanged — existing registry still
preserved on every ValueError path.

Misleading shutdown() comment
The ``finally`` comment claimed shutdown() was closing clients the
throwaway registry created during DB load.  ``load_model_registry`` only
constructs ModelConfigs and the bare ``ModelRegistry(...)``;
``ModelRegistry.__init__`` leaves ``_clients`` / ``_providers`` empty
and they populate lazily on first resolve.  Today shutdown() iterates
empty dicts.  Comment now says so explicitly while keeping the call
(and its try/except) for forward-compat against an eager-init future.

Stale "probe" wording in test docstring
``test_helper_preserves_registry_when_db_probe_fails`` →
``test_helper_preserves_registry_when_strict_load_fails``.  The
explicit probe was removed in commit 1ba17ed when the helper switched
to ``load_model_registry(..., strict=True)``; the test name and
docstring still talked about a probe.  Updated wording reflects that
the loader's strict-mode re-raise is what the helper catches now.

132 tests pass.
2026-04-29 20:20:38 -07:00
Patrick Buckley 3dd0e196fe refactor(coord): hygiene pass on coord_registry refresh — async + selective teardown + test cleanup
Hygiene follow-ups from the multi-stage code review on #453.

perf-1 — sync helper called from async route handlers
``_refresh_coord_registry`` runs two sync DB reads and a registry reload
that takes ``_client_lock``; calling it directly from an async handler
held the event loop for the duration.  All four call sites now
``await asyncio.to_thread(_refresh_coord_registry, ...)``, matching the
pattern from commit ``1f7d6ad`` (offloaded ``tenant_check``).

perf-3 — ModelRegistry.reload() tore down all clients unconditionally
The reload always closed every cached client and provider, even when
the changed fields (``model``, ``temperature``, ``context_window``)
didn't touch the connection target.  Now selective: clients drop only
when alias removed or ``(base_url, api_key, provider)`` differs;
providers drop only when alias removed or ``provider`` string differs.
Keeps connection pools warm across the common admin-edit case where
only metadata changed.  Two new ``test_model_registry`` cases lock the
keep-warm vs drop-on-change behaviour, and the existing
``test_reload_clears_clients`` was updated (it asserted the old
overly-aggressive contract) into
``test_reload_keeps_clients_when_connection_target_unchanged``.

q-5 — helper rename
``_refresh_console_coord_registry`` → ``_refresh_coord_registry``.  The
``console_`` prefix was redundant given the function lives in
``turnstone/console/server.py`` and sibling helpers there
(``_notify_nodes_model_reload``, ``_publish_config_change``,
``_collect_model_status``) all omit it.

q-1 — shared test middleware
``tests/test_admin_model_registry_refresh`` now imports the
header-driven ``_AuthMiddleware`` from ``tests/_coord_test_helpers``
and sets default ``X-Test-User`` / ``X-Test-Perms`` headers on the
``TestClient``.  The local hardcoded variant duplicated infrastructure
the helper module exists to centralise.

q-3 — multi-alias test registry
``_make_registry`` extracted a ``_make_config`` helper and gained an
``extras={alias: model}`` param so multi-alias scenarios stop
hand-building ``ModelConfig`` literals.
``test_delete_endpoint_refreshes_registry`` now uses the helper.

310 tests pass across the related coordinator + model surfaces.
2026-04-29 20:20:38 -07:00
Patrick Buckley 19c3db5329 test(coord): lock the empty-body gate with a refresh-call spy
bug-3 / q-2 from the multi-stage review on #453: the previous test
``test_update_endpoint_with_empty_body_does_not_blow_up`` asserted only
that the registry's model name was unchanged after an empty PUT, which
holds whether or not the refresh ran (DB row matches registry → refresh
is idempotent).  A regression that always called
``_refresh_console_coord_registry`` — exactly the gate this test was
meant to lock — would have left the assertion green.

Rename to ``test_update_endpoint_skips_refresh_on_empty_body`` and spy
on the helper via ``monkeypatch.setattr``.  Empty-body PUT must register
zero calls; any future change that drops the ``if updates:`` gate now
fails loudly.
2026-04-29 20:20:38 -07:00
Patrick Buckley 0bea72019e fix(coord): strict-mode loader + guarded shutdown for coord_registry refresh
Two correctness follow-ups from the multi-stage code review on #453.

bug-2 / perf-2 (DB probe was theatre + double scan)
The previous probe defended nothing the loader didn't already swallow
on the next line: ``load_model_registry``'s row-loop catches Exception
internally, so a transient DB error after the probe still degrades to
a config.toml-only registry that ``existing.reload()`` would apply,
silently dropping every DB-sourced alias.  And on the happy path each
CRUD paid for two scans of ``model_definitions``.

Add a ``strict: bool = False`` flag to ``load_model_registry``.  When
strict, the row-loop's except re-raises instead of swallowing.  The
helper passes ``strict=True`` and drops the probe — single DB scan,
real failure isolation, the loader's silent fallback can no longer
mask a partial-result regression.  Default ``strict=False`` so CLI /
lifespan callers keep their boot-with-config-fallback behaviour.

bug-1 (shutdown could escape after a successful reload)
``ModelRegistry.shutdown()`` calls ``client.close()`` unguarded, and the
helper's ``finally`` block ran it outside the try/except.  A raising
close() after a successful ``existing.reload()`` would surface as 500
with the registry already mutated and the audit row already recording
success.  Wrap ``new_registry.shutdown()`` in its own try/except that
matches the helper's belt-and-suspenders error policy elsewhere.

The helper's docstring also drops the obsolete probe paragraph; the
``if existing is None: return`` branch gets a one-line inline comment
about the boot-from-empty case (the multi-paragraph version restated
behaviour the line itself documents).

129 tests pass (test_admin_model_registry_refresh + test_model_registry).
2026-04-29 20:20:38 -07:00
Patrick Buckley b9ff52d582 fix(coord): tighten coord_registry refresh — DB probe + accurate boot-from-empty docstring
Two follow-ups from Copilot review of #453:

1. ``load_model_registry`` swallows storage read errors internally
   (logs + continues with config.toml-only models).  Without a strict
   probe in the helper, a transient DB outage on an admin CRUD would
   apply a truncated registry that drops every DB-sourced alias —
   silently, since the loader returns a non-empty registry built from
   ``[models.*]`` config.toml entries.  Add an explicit
   ``storage.list_model_definitions(enabled_only=True)`` probe before
   the loader call so the failure is visible here and the existing
   registry is preserved on outage.

2. The previous docstring claimed ``admin_model_reload`` "has its own
   boot-from-empty story."  It doesn't — it just calls this helper,
   which no-ops when ``coord_registry`` is None.  When no model rows
   existed at boot, lifespan leaves the entire coord subsystem
   uninitialized (no ``coord_mgr``, no ``coord_adapter``, no
   ``session_factory``), and a console restart remains required after
   the operator adds the first row.  Tighten the docstring to admit
   that limitation rather than overstating the helper's reach.

New test ``test_helper_preserves_registry_when_db_probe_fails``
monkeypatches ``list_model_definitions`` to raise and asserts the
existing registry stays intact.
2026-04-29 20:20:38 -07:00
Patrick Buckley 961f999c93 fix(coord): auto-refresh console coord_registry on model-definition changes
The console builds ``app.state.coord_registry`` once at lifespan startup
and the coordinator session factory closes over that exact instance.
Until now, the model-definition admin endpoints (create/update/delete)
wrote to the DB but never touched the in-process registry — and the
explicit reload button only fanned out to nodes via HTTP, also leaving
the console's own registry stale.

Symptom: an operator who changed the underlying model name behind a
local-LLM alias (same alias, same endpoint) saw the DB row update
immediately, but coordinator sessions kept calling the prior model
name until the console process was restarted.

Fix: a new helper ``_refresh_console_coord_registry`` rebuilds a fresh
ModelRegistry from DB and applies it to ``app.state.coord_registry``
via the existing thread-safe ``ModelRegistry.reload()`` — in-place
mutation preserves object identity so the factory closure keeps
working, and active coord sessions auto-pick up the swap on their
next ``send()`` via ``ChatSession._refresh_model_from_registry``.

Wired into four endpoints in ``console/server.py``:

- ``admin_create_model_definition`` — after the DB write
- ``admin_update_model_definition`` — after the DB write, gated on
  ``if updates:`` so a no-op PUT skips the rebuild
- ``admin_delete_model_definition`` — after the DB write
- ``admin_model_reload`` — between ``_publish_config_change`` and
  ``_notify_nodes_model_reload`` so the console mirrors what the
  reload broadcasts to nodes

Failure isolation: a load or reload error leaves the existing registry
intact (logged + swallowed). Coord stays usable while the operator
investigates; the explicit reload remains the user-facing recovery path.

No node fan-out on CRUD — the explicit reload button continues to gate
cluster-wide HTTP propagation, preserving today's UX semantics on shared
clusters.

Tests in ``tests/test_admin_model_registry_refresh.py`` cover:

- helper-level: rebuild from DB, identity preservation, no-op when
  registry is None, preservation on load failure / no-enabled-rows /
  reload validation error
- endpoint-level: create / update / delete / explicit-reload all
  refresh the registry; an empty PUT skips the rebuild
2026-04-29 20:20:38 -07:00
Patrick Buckley 8bdb916064 fix(coord): raise wait_for_workstream message cap to 10 KiB
Production fan-outs are frequently hitting the 6 KiB per-child cap by
just 1-2 KiB, forcing the coordinator into a follow-up inspect_workstream
round-trip per truncated child to recover the tail. Bumping the cap to
10 KiB absorbs the common overshoot without changing the truncation
semantics — truncated=True still fires for genuinely oversized messages,
and inspect_workstream remains the unbounded follow-up.

Worst-case context impact: a 32-child fan-out at the cap is now ~320 KiB
(was ~192 KiB), still well within commercial model context windows.
Typical fan-outs of 1-5 children land at 10-50 KiB.

LAST_ERROR_MAX_LEN (1 KiB) is unchanged — it's intentionally smaller
than the wait cap so error truncation happens at write time, and
1 KiB still sits well below 10 KiB.

WAIT_MESSAGE_MAX_BYTES is referenced by name (not literal 6144) in the
truncation test, so no test value needs updating.
2026-04-29 20:20:38 -07:00
Patrick Buckley 25fe4e728a fix(coord): make coordinator fan out independent work by default
The coordinator system message was descriptive about parallelism rather
than prescriptive — "while multiple children run in parallel" framed
fan-out as incidental, and "a tasks entry, a child to own it" primed
singular delegation. The spawn_batch example (benchmark A, benchmark B,
prototype the winner) showed dependent work under a fan-out framing,
teaching the wrong shape.

In practice the coordinator failed to decompose enumerable requests
("top stories on HN, Lobsters, /r/programming, …") without explicit
"please fan this out" instructions, on both GPT-5.5 and Claude Opus.

base_coordinator.md
- Replace singular "a tasks entry, a child to own it" with plural
  "enumerate the independent units of work, spawn one child per unit,
  run them in parallel by default. Sequential only when one child's
  output feeds the next."
- Tighten the delegation paragraph.

tools_coordinator.md
- Drop the persona repetition that duplicated base_coordinator.md.
- Drop the prescriptive "## Workflow shape" section (the cost note is
  already in wait_for_workstream's tool description; the edit-X
  redirect is already in the persona).
- Drop "in one approval" / "single approval" mentions to avoid
  surfacing approval mechanics to the model.
- Replace the misleading spawn_batch example with truly independent
  items; drop "(up to 10)" which overstated the cap (it's per-call,
  not global, and is documented in the tool schema).
- Add a course-correction example to send_to_workstream — the pattern
  coordinators most often replace with cancel-and-respawn.
- Drop the read action from the tasks examples to keep the lifecycle
  (add → update → remove) coherent.

Coord system message ~16% shorter (4440 → 3722 chars). Both GPT-5.5
and Claude Opus now naturally decompose the news-board prompt without
explicit fan-out instructions. 29 prompt-composition tests pass.
2026-04-29 20:20:38 -07:00
Robert DeAngelis 0d1a32ff65 fix(server): accept --skip-permissions CLI flag (#450)
The server's --help epilog and compose.yaml both reference
--skip-permissions, but the argparser never defined it, so any
container started with SKIP_PERMISSIONS=1 exited with
"unrecognized arguments: --skip-permissions".

Wire the flag through to app.state.skip_permissions, OR-ing it
with the existing tools.skip_permissions config-store setting so
the stored value still works on its own.
2026-04-29 20:20:38 -07:00
291 changed files with 11627 additions and 68724 deletions
+1 -11
View File
@@ -156,17 +156,7 @@ jobs:
- run: uv sync --frozen --all-extras
- run: uv pip install pip-audit
- name: Security audit (dependencies)
# PYSEC-2025-183 (pyjwt): "weak encryption" — disputed by the
# supplier because the key length is chosen by the calling
# application, not the library. Turnstone generates its JWT
# signing keys via the standard ``secrets`` module at
# operator-controlled strength (see ``turnstone/core/auth.py``),
# so the advisory does not apply. pyjwt 2.12.1 is the current
# latest release; no fix version exists.
run: >-
uv export --no-emit-project --frozen
| uv run pip-audit --strict --desc -r /dev/stdin
--ignore-vuln PYSEC-2025-183
run: uv export --no-emit-project --frozen | uv run pip-audit --strict --desc -r /dev/stdin
security-ts:
runs-on: ubuntu-latest
+3 -643
View File
@@ -8,653 +8,13 @@ version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
Three release tracks are maintained:
- **`stable/1.0`** — patch-only (`v1.0.x`)
- **`stable/1.3`** — patch-only (`v1.3.x`)
- **`stable/1.4`** — patch-only (`v1.4.x`)
- **`stable/1.5`** — patch-only (`v1.5.x`)
- **`main`** — experimental (next major)
- **`main`** — experimental (`v1.5.0aN`)
## [Unreleased]
### Added
- **`turnstone-admin` reads `config.toml`** — the admin CLI now honors
the same `[database]` section that `turnstone-server` does, with the
same precedence (`CLI / config.toml > TURNSTONE_DB_* env > defaults`).
Operators with DB credentials in `config.toml` no longer need to
re-export `TURNSTONE_DB_URL` before every admin invocation. Newly
plumbed through to `init_storage`: `pool_size`, `sslmode`,
`sslrootcert`, `sslcert`, `sslkey` — previously the admin CLI
silently dropped these. A new `--config PATH` flag mirrors the
one already on `turnstone-server`.
### Security
- **Permissive `config.toml` now warns** — `turnstone.core.config.load_config`
logs a single warning when the resolved config file is group- or
world-readable (any bit in `0o077`). DB password and TLS key paths
live in `[database]`; operators usually want the file at `0600`.
## [1.5.17]
Backports a clutch of coordinator-tool clarity fixes plus a watch-delivery
correctness fix from `main` to the `stable/1.5` track, plus a previously-
latent intent-verdicts persistence bug exposed by the new heuristic-verdict
INSERT paths. No schema changes.
### Fixed
- **`intent_verdicts` PK collisions on every llm_fallback delivery** —
async LLM-tier "llm_fallback" verdicts (`turnstone/core/judge.py`
`_deliver_fallbacks` and the in-loop fallback path) deliberately
reuse the heuristic verdict's `verdict_id` so the row gets
"upgraded in place" from `tier="heuristic"``tier="llm_fallback"`
when the LLM judge times out, is cancelled, or returns no content.
The consumer `_persist_intent_verdict` was doing a plain INSERT,
hitting the `intent_verdicts_pkey` constraint on every fallback
delivery; Postgres logged the duplicate-key error, the application
try/except swallowed it at `log.debug`, and the row never actually
got upgraded — the LLM judge's annotation
(`"(LLM judge did not return a verdict)"`) was lost. The collision
rate exploded on this release because the new heuristic-INSERT
paths in the auto-approve early-return branches of `approve_tools`
(introduced below) leave no gap for the fallback to land cleanly
into. Fix: new `upsert_intent_verdict` storage method using
`ON CONFLICT (verdict_id) DO UPDATE` that updates only `tier`,
`reasoning`, `judge_model` — the three fields that genuinely
change between heuristic and llm_fallback. Every other column
(identity, carried-verbatim, and `user_decision`) is excluded;
`user_decision` in particular would otherwise be clobbered back
to `"pending"` when a fallback arrives after the operator has
already resolved the approval. The bulk-INSERT path stays as
plain INSERT — fresh UUIDs in `judge.evaluate` make in-turn dups
impossible; the inverse race (fallback wins before bulk lands) is
reachable but unchanged in observable behavior by this fix,
documented at the bulk site for a future hardening pass.
- **Coordinator LLM re-spawn loops on large fan-outs** — the spawn-tool
return JSON used `ws_id` as its key, which primed the model's recency
bias to feed the spawn result straight back into another
`spawn_workstream(ws_id=...)` call instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this cascaded
into self-inflicted re-spawn loops. The LLM-facing tool result now emits
`child_ws_id` (the storage column / HTTP API contract is unchanged); the
field name is already an existing project term so the rename aligns
rather than introduces new vocabulary. Also handles the silent
upstream-omits-ws_id success-shape edge that previously emitted
`{"child_ws_id": null}` to the LLM — now surfaces a tool error so the
model retries rather than chasing a null id.
- **`inspect_workstream` blowing the coordinator context budget** — a
coord doing a fan-out wave against tool-heavy children could land
>100 KB of raw output per inspect call, and the previous safety net
(`_truncate_output`'s head+tail strategy) silently dropped *middle*
messages — exactly the wrong shape for understanding a child's
trajectory (the FIRST sets the brief, the LAST shows the conclusion,
the middle is the connective tissue). Output now goes through a
three-tier degradation ladder mirroring the search tool's
`_format_search_results`: `_tier="full"` (every message verbatim) →
`_tier="compact"` (per-message head/tail-snipped content + snipped
`tool_calls.arguments`, falling through a `(20,30)` / `(10,20)` /
`(5,10)` message-list trim ladder) → `_tier="skeleton"` (counts, role
distribution, last-assistant preview). Budget 32 KiB matches the
search tool's; the chosen tier is annotated on the response so the
model can recall with a tighter `message_limit` if signal was lost.
- **Auto-approved verdicts indistinguishable from pending review** —
`intent_verdict` rows for auto-approved tool calls landed with
`user_decision=""`, which read identically to "still waiting for the
operator" in the audit trail and led to a real misdiagnosis incident.
The column now carries an explicit vocabulary at insert: `pending` /
`approved` / `denied` / `timeout` / `policy` / `blanket` / `skill` /
`always` / `auto_approve_tools`. The auto-approve early-return
branches in `approve_tools` now persist heuristic verdicts stamped
with their reason (previously dropped on the floor), and late LLM-tier
verdicts that arrive for an already-auto-approved call_id are stamped
via a TTL-pruned lookup map — so the audit row carries the
auto-approve reason even when the LLM judge daemon completes after
the synchronous approval cycle finished. `resolve_approval` gains a
`timeout` kwarg writing `"timeout"` (the previous shape collapsed
passive timeouts and active denials into the same column).
- **`list_skills` empty `allowed_tools` misread as "no tool access"** —
the response previously emitted `"allowed_tools": []` for every skill
that hadn't declared an auto-approve allowlist, which a coordinator
model read as "this skill can't use any tools" (real misdiagnosis: a
code-review child appeared to have been spawned with zero tool
access). The field is now omitted entirely when empty — absence
carries the unambiguous meaning "no tool is pre-approved for this
skill", presence (non-empty list) keeps the standard Claude Code
skill-spec shape. The tool description rewrite makes the
auto-approve-allowlist semantics explicit so a future reader doesn't
re-derive the gating misread.
- **Watch terminal-fires silently dropped on backpressure** —
delivery now routes terminal events through the same path as
normal fires instead of being filtered out when the consumer was
saturated.
### Documentation
- **Storage `LIKE_ESCAPE` contract** — clarify that callers passing
`.like(escape=...)` must use the same escape character that the
storage helper assumes; previous wording let a reader pass a
different escape and silently produce no matches.
## [1.5.15]
### Fixed
- **Admin console blank-page on MCP server rows with consented users** — a
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
delimiters on the bulk-revoke button HTML literal, but the literal embeds
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
into bare tokens, and the resulting `SyntaxError` wiped out every global
in `admin.js``showAdmin` and all other admin entry points became
undefined, so the console UI was non-functional whenever the rendered MCP
server list contained at least one row with `consented_users_count > 0`.
Switch the literal to single-quote delimiters to match the surrounding
block.
## [1.5.14]
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
### Added
- **OAuth-MCP Phase 9 — admin status, deferred-consent persistence, operator
docs** — completes the per-(user, server) OAuth-MCP build-out. The sync pool
dispatchers now upsert into a new `mcp_pending_consent` table on
`mcp_consent_required` / `mcp_insufficient_scope`, so a non-interactive run
(scheduled / channel) that hits an unconsented server surfaces the deferred
prompt to the user on their next dashboard load via the gear-icon badge —
rows are cleared automatically by the OAuth callback handler on consent
completion, or via new DELETE endpoints for manual dismiss. The MCP Servers
admin row gains a `consented_users_count` pill and a two-step-confirm
bulk-revoke button for `auth_type=oauth_user` servers (upstream RFC 7009
revoke is intentionally not attempted in bulk to avoid N synchronous
round-trips against the provider). Operator-facing docs land at
`docs/mcp-oauth.md` and `docs/operations/mcp-oauth-headless.md`.
Introduces forward-only migrations `054_mcp_pending_consent` and
`055_mcp_user_tokens_server_index`.
## [1.5.13]
This release introduces one forward-only schema migration:
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
the dispatcher uses in-process fan-out).
### Added
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
per-channel handlers on a separate dispatch thread. The cluster collector
subscribes to a new `services` channel and reacts to node register /
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
loop; the 60 s loop is retained as the backstop for crash-shaped loss
(NOTIFY only fires on real writes). The storage layer also gains a uniform
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
`[database] listen_url` in `config.toml`) points the dispatcher at a
direct-to-Postgres URL; defaults to the main DB URL when unset.
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
whenever a child state change is dispatched to the UI, and the wait loop
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
storage ~2400 times now wakes only on real state transitions, with ~4× lower
SSE traffic in the quiescent case.
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
route previously emitted only `memory.delete`, so tool-initiated mutations
had no audit footprint). All emissions are best-effort and never break the
tool call itself.
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
an optional `skill=<name>` argument that loads the named skill's content as
the sub-agent's persona in place of the hardcoded identity statement. The
fixed operating-guidance block (one-shot, tool-use over narration,
no follow-up questions) is still layered on top of every persona. High- and
critical-risk skills surface their risk tier in the approval header and
emit a `task_agent.high_risk_skill` warning, matching the existing
session-load gate.
### Fixed
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
back-compat `default` alias auto-synthesised by `load_model_registry`
remained visible to the model even when an operator had configured
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
routed to whichever backend the synthesised alias was attached to at boot
instead of the configured per-role default. The synthesised alias is now
only added when neither the DB nor `[models.*]` populates the registry,
filtered out of the LLM-visible alias list, and explicitly rejected at the
validator chokepoint as defense-in-depth.
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
node labels were re-entering the shape parser and producing
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
affected label forms (`|content|` and `ID[content]`) before the SVG cache
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
highlights code blocks progressively as they stream rather than only after
completion.
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
POST went to `/v1/api/auth/login` and was rewritten to
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
public path, and `proxy_api` would have forwarded the login request to the
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
handlers. The dispatch table is a single `(method, path) → handler` mapping
so the test parametrize list can't drift from the implementation.
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
overlay was covering the entire appbar, hiding the proxy-injected node
picker. The overlay now starts at `top: 48px` and the dashboard's role
downgrades from `dialog+aria-modal` to `region` so the appbar above it
remains reachable. The gear icon converts from a direct settings-panel
click into a dropdown with "MCP connections" and "Logout" (the latter with
`.destructive` styling). The settings-menu keydown handler is now attached
synchronously so `Escape` can't fall through the brief window between the
menu opening and its listeners being installed.
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
`services_notify` trigger lives only in the alembic chain, but the test
fixture creates tables via `metadata.create_all`. The trigger function +
trigger are now declared in `_schema.py` and attached via
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
PostgreSQL dialect, with the same SQL constants imported by migration 053
so there's a single source of truth.
## [1.5.12]
### Added
- **Enriched backend error messages** — provider name and attempted URL are now
included in session error responses, so operators can triage connectivity
failures without enabling debug logging.
### Fixed
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
had no messages remaining after a rewind the history event was skipped,
leaving connected UIs with stale content and blocking edit-and-resend flows.
## [1.5.11]
This release introduces one forward-only schema migration:
`052_model_reasoning_persistence``surface_persisted_reasoning` and
`replay_reasoning_to_model` flag columns on `model_definitions`.
### Added
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
restore) now receive an `in_progress_snapshot` event carrying the buffered
partial response, so the UI can resume rendering the in-flight turn without
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
counter so a reconnecting client can skip events it already saw.
- **Reasoning persistence** (Phases 14) — model reasoning text can now be
persisted to conversation history and optionally replayed to the model on
subsequent turns. Phase 1 persists reasoning text on the history payload.
Phase 2 wires a build-time shape filter and a per-model
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
new per-model capability flags (`surface_persisted_reasoning`,
`replay_reasoning_to_model`) both default `False` on unknown and
local-server models.
- **Console home composer: placeholders + toggle** — the console landing-page
composer now shows context-aware placeholder text and a toggle component for
advanced options; an admin polish pass tightened spacing and focus behaviour
across the form.
### Changed
- **`judge.model` now requires a named alias** — raw provider model IDs on
`judge.model` in config are no longer accepted; the judge must reference an
alias registered in the model registry. The session-provider raw-model
fallback is removed. Existing configs using an unregistered model ID need a
corresponding alias entry.
### Fixed
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
flag for a model that does not declare reasoning-replay support now silently
no-ops instead of forwarding reasoning blocks and triggering a provider error.
- **Coordinator alias resolution unified across placeholder + factory** — a
placeholder coordinator and the real coordinator factory could previously
resolve to different model aliases, producing a visible mismatch in the model
display. Both paths now share the same resolution logic.
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
under-initialised coordinator state no longer 500s when the models endpoint
is hit before the coordinator subsystem is fully bootstrapped.
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
counter that broke `state_change` / `in_progress_snapshot` ordering on
reconnect.
- **Reasoning persistence shape + replay fixes** — per-block
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
synthesised alongside non-reasoning `provider_blocks` so both appear
together in the history payload.
## [1.5.10]
This release introduces one forward-only schema migration:
`051_skill_notify_on_complete_array_default` — backfills
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
### Added
- **Skills unlock action** — operators can unlock an installed skill to allow
local customisation. Once unlocked, the skill's resource content, system
prompt additions, and notify configuration are editable through the admin UI.
Skills shipped as part of a bundle remain locked (read-only) until explicitly
unlocked; the unlock is logged to the audit trail. A lock icon in the
top-right of the Skills detail pane doubles as the unlock trigger.
### Fixed
- **`skills.sh` install endpoint** — the install script was targeting an
endpoint removed in an earlier refactor; switched to `/api/download`.
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
(object) instead of `[]` (array), causing notify configurations to be
rejected at schema validation.
- **Skills admin UI modal errors** — `.is-visible` class used consistently
instead of inline `style.display`; stale error text is cleared on submit;
designer-review lock-icon UX applied.
## [1.5.9]
### Fixed
- **`repair=False` on all display-read `load_messages` call sites** —
passing `repair=True` on display paths was silently mutating the stored
message list, causing divergence between what the UI showed and what the
model received on the next turn.
## [1.5.8]
This release introduces two forward-only schema migrations:
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
`050_conversations_source_and_reminders``_source` and `_reminders` columns
on `conversations`.
### Added
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
configured with a client ID and secret through the admin UI. The full token
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
stored encrypted at rest using a key derived from the JWT secret. The consent
flow runs in-browser via a provider redirect. Rolled out in phases:
- Minimum admin form and OAuth schema (`21663d15`).
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
- SDK 401/403 introspection via httpx response hook (`bde09134`).
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
their OAuth token is permitted to call (`cfc8a6c8`).
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
use of an OAuth-gated server and can revoke consent from their profile;
admins see per-server consent counts in the MCP Servers tab (`61051339`).
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
that delivers at most one nudge per turn, preventing multi-channel pile-ups
that inflate context. Observable changes:
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
through to the system message so the model can reason about recency.
- Coordinator idle-children observer: a coordinator with no in-flight
children for longer than the configured idle threshold receives a nudge.
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
rather than the previous `_watch_pending` list, giving them the same
delivery guarantees and priority handling as other advisories.
- **Structured watch-result card** — the UI renders watch results as a styled
card with a system-nudge marker, distinct from the assistant message body.
On history replay, system-nudge turns are visually distinguished from normal
assistant turns.
- **Side-channel persistence** — `_source` and `_reminders` side-channel
fields are persisted to the `conversations` storage table and restored on
session resume, so metacognitive context survives process restarts. A
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
### Fixed
- **Replay consistency** — queued user messages captured mid-loop are now
persisted and replayed in the correct order on a subsequent `events`
subscription. Coordinator history replay fixed: blank assistant cards and
out-of-order tool results on the coordinator tree no longer occur when the
coordinator has mixed queued + delivered messages.
- **Session reminder preservation on fork + resume** — `_source` and
`_reminders` are carried through workstream fork and restored from storage
on resume.
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
columns; `_source` and `_reminders` now strip NUL bytes on write.
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
now committed atomically on first model add; startup teardown is offloaded
to avoid blocking the event loop.
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
now replaces the pool entry with a fresh session; the carrier token is
owned by the pool entry to prevent a race between the 401 handler and a
concurrent request.
- **OIDC hardening** — multiple security and correctness fixes:
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
callback robustness — typed exceptions, shape checks, log sanitization, JS
race (bug-46, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
stranded-user self-heal on role-mapping failure (cumulative bug-1).
## [1.5.7]
### Added
- **Inline node picker** — a compact node-switcher dropdown in the console
header replaces the "← Back to console" banner, so operators can switch
between nodes without a full navigation.
### Fixed
- **Queued user messages injected mid-loop** — messages queued while a
generation was in progress were not being delivered at the correct seam and
could be dropped or reordered when the worker consumed the queue.
- **Search tool output bounded** — pathological inputs (very long lines with
no whitespace) could produce search results exceeding the context budget.
Output is now clamped before reaching the message.
## [1.5.6]
### Added
- **`api_surface` toggle** — model definitions gain an `api_surface` field
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
the provider client uses. Enables Mistral Medium reasoning via the Responses
surface; Chat Completions remains the default for all other models.
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
includes a `healthy_aliases` list per node, so the coordinator and operators
can see which model aliases are currently reachable without a separate
per-model health probe.
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
configure per-kind reasoning effort and alias overrides from the UI rather
than editing `config.toml`. Live-refresh dropdowns update in place when
model definitions change.
### Fixed
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
query-aware candidate-set selection, dramatically improving recall for
queries whose terms span multiple stored entries.
- **Workstream model + config preserved on rehydrate** — reopening a closed
workstream no longer overwrites the model alias and per-workstream config
with session defaults.
- **Console home composer: attachments + user-message pills** — multipart
attachments in the home composer were not forwarded correctly; user-message
pills in the coordinator chat pane were missing.
## [1.5.5]
### Fixed
- **Saved-workstream tool result rendering** — tool results in closed
workstreams were not rendering on history replay. Audit-trail decoration for
tool calls is now applied on the replay path.
## [1.5.4]
### Added
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
first-class citizens in the cluster event bus. `child_ws_state` events are
pushed through the cluster SSE stream so the console tree view updates in
real time without polling. `list_children` and `get_child` primitives on
`SessionManager` provide a consistent cross-node view of the coordinator's
spawn tree.
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
in the console admin panel now supports checkbox multi-select with a
bulk-delete action.
## [1.5.3]
This release introduces one forward-only schema migration:
`048_workstream_reaper_index` — partial composite index on `workstreams` for
the orphan-reaper query.
### Fixed
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
`close_idle` pass now scopes the DB-orphan reaper by
`services.last_heartbeat` so workstreams belonging to a live node are not
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
storage primitives added; a partial composite index keeps the reaper scan
cheap.
- **Coordinator pool idle cleanup** — a periodic task on the console now
closes coordinator pool entries whose session has gone idle past the
configurable threshold, preventing pool exhaustion on long-running consoles.
## [1.5.2]
### Added
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
nudges are rendered as a distinct styled bubble rather than being injected
inline into the assistant message, making it easier to distinguish model
output from metacognitive annotations. The CLI REPL gains matching
`on_user_reminder` / `on_tool_reminder` callbacks.
### Fixed
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
now fires correctly on the third repetition; a write-success-clear that
reset the counter after a successful tool call (preventing streaks across
mixed-outcome sequences) was removed.
- **Metacog reminders isolated to side-channel** — reminder text no longer
appears in the user content turn; it flows through a dedicated side-channel
the session injects into the system context, preventing the model from
attributing it to the user.
## [1.5.1]
### Added
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
now receive the child's pending approval detail in `child_ws_state` events,
enabling the coordinator to surface approval prompts without a separate poll.
### Fixed
- **Coordinator registry auto-refresh** — the console coordinator registry now
refreshes when model definitions change, so a newly added alias is visible
to coordinators without restarting.
- **Coordinator fan-out default** — coordinators now fan out to independent
child workstreams by default instead of serialising them, matching the
documented contract for parallel-work patterns.
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
summaries and tool results from child workstreams were silently truncated at
the previous 4 KiB cap.
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
polling now runs on a dedicated 200-thread executor, matching interactive's
`sse_executor`, so coordinator long-poll blocking no longer contends with
storage and routing workers on the default pool.
## [1.5.0]
User-visible additions: a unified workstream HTTP surface (interactive and
coordinator under one URL family), inline child approvals, coordinator
composer parity, progressive rendering, OIDC authentication, MCP OAuth
foundations, and a redesigned UI built on the Design System v1 token layer.
This release removes the pre-1.5 body-keyed and query-keyed URL family.
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
This release introduces the following forward-only schema migrations that the
server applies automatically on first startup. All are additive; no data loss.
- `039_workstream_kind``kind` + `parent_ws_id` columns on `workstreams`.
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
`admin.cluster.inspect` to the builtin-admin role.
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
introduced by 039.
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
builtin-admin.
- `043_skill_description_required` — backfills empty `description` rows in
`prompt_templates`.
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
(`interactive` / `coordinator` / `any`).
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status`
`risk_level`.
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
by rendezvous routing in 1.4.
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
rows removed from the coordinator in 1.5.0a4.
### Added
- **Inline child approvals** — pending tool approvals on coordinator child
workstreams surface directly in the coordinator tree view. A risk pill shows
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
buttons appear inline so operators do not need to navigate to the child's
workstream. `pending_approval_detail` is exposed on
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
so all connected clients render approval prompts simultaneously. LLM judge
verdicts are cached client-side and replayed on SSE reconnect.
- **Coordinator composer parity** — the coordinator composer now supports
Stop, Send-to-queue, and Attach (file upload), matching the interactive
workstream composer feature set.
- **Per-call model and judge override on coordinator composer** — operators
can override the model alias and judge model for a single coordinator send
from the composer, without changing the node-wide or role-wide defaults. Bad
aliases return a corrective error listing available choices.
- **Coordinator status bar + richer history replay** — each coordinator
workstream gains a per-coordinator status bar showing active children, token
spend, and generation state. History replay in the coordinator panel is
extended to include tool results and thinking blocks.
- **Coordinator child error surfacing + memory tool** — child workstream
errors are surfaced as distinct error rows in the coordinator tree view
rather than disappearing silently. The coordinator gains access to a
`memory` tool (same interface as interactive) for retrieving stored facts.
- **Coordinator inline tool-batch construct** — the coordinator tool approval
UI replaces the separate approval dock with an inline batch construct that
groups all pending tool calls for a given turn into a single review card.
- **Node capability auto-detection** — nodes report kernel-level capabilities
(available memory, CPU count, accelerator presence) via
`/v1/api/node/capabilities` at startup, enabling the console to filter model
aliases offered to coordinators routing to that node.
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
a `SKILL.md` file's content into the modal auto-populates the name,
description, and configuration fields.
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
soon as a complete diagram block is detected in the stream rather than
waiting for the full response; the diagram re-renders in place as the model
extends it.
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
math delimiters are now recognised alongside the existing `$$` fences.
### Removed (BREAKING — 1.5.0)
- **Legacy body-keyed and query-keyed URL family for the workstream
+1 -1
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.14 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
+1 -46
View File
@@ -281,7 +281,6 @@ Each message in the `messages` array has:
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
Each entry in `tool_calls`:
@@ -326,44 +325,6 @@ finalize any in-progress assistant message.
{"type": "stream_end"}
```
**`state_change`** -- the worker thread transitioned to a new state. Drives
the client's busy-mode (composer in send vs. stop, spinner indicators,
auto-focus on idle). Sent live during normal operation AND on every fresh
SSE subscribe (so a mid-stream page refresh restores the correct composer
state without waiting for the next live transition).
```json
{"type": "state_change", "state": "running"}
```
| Field | Type | Description |
|----------|--------|----------------------------------------------------------------------|
| `state` | string | One of `"running"`, `"thinking"`, `"attention"`, `"idle"`, `"error"` |
**`in_progress_snapshot`** -- one-shot replay of the in-progress turn's
content + reasoning text-so-far when this client connects mid-stream.
Lets a refreshing browser tab restore partial assistant text immediately
instead of waiting for the response to complete. Yielded once after the
kind-specific replay phase (history + pending), only when at least one
of `content` / `reasoning` is non-empty. Both halves render into the same
assistant bubble the live `content` / `reasoning` events would target;
clients should treat the snapshot as idempotent (skip overwrite if the
current local buffer is already a superset prefix — covers EventSource
auto-reconnect re-replays).
```json
{
"type": "in_progress_snapshot",
"content": "Here is the answer so far: it depends on ",
"reasoning": "The user is asking about a comparison; let me think about..."
}
```
| Field | Type | Description |
|--------------|--------|------------------------------------------------------------|
| `content` | string | Joined assistant content text accumulated this turn |
| `reasoning` | string | Joined reasoning / chain-of-thought text accumulated |
**`tool_info`** -- one or more tool calls that were auto-approved (no user
action required).
@@ -561,13 +522,7 @@ Each SSE connection to a workstream receives its own delivery queue. Events
produced by the worker thread are fanned out to all registered listener queues,
so multiple consumers (browser, console proxy, SDK) can connect
simultaneously and each receives every event. On reconnect the client receives
the kind-specific replay (`connected` + `status` + `history` + pending
approval / plan for interactive; `connected` + `status` + pending for coord)
followed by a `state_change` carrying the current worker state and an
optional `in_progress_snapshot` carrying any partial content / reasoning
buffered for the in-progress turn — so a mid-stream refresh restores both
the busy-mode UI and the partial assistant text without waiting for the
response to complete.
a full history replay, so no catch-up mechanism is needed.
---
+12 -51
View File
@@ -91,7 +91,7 @@ turnstone/
discord/ Discord adapter (bot, cog, views, streaming, config)
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
katex-0.16.47/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
ui/
colors.py ANSI color constants with NO_COLOR support
markdown.py Streaming terminal markdown renderer (line-buffered)
@@ -231,13 +231,11 @@ The engine emits state changes via `_emit_state()` which calls
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 16
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
methods. Every frontend must implement all of them.
```python
class SessionUI(Protocol):
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -254,14 +252,6 @@ class SessionUI(Protocol):
def on_rename(self, name: str) -> None: ... # propagate alias to tab/UI label
```
`on_turn_start` fires at the top of each iteration of the send-loop;
`on_turn_committed` fires immediately after `messages.append(assistant_msg)`.
`SessionUIBase` uses both to reset the per-turn inflight buffers
(`_ws_inflight_content` / `_ws_inflight_reasoning` / `_ws_inflight_seq`)
that fuel the SSE refresh-resume `in_progress_snapshot` event — see
the per-workstream events stream in
[`docs/api-reference.md`](api-reference.md#get-v1apiworkstreamsws_idevents).
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
### Three Implementations
@@ -556,9 +546,11 @@ adds, removes, or reconnects servers as needed.
6. `_exec_mcp_tool()` calls `call_tool_sync()` which dispatches to the async loop
via `asyncio.run_coroutine_threadsafe()`
**Tool refresh:** Two mechanisms keep tools up-to-date without restart:
**Tool refresh:** Three mechanisms keep tools up-to-date without restart:
- **Push:** Servers declaring `tools.listChanged` send `ToolListChangedNotification`;
the registered `message_handler` triggers immediate single-server refresh.
- **Periodic:** Servers without push support are polled on a staggered interval
(default 4 h, configurable via `[mcp] refresh_interval` or `--mcp-refresh-interval`).
- **Manual:** `/mcp refresh [server]` calls `refresh_sync()` for on-demand refresh
(also attempts reconnection for disconnected servers).
@@ -580,11 +572,10 @@ from a healthy connection do not trip the breaker. When the cooldown expires
(`call_tool_sync`, `read_resource_sync`, `get_prompt_sync`, `refresh_sync`)
cancel orphaned futures on timeout to prevent coroutine accumulation on the
background event loop. Push notification refreshes are debounced (5 s per
server) to protect against notification storms. Operators can force a
catalog refresh or full reconnect from the admin panel; reconnects clear
the circuit breaker and run a fresh handshake. Transport stream references
are pre-closed before stack teardown to work around the MCP SDK's anyio
cancel-scope CPU busy-loop (SDK #2147).
server) to protect against notification storms. The periodic refresh loop
attempts reconnection for disconnected servers with exponential backoff
(60 s1 h). Transport stream references are pre-closed before stack teardown to
work around the MCP SDK's anyio cancel-scope CPU busy-loop (SDK #2147).
**Error isolation:** Per-server connection/refresh failures are caught and logged; other
servers are unaffected. Tool execution errors return error strings to the LLM
@@ -629,15 +620,14 @@ LLMProvider (protocol)
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
| `retryable_error_names` | Exception class names that trigger retry |
| `extract_reasoning_text()` | Walk stored `provider_blocks`, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic `thinking`, OpenAI Responses `reasoning`, OpenAI Chat synthetic `reasoning_text`) |
**Normalized data types:**
| Type | Fields |
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
@@ -725,35 +715,6 @@ and `"openai-compatible"`.
`max_tokens`, and `reasoning_effort` to override the global defaults from
ConfigStore. When unset (`NULL`), the global default is used.
**Per-model reasoning persistence:** Two booleans on `model_definitions`
(migration 052) control how reasoning text round-trips:
* `surface_persisted_reasoning` (default `True`) — gates whether stored
reasoning text is surfaced on `/history` payloads for UI rehydration.
**Storage of reasoning bytes happens regardless of this flag** — they
ride in `provider_data` independently. Phase-1 admin UI label "Surface
persisted reasoning."
* `replay_reasoning_to_model` (default `False`) — gates whether stored
reasoning blocks are sent back to the provider on subsequent turns.
Capability-gated: `ModelCapabilities.supports_reasoning_replay` must
also be `True` for the wire path to actually replay (canonical OpenAI
gpt-5*/o-series and Anthropic Claude entries set it; unknown / local-
server models default to `False`).
Three reasoning paths are recognised:
| Path | Provider | Capture | Persist | Replay |
|------|----------|---------|---------|--------|
| 1 | Anthropic Messages API | `thinking_delta` | `provider_blocks` (`type="thinking"`) | Verbatim via `_provider_content` |
| 2 | OpenAI Responses (gpt-5*, o-series) | `response.reasoning_text.delta` events | `provider_blocks` (`type="reasoning"`) — only when `include=["reasoning.encrypted_content"]` | `ResponseReasoningItemParam` input items |
| 3 | OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat) | `delta.reasoning_content` Pydantic extras | Synthetic `{type: "reasoning_text", text, source}` block stamped at end-of-stream | None — no API surface for replay on Chat Completions |
Cross-provider safety is enforced by `ANTHROPIC_VALID_BLOCK_TYPES` (a
shape filter in `_anthropic.py:_convert_messages`): foreign blocks
(OpenAI `reasoning`, synthetic `reasoning_text`) fall through to the
text+tool_calls rebuild path rather than reaching Anthropic's input
boundary as malformed content.
```toml
[models.local]
base_url = "http://localhost:8000/v1"
+2 -9
View File
@@ -110,18 +110,11 @@ owns it; the node is just currently unreachable.
### Example — `spawn_batch`
This is the coordinator-tool result shape (the JSON the LLM receives),
not an HTTP API response — the table above keys it under "model tool"
to distinguish it from the `/v1/api/...` endpoints in the same table.
The underlying HTTP spawn endpoint still returns `ws_id`; the tool
result re-keys it to `child_ws_id` to defuse a coordinator-LLM recency
bias (see `docs/coordinator-skills.md`).
```json
{
"results": {
"0": {"child_ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"child_ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
"0": {"ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
},
"denied": [
{"idx": 1, "reason": "skill not found: nonexistent-skill"}
+4 -9
View File
@@ -115,8 +115,7 @@ with a `type` field. The recurring shapes a UI has to handle:
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `state_change` | Worker-thread state transition | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `rename` | Session's display name changed | `name` |
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
@@ -131,13 +130,9 @@ with a `type` field. The recurring shapes a UI has to handle:
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
is re-sent if unresolved) and any in-flight `wait_*` / `batch_*`
indicator — so a tab refresh mid-approval doesn't strand the
operator.
---
+5 -12
View File
@@ -169,21 +169,14 @@ validates ws_id against `parent_ws_id=coord_ws_id` AND
the wait into reporting "complete".
Pattern: capture each spawn result in the next tool call's input.
The JSON tool-result carries `{"child_ws_id": "...", "name": "...",
The JSON tool-result carries `{"ws_id": "...", "name": "...",
"node_id": "...", "routing_strategy": "..."}`; the model should
extract the `child_ws_id` and pass it as `ws_id` (or in the `ws_ids`
list) to `inspect_workstream` / `wait_for_workstream` /
`send_to_workstream` / `close_workstream` verbatim. The asymmetry
— spawn returns `child_ws_id` but the other tools accept `ws_id` /
`ws_ids` — is intentional: it defuses a coordinator-LLM recency
bias where seeing `ws_id` in a spawn return primed re-spawn loops
instead of progression to the wait phase.
extract the ws_id and pass it to `inspect_workstream` /
`wait_for_workstream` / `send_to_workstream` / `close_workstream`
verbatim.
A UI that wants human-readable identifiers should render the `name`
field and keep the workstream id as the click-through key — note
that the id *value* is the same regardless of whether it arrived
under the `child_ws_id` key (spawn return) or the `ws_id` key
(every other tool's input/output); only the field name differs.
field and keep the ws_id as the click-through key.
---
+1 -1
View File
@@ -40,7 +40,7 @@ package "turnstone/core/" <<Rectangle>> {
component [auth.py\nAuthentication] as auth <<core>>
component [healthcheck.py\nBackendHealthMonitor] as healthcheck <<core>>
component [ratelimit.py\nRateLimiter] as ratelimit <<core>>
component [mcp_client.py\nMCPClientManager\n(push + manual refresh)] as mcp <<core>>
component [mcp_client.py\nMCPClientManager\n(push + periodic refresh)] as mcp <<core>>
component [tool_search.py\nToolSearchManager, BM25] as toolsearch <<core>>
component [model_registry.py\nModelRegistry] as registry <<core>>
}
+3 -5
View File
@@ -69,10 +69,9 @@ class "NullUI" as NullUI {
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
+ create_streaming(client, model, messages, ...) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ...) → CompletionResult
+ convert_tools(tools) → list[dict]
+ extract_reasoning_text(provider_blocks) → str
+ retryable_error_names: frozenset[str] {property}
--
core/providers/_protocol.py
@@ -127,7 +126,6 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_web_search: bool
+ supports_tool_search: bool
+ supports_vision: bool
+ supports_reasoning_replay: bool
}
' ChatSession
@@ -255,7 +253,7 @@ class "MCPClientManager" as MCPMgr {
Background asyncio event loop
bridges async MCP SDK to
sync ChatSession dispatch.
Push + manual refresh.
Push + periodic + manual refresh.
Resources + prompts discovered
alongside tools at startup.
--
-16
View File
@@ -24,14 +24,6 @@ CS -> DB : save_message(ws_id, "user", input)
group loop [while tool_calls present]
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
@@ -81,14 +73,6 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
+11 -15
View File
@@ -190,25 +190,21 @@ group Push Notifications (debounced 5s per server)
MCPMgr -> Storage : sync_prompts_to_storage()
end
group Manual Refresh
Session -> MCPMgr : refresh_sync()
group Periodic Polling (default 4h)
MCPMgr -> MCPMgr : _periodic_refresh()
note right
/mcp refresh [server] —
re-fetches catalog and
attempts reconnect for
disconnected servers.
Only polls capabilities
without push support.
Staggered per-server.
Disconnected servers get
reconnect attempts with
exponential backoff (60s-1h).
end note
end
group Manual Reconnect
Session -> MCPMgr : reconnect_sync(name)
note right
Operator-driven via the
console admin panel —
tears down session, clears
circuit breaker, runs a
fresh handshake.
end note
group Manual Refresh
Session -> MCPMgr : refresh_sync()
note right: /mcp refresh [server]
end
== Policy Evaluation ==
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:95dd5ebc899a1261d516686a5aa3319a7f45015d411302825fa28afbfc82e1ce
size 326766
oid sha256:a3b5c59403a6febd81667fc8fd2a7d22bc59da6130eba0dea5449c42668d0ede
size 387044
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
oid sha256:474b900448ec04d1117b48a2b55614524721b2f04ac4bda66170bd0a06aae0f2
size 624573
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c14dfbb2db8dcb22cd332b2cf0e53ba75141adb213dae47dd9dbfd389ed482fe
size 354799
oid sha256:3aa8d972bba40d78152f9f0c762b9f5ec616d8052c45fa52b7dd1c679ed81d61
size 325245
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d6aff446a062aa08f316985d00c2183148694f786d7f22172bc50b30046c728b
size 379259
oid sha256:7623df33be9baf7647ca1c2450640df57e1cd73e8be1f8168aae16e546ad683c
size 459941
-1
View File
@@ -84,7 +84,6 @@ Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
|----------|---------|-------------|
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_LISTEN_URL` | (falls back to `TURNSTONE_DB_URL`) | Direct-to-PostgreSQL URL for the console's dedicated `LISTEN` connection. Set this when `TURNSTONE_DB_URL` points at PgBouncer in transaction pooling mode — LISTEN is session state and the transaction-pooled connection can't hold it. See [pgbouncer.md](pgbouncer.md). |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
-117
View File
@@ -1,117 +0,0 @@
# MCP OAuth — per-user authorization for MCP servers
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
---
## When to use which `auth_type`
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
| `auth_type` | What it means | When to use |
|---|---|---|
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
## Prerequisites for `auth_type=oauth_user`
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata** *or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
3. **OAuth client registration**. Two paths:
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
---
## Configuration
### Per-server fields (admin UI)
| Field | Required | Description |
|---|---|---|
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
| Multitenant Authorization | Yes | `none` / `static` / `oauth_user` (recommended). |
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
| Client Registration | Yes (oauth_user) | `preregistered` or `dynamic`. |
| Client ID | Yes (preregistered) | OAuth 2.0 client ID. Stored unencrypted. |
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
### Encryption key
```toml
[security]
mcp_token_encryption_key = "base64-fernet-key"
# For rotation, list the keys in priority order — first is used for new
# writes, all are tried for reads.
# mcp_token_encryption_keys = ["new-key", "old-key"]
```
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
---
## Admin status indicators
The MCP Servers admin tab shows per-server status pills (Phase 9):
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
---
## Auth-type transitions
| From | To | What happens |
|---|---|---|
| `none` / `static``oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
+13 -82
View File
@@ -39,19 +39,18 @@ are set.
| `TURNSTONE_OIDC_ROLE_CLAIM` | No | — | ID token claim containing role/group values (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_ROLE_MAP` | No | — | Mapping from claim values to Turnstone role IDs (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
| `TURNSTONE_OIDC_REDIRECT_BASE` | No | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Recommended when running behind a reverse proxy. When unset, derived from the request Host header. |
All four required fields issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
is disabled at startup (an error is logged when only `redirect_base`
is missing) and the login screen shows only the password form.
OIDC is enabled when all three required fields (issuer, client ID, client
secret) are non-empty. If any is missing, OIDC is silently disabled and
the login screen shows only the password form.
### Redirect base (required)
### Reverse Proxy / Load Balancer
`TURNSTONE_OIDC_REDIRECT_BASE` pins the redirect URI sent to the identity
provider to a known externally-visible origin. Set it to the public origin
of your Turnstone deployment:
When Turnstone runs behind a reverse proxy, the internal `Host` header may
not match the externally-reachable URL. Set `TURNSTONE_OIDC_REDIRECT_BASE`
to the public origin so the redirect URI sent to the identity provider is
correct:
```bash
TURNSTONE_OIDC_REDIRECT_BASE=https://app.example.com
@@ -61,44 +60,6 @@ The resulting callback URL will be
`https://app.example.com/v1/api/auth/oidc/callback` — register this as the
authorized redirect URI in your identity provider.
OIDC will refuse to start when this variable is unset. There is no
Host-header fallback: a permissive reverse proxy or direct backend access
would otherwise let an attacker spoof `Host` and steer the IdP redirect
to a callback origin they control.
### Cross-host endpoints
By default, every endpoint in the IdP discovery document
(`token_endpoint`, `jwks_uri`, `userinfo_endpoint`) must share the
issuer's `(scheme, host, port)`. This prevents a hostile or compromised
IdP from redirecting the token-exchange POST (which carries
`client_secret`) to an arbitrary host, and prevents JWKS fetches from
being aimed at internal services.
A few public IdPs legitimately split endpoints across hostnames. Google
is the canonical example:
| Field | Hostname |
|-------|----------|
| issuer | `accounts.google.com` |
| token_endpoint | `oauth2.googleapis.com` |
| jwks_uri | `www.googleapis.com` |
| userinfo_endpoint | `openidconnect.googleapis.com` |
Google's set is built in — operators using `https://accounts.google.com`
need no extra configuration.
For other IdPs whose discovery document references a non-issuer host,
extend the allow-list explicitly:
```bash
TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS=token.example.com,keys.example.com
```
The same scheme / no-userinfo / SSRF rules apply to allow-listed hosts —
this knob only relaxes the same-origin check, not the security gates.
Each entry is a hostname (no scheme, no path).
### config.toml alternative
```toml
@@ -237,19 +198,6 @@ TURNSTONE_OIDC_ROLE_MAP="admin:builtin-admin,engineering:builtin-operator,viewer
the user authenticates via OIDC, so new group memberships are picked
up on the next login.
### `assigned_by` markers
Role assignments record an `assigned_by` value that controls how the
sync logic treats them. OIDC-driven flows use two distinct markers:
- `oidc` — set by claim-driven role mapping; revoked automatically on
the next login when the corresponding claim value is no longer
present.
- `oidc-default` — applied to brand-new OIDC users who have no
claim-mapped roles, as a safety net so they still get
`builtin-viewer` access on first login. Survives subsequent logins
regardless of claim contents and is never revoked by `apply_role_mapping`.
### Built-in Roles
| Role ID | Permissions |
@@ -427,27 +375,10 @@ callback validation. Entries are automatically cleaned up after 5 minutes.
### "OIDC not configured"
All four required environment variables must be set:
`TURNSTONE_OIDC_ISSUER`, `TURNSTONE_OIDC_CLIENT_ID`,
`TURNSTONE_OIDC_CLIENT_SECRET`, and `TURNSTONE_OIDC_REDIRECT_BASE`.
Check that none are empty or whitespace-only.
### "OIDC enabled but TURNSTONE_OIDC_REDIRECT_BASE is unset"
This error is logged when the three credential variables are set but
`TURNSTONE_OIDC_REDIRECT_BASE` is missing. OIDC is disabled at startup
to prevent Host-header-derived redirect URI spoofing. Set the variable
to your service's externally-visible origin (e.g.
`https://app.example.com`) and restart the server. See
[Redirect base](#redirect-base-required) for the rationale.
### Discovery silently disables OIDC with "host does not match issuer"
The IdP discovery document points `token_endpoint`, `jwks_uri`, or
`userinfo_endpoint` at a hostname that doesn't share the issuer's
origin. If the IdP is legitimate, add the additional hostname(s) to
`TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS`. Google is allow-listed
automatically; see [Cross-host endpoints](#cross-host-endpoints).
All three required environment variables must be set:
`TURNSTONE_OIDC_ISSUER`, `TURNSTONE_OIDC_CLIENT_ID`, and
`TURNSTONE_OIDC_CLIENT_SECRET`. Check that none are empty or
whitespace-only.
### "Login session expired"
-29
View File
@@ -1,29 +0,0 @@
# MCP OAuth in headless / scheduled / channel-driven runs
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
**Affected surfaces**:
- Scheduled workstreams (`turnstone-console` task scheduler).
- Discord adapter runs.
- Slack adapter runs.
- Any future channel adapter without an interactive browser session.
**What happens when consent is missing**:
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
**Recovery**:
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
**Pre-consent recipe**:
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
1. Open the dashboard.
2. Open the settings modal (gear icon).
3. Click Connect on each MCP server the schedule will use.
4. Confirm consent in the popup.
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
-24
View File
@@ -199,28 +199,4 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
not use server-side prepared statements by default, so this is not an
issue.
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
transaction pooling assigns a real server connection only for the
duration of each transaction, then returns it to the pool. PostgreSQL
`LISTEN` is session state — a transaction-pooled client can't hold the
multi-statement session a long-lived `LISTEN` needs. The console's
`NotifyDispatcher` (reactive node discovery via the `services` channel)
therefore opens a **dedicated, direct-to-Postgres** connection that
bypasses PgBouncer.
Configure via `config.toml` `[database] listen_url` (preferred —
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
(config.toml wins when both are set). Defaults to the main DB URL when
unset.
| Setting | Behaviour |
|---|---|
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
Set this whenever PgBouncer is in transaction mode (the recommended
setting per this doc). The override only adds one long-lived PG
connection per console process — sized into the cluster's
`max_connections` budget alongside the pool.
See also: [Docker deployment](docker.md) · [Security](security.md)
-3
View File
@@ -138,9 +138,6 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `error` | `ErrorEvent` | `message` |
| `info` | `InfoEvent` | `message` |
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `cancelled` | `CancelledEvent` | — |
**Global events** (from `stream_global_events()`):
+1 -16
View File
@@ -59,21 +59,6 @@ from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Reasoning persistence (per-model)
Two boolean flags on `model_definitions` (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|------|---------|--------|
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
provider-side mechanics (Anthropic `thinking`, OpenAI Responses
`reasoning` + `include=["reasoning.encrypted_content"]`, synthetic
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
### Plan / task agent overrides
`plan_agent` and `task_agent` sub-sessions resolve independently from the
@@ -115,7 +100,7 @@ initialization:
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
| `cluster` | node_fan_out_limit, mcp_max_servers |
| `mcp` | config_path, registry_url |
| `mcp` | config_path, refresh_interval, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
+14 -5
View File
@@ -758,18 +758,22 @@ MCP tools (3):
### Dynamic tool refresh
MCP tool lists stay up-to-date without restart through two mechanisms:
MCP tool lists stay up-to-date without restart through three mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
2. **Periodic timer** -- Servers that do *not* support push notifications are polled
on a configurable interval (default 4 hours). The timer is staggered using a
launch-time seed (`monotonic_ns ^ pid`) so cluster nodes don't all hit MCP
servers simultaneously. Configure via `[mcp] refresh_interval` in `config.toml`
or `--mcp-refresh-interval SECONDS` on the CLI. Set to `0` to disable.
3. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
manual refresh attempts reconnection. The console admin panel exposes the
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
manual refresh attempts reconnection.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
@@ -777,6 +781,11 @@ instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`, `_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```toml
[mcp]
refresh_interval = 14400 # seconds (default 4h), 0 to disable
```
```
/mcp refresh
MCP refresh complete:
+5 -6
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.6.0a2"
version = "1.5.7"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -22,9 +22,9 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
"openai>=2.37",
"openai>=2.24",
"httpx>=0.28",
"mcp>=1.27",
"mcp>=1.6",
"starlette>=0.45",
"uvicorn>=0.34",
"sse-starlette>=2.0",
@@ -35,7 +35,6 @@ dependencies = [
"structlog>=24.1",
"PyJWT>=2.8",
"bcrypt>=4.0",
"cryptography>=42",
"python-frontmatter>=1.0",
]
@@ -82,9 +81,9 @@ include = [
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.16.47/**/*",
"turnstone/shared_static/katex-0.16.45/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.15.0/**/*",
"turnstone/shared_static/mermaid-11.14.0/**/*",
"turnstone/shared_static/hls-1.6.16/**/*",
"turnstone/sdk/py.typed",
"turnstone/deploy/*.yaml",
+3 -6
View File
@@ -50,14 +50,11 @@ update_refs() {
local old_pattern="$1" # e.g. katex-0.16.38
local new_pattern="$2" # e.g. katex-0.16.39
# Find all files with version references. Excludes the old versioned vendor
# directory itself (about to be rm -rf'd anyway) so we don't bother rewriting
# self-references inside it — but does NOT exclude all of shared_static/,
# because shared_static/renderer.js loads the vendored libs and needs the bump.
# Find all files with version references (excludes vendored JS and worktrees)
local files
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' --include='*.py' \
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' \
-F "$old_pattern" . \
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir="$old_pattern" \
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir='shared_static' \
2>/dev/null || true)
for f in $files; do
sed -i "s|${old_pattern}|${new_pattern}|g" "$f"
+127 -127
View File
@@ -74,9 +74,9 @@
}
},
"node_modules/@oxc-project/types": {
"version": "0.130.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.130.0.tgz",
"integrity": "sha512-ibD2usx9JRu7f5pu2tMKMI4cpA4NgXJQoYRP4pQ7Pxmn1l6k/53qWtQWZayhYy3X4QZkt90Ot+mJEaeXouio6Q==",
"version": "0.127.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz",
"integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==",
"dev": true,
"license": "MIT",
"funding": {
@@ -84,9 +84,9 @@
}
},
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.1.tgz",
"integrity": "sha512-fJI3I0r3C3Oj/zdBCpaCmBRZYf07xpaq4yCfDDoSFm+beWNzbIl26puW8RraUdugoJw/95zerNOn6jasAhzSmg==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==",
"cpu": [
"arm64"
],
@@ -101,9 +101,9 @@
}
},
"node_modules/@rolldown/binding-darwin-arm64": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.1.tgz",
"integrity": "sha512-cKnAhWEsV7TPcA/5EAteDp6KcJZBQ2G+BqE7zayMMi7kMvwRsbv7WT9aOnn0WNl4SKEIf43vjS31iUPu80nzXg==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==",
"cpu": [
"arm64"
],
@@ -118,9 +118,9 @@
}
},
"node_modules/@rolldown/binding-darwin-x64": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.1.tgz",
"integrity": "sha512-YKrVwQjIRBPo+5G/u03wGjbdy4q7pyzCe93DK9VJ7zkVmeg8LJ7GbgsiHWdR4xSoe4CAXRD7Bcjgbtr64bkXNg==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz",
"integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==",
"cpu": [
"x64"
],
@@ -135,9 +135,9 @@
}
},
"node_modules/@rolldown/binding-freebsd-x64": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.1.tgz",
"integrity": "sha512-z/oBsREo46SsFqBwYtFe0kpJeBijAT48O/WXLI4suiCLBkr03RTtTJMCzSdDd2znlh8VJizL09XVkQgk8IZonw==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz",
"integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==",
"cpu": [
"x64"
],
@@ -152,9 +152,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.1.tgz",
"integrity": "sha512-ik8q7GM11zxvYxFc2PeDcT6TBvhCQMaUxfph/M5l9sKuTs/Sjg3L+Byw0F7w0ZVLBZmx30P+gG0ECzzN+MFcmQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz",
"integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==",
"cpu": [
"arm"
],
@@ -169,9 +169,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-gnu": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.1.tgz",
"integrity": "sha512-QoSx2EkyrrdZ6kcyE8stqZ62t0Yra8Fs5ia9lOxJrh6TMQJK7gQKmscdTHf7pOXKREKrVwOtJcQG3qVSfc866A==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==",
"cpu": [
"arm64"
],
@@ -189,9 +189,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-musl": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.1.tgz",
"integrity": "sha512-uwNwFpwKeNiZawfAWBgg0VIztPTV3ihhh1vV334h9ivnNLorxnQMU6Fz8wG1Zb4Qh9LC1/MkcyT3YlDXG3Rsgg==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz",
"integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==",
"cpu": [
"arm64"
],
@@ -209,9 +209,9 @@
}
},
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.1.tgz",
"integrity": "sha512-zY1bul7OWr7DFBiJ++wofXvnr8B45ce3QsQUhKrIhXsygAh7bTkwyeM1bi1a2g5C/yC/N8TZyGDEoMfm/l9mpg==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==",
"cpu": [
"ppc64"
],
@@ -229,9 +229,9 @@
}
},
"node_modules/@rolldown/binding-linux-s390x-gnu": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.1.tgz",
"integrity": "sha512-0frlsT/f4Ft6I7SMESTKnF3cZsdicQn1dCMkF/jT9wDLE+gGoiQfv1nmT9e+s7s/fekvvy6tZM2jHvI2tkbJDQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==",
"cpu": [
"s390x"
],
@@ -249,9 +249,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-gnu": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.1.tgz",
"integrity": "sha512-XABVmGp9Tg0WspTVvwduTc4fpqy6JnAUrSQe6OuyqD/03nI7r0O9OWUkMIwFrjKAIqolvqoA4ZrJppgwE0Gxmw==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==",
"cpu": [
"x64"
],
@@ -269,9 +269,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-musl": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.1.tgz",
"integrity": "sha512-bV4fzswuzVcKD90o/VM6QqKxnxlDq0g2BISDLNVmxrnhpv1DDbyPhCIjYfvzYLV+MvkKKnQt2Q6AO86SEBULUQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz",
"integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==",
"cpu": [
"x64"
],
@@ -289,9 +289,9 @@
}
},
"node_modules/@rolldown/binding-openharmony-arm64": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.1.tgz",
"integrity": "sha512-/Mh0Zhq3OP7fVs0kcQHZP6lZEthMGTaSf8UBQYSFEZDWGXXlEC+nJ6EqenaK2t4LBXMe3A+K/G2BVXXdtOr4PQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==",
"cpu": [
"arm64"
],
@@ -306,9 +306,9 @@
}
},
"node_modules/@rolldown/binding-wasm32-wasi": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.1.tgz",
"integrity": "sha512-+1xc9X45l8ufsBAm6Gjvx2qDRIY9lTVt0cgWNcJ+1gdhXvkbxePA60yRTwSTuXL09CMhyJmjpV7E3NoyxbqFQQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz",
"integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==",
"cpu": [
"wasm32"
],
@@ -325,9 +325,9 @@
}
},
"node_modules/@rolldown/binding-win32-arm64-msvc": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.1.tgz",
"integrity": "sha512-1D+UqZdfnuR+Jy1GgMJwi85bD40H21uNmOPRWQhw4oRSuolZ/B5rixZ45DK2KXOTCvmVCecauWgEhbw8bI7tOw==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz",
"integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==",
"cpu": [
"arm64"
],
@@ -342,9 +342,9 @@
}
},
"node_modules/@rolldown/binding-win32-x64-msvc": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.1.tgz",
"integrity": "sha512-INAycaWuhlOK3wk4mRHGsdgwYWmd9cChdPdE9bwWmy6rn9VqVNYNFGhOdXrofXUxwHIncSiPNb8tNm8knDVIeQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz",
"integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==",
"cpu": [
"x64"
],
@@ -359,9 +359,9 @@
}
},
"node_modules/@rolldown/pluginutils": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz",
"integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz",
"integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==",
"dev": true,
"license": "MIT"
},
@@ -402,23 +402,23 @@
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
"integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==",
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
"dev": true,
"license": "MIT"
},
"node_modules/@vitest/expect": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.6.tgz",
"integrity": "sha512-7EHDquPthALSV0jhhjgEW8FXaviMx7rSqu8W6oqCoAuOhKov814P99QDV1pxMA3QPv21YudvJngIhjrNI4opLg==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.5.tgz",
"integrity": "sha512-PWBaRY5JoKuRnHlUHfpV/KohFylaDZTupcXN1H9vYryNLOnitSw60Mw9IAE2r67NbwwzBw/Cc/8q9BK3kIX8Kw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@standard-schema/spec": "^1.1.0",
"@types/chai": "^5.2.2",
"@vitest/spy": "4.1.6",
"@vitest/utils": "4.1.6",
"@vitest/spy": "4.1.5",
"@vitest/utils": "4.1.5",
"chai": "^6.2.2",
"tinyrainbow": "^3.1.0"
},
@@ -427,13 +427,13 @@
}
},
"node_modules/@vitest/mocker": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.6.tgz",
"integrity": "sha512-MCFc63czMjEInOlcY2cpQCvCN+KgbAn+60xu9cMgP4sKaLC5JNAKw7JH8QdAnoAC88hW1IiSNZ+GgVXlN1UcMQ==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.5.tgz",
"integrity": "sha512-/x2EmFC4mT4NNzqvC3fmesuV97w5FC903KPmey4gsnJiMQ3Be1IlDKVaDaG8iqaLFHqJ2FVEkxZk5VmeLjIItw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "4.1.6",
"@vitest/spy": "4.1.5",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.21"
},
@@ -454,9 +454,9 @@
}
},
"node_modules/@vitest/pretty-format": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.6.tgz",
"integrity": "sha512-h5SxD/IzNhZYnrSZRsUZQIC+vD0GY8cUvq0iwsmkFKixRCKLLWqCXa/FIQ4S1R+sI+PGoojkHsdNrbZiM9Qpgw==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.5.tgz",
"integrity": "sha512-7I3q6l5qr03dVfMX2wCo9FxwSJbPdwKjy2uu/YPpU3wfHvIL4QHwVRp57OfGrDFeUJ8/8QdfBKIV12FTtLn00g==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -467,13 +467,13 @@
}
},
"node_modules/@vitest/runner": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.6.tgz",
"integrity": "sha512-nOPCmn2+yD0ZNmKdsXGv/UxMMWbMuKeD6GyYncNwdkYDxpQvrPSKYj2rWuDjC2Y4b6w6hjip5dBKFzEUuZe3vA==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.5.tgz",
"integrity": "sha512-2D+o7Pr82IEO46YPpoA/YU0neeyr6FTerQb5Ro7BUnBuv6NQtT/kmVnczngiMEBhzgqz2UZYl5gArejsyERDSQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "4.1.6",
"@vitest/utils": "4.1.5",
"pathe": "^2.0.3"
},
"funding": {
@@ -481,14 +481,14 @@
}
},
"node_modules/@vitest/snapshot": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.6.tgz",
"integrity": "sha512-YhsdE6xAVfTDmzjxL2ZDUvjj+ZsgyOKe+TdQzqkD72wIOmHka8NuGQ6NpTNZv9D2Z63fbwWKJPeVpEw4EQgYxw==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.5.tgz",
"integrity": "sha512-zypXEt4KH/XgKGPUz4eC2AvErYx0My5hfL8oDb1HzGFpEk1P62bxSohdyOmvz+d9UJwanI68MKwr2EquOaOgMQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.6",
"@vitest/utils": "4.1.6",
"@vitest/pretty-format": "4.1.5",
"@vitest/utils": "4.1.5",
"magic-string": "^0.30.21",
"pathe": "^2.0.3"
},
@@ -497,9 +497,9 @@
}
},
"node_modules/@vitest/spy": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.6.tgz",
"integrity": "sha512-JFKxMx6udhwKh/Ldo270e17QX710vgunMkuPAvXjHSvC6oqLWAHhVhjg/I71q0u0CBSErIODV1Kjv0FQNSWjdg==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.5.tgz",
"integrity": "sha512-2lNOsh6+R2Idnf1TCZqSwYlKN2E/iDlD8sgU59kYVl+OMDmvldO1VDk39smRfpUNwYpNRVn3w4YfuC7KfbBnkQ==",
"dev": true,
"license": "MIT",
"funding": {
@@ -507,13 +507,13 @@
}
},
"node_modules/@vitest/utils": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.6.tgz",
"integrity": "sha512-FxIY+U81R3LGKCxaHHFRQ5+g6/iRgGLmeHWdp2Amj4ljQRrEIWHmZyDfDYBRZlpyqA7qKxtS9DD1dhk8RnRIVQ==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.5.tgz",
"integrity": "sha512-76wdkrmfXfqGjueGgnb45ITPyUi1ycZ4IHgC2bhPDUfWHklY/q3MdLOAB+TF1e6xfl8NxNY0ZYaPCFNWSsw3Ug==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.6",
"@vitest/pretty-format": "4.1.5",
"convert-source-map": "^2.0.0",
"tinyrainbow": "^3.1.0"
},
@@ -959,9 +959,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.14",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
"version": "8.5.13",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.13.tgz",
"integrity": "sha512-qif0+jGGZoLWdHey3UFHHWP0H7Gbmsk8T5VEqyYFbWqPr1XqvLGBbk/sl8V5exGmcYJklJOhOQq1pV9IcsiFag==",
"dev": true,
"funding": [
{
@@ -988,14 +988,14 @@
}
},
"node_modules/rolldown": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.1.tgz",
"integrity": "sha512-X0KQHljNnEkWNqqiz9zJrGunh1B0HgOxLXvnFpCOcadzcy5qohZ3tqMEUg00vncoRovXuK3ZqCT9KnnKzoInFQ==",
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz",
"integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@oxc-project/types": "=0.130.0",
"@rolldown/pluginutils": "^1.0.0"
"@oxc-project/types": "=0.127.0",
"@rolldown/pluginutils": "1.0.0-rc.17"
},
"bin": {
"rolldown": "bin/cli.mjs"
@@ -1004,21 +1004,21 @@
"node": "^20.19.0 || >=22.12.0"
},
"optionalDependencies": {
"@rolldown/binding-android-arm64": "1.0.1",
"@rolldown/binding-darwin-arm64": "1.0.1",
"@rolldown/binding-darwin-x64": "1.0.1",
"@rolldown/binding-freebsd-x64": "1.0.1",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.1",
"@rolldown/binding-linux-arm64-gnu": "1.0.1",
"@rolldown/binding-linux-arm64-musl": "1.0.1",
"@rolldown/binding-linux-ppc64-gnu": "1.0.1",
"@rolldown/binding-linux-s390x-gnu": "1.0.1",
"@rolldown/binding-linux-x64-gnu": "1.0.1",
"@rolldown/binding-linux-x64-musl": "1.0.1",
"@rolldown/binding-openharmony-arm64": "1.0.1",
"@rolldown/binding-wasm32-wasi": "1.0.1",
"@rolldown/binding-win32-arm64-msvc": "1.0.1",
"@rolldown/binding-win32-x64-msvc": "1.0.1"
"@rolldown/binding-android-arm64": "1.0.0-rc.17",
"@rolldown/binding-darwin-arm64": "1.0.0-rc.17",
"@rolldown/binding-darwin-x64": "1.0.0-rc.17",
"@rolldown/binding-freebsd-x64": "1.0.0-rc.17",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17",
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17",
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.17",
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.17",
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.17",
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17",
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17"
}
},
"node_modules/siginfo": {
@@ -1119,16 +1119,16 @@
}
},
"node_modules/vite": {
"version": "8.0.13",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.13.tgz",
"integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==",
"version": "8.0.10",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
"dev": true,
"license": "MIT",
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
"postcss": "^8.5.14",
"rolldown": "1.0.1",
"postcss": "^8.5.10",
"rolldown": "1.0.0-rc.17",
"tinyglobby": "^0.2.16"
},
"bin": {
@@ -1145,7 +1145,7 @@
},
"peerDependencies": {
"@types/node": "^20.19.0 || >=22.12.0",
"@vitejs/devtools": "^0.1.18",
"@vitejs/devtools": "^0.1.0",
"esbuild": "^0.27.0 || ^0.28.0",
"jiti": ">=1.21.0",
"less": "^4.0.0",
@@ -1197,19 +1197,19 @@
}
},
"node_modules/vitest": {
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.6.tgz",
"integrity": "sha512-6lvjbS3p9b4CrdCmguzbh2/4uoXhGE2q71R4OX5sqF9R1bo9Xd6fGrMAfvp5wnCzlBnFVdCOp6onuTQVbo8iUQ==",
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.5.tgz",
"integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/expect": "4.1.6",
"@vitest/mocker": "4.1.6",
"@vitest/pretty-format": "4.1.6",
"@vitest/runner": "4.1.6",
"@vitest/snapshot": "4.1.6",
"@vitest/spy": "4.1.6",
"@vitest/utils": "4.1.6",
"@vitest/expect": "4.1.5",
"@vitest/mocker": "4.1.5",
"@vitest/pretty-format": "4.1.5",
"@vitest/runner": "4.1.5",
"@vitest/snapshot": "4.1.5",
"@vitest/spy": "4.1.5",
"@vitest/utils": "4.1.5",
"es-module-lexer": "^2.0.0",
"expect-type": "^1.3.0",
"magic-string": "^0.30.21",
@@ -1237,12 +1237,12 @@
"@edge-runtime/vm": "*",
"@opentelemetry/api": "^1.9.0",
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
"@vitest/browser-playwright": "4.1.6",
"@vitest/browser-preview": "4.1.6",
"@vitest/browser-webdriverio": "4.1.6",
"@vitest/coverage-istanbul": "4.1.6",
"@vitest/coverage-v8": "4.1.6",
"@vitest/ui": "4.1.6",
"@vitest/browser-playwright": "4.1.5",
"@vitest/browser-preview": "4.1.5",
"@vitest/browser-webdriverio": "4.1.5",
"@vitest/coverage-istanbul": "4.1.5",
"@vitest/coverage-v8": "4.1.5",
"@vitest/ui": "4.1.5",
"happy-dom": "*",
"jsdom": "*",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
-29
View File
@@ -13,20 +13,6 @@ export interface ConnectedEvent {
export interface HistoryEvent {
type: "history";
/**
* Per-message dicts the frontend consumes directly. Common optional keys:
* - `role`: "user" | "assistant" | "tool"
* - `content`: string or list (image/document parts)
* - `tool_calls`: assistant turns list of `{id, name, arguments, verdict?, output_assessment?}`
* - `tool_call_id`: tool turns id of the originating call
* - `reminders`: metacognitive nudge bubbles (user/tool channels)
* - `advisories`: extracted `UserInterjection` payloads on tool turns
* - `reasoning`: concatenated reasoning text for assistant turns whose
* `provider_data` carried reasoning-bearing blocks (Anthropic
* `thinking`, OpenAI Responses `reasoning`, or synthetic
* `reasoning_text` from path-3 servers). Present only when the
* active model's `surface_persisted_reasoning` flag is true.
*/
messages: Array<Record<string, unknown>>;
}
@@ -52,14 +38,6 @@ export interface StreamEndEvent {
type: "stream_end";
}
/** One-shot replay of the in-progress turn's content + reasoning emitted
* by the events SSE handler when a fresh subscriber connects mid-stream. */
export interface InProgressSnapshotEvent {
type: "in_progress_snapshot";
content: string;
reasoning: string;
}
export interface StateChangeEvent {
type: "state_change";
state: "idle" | "thinking" | "running" | "attention" | "error";
@@ -184,7 +162,6 @@ export type ServerEvent =
| ContentEvent
| ReasoningEvent
| StreamEndEvent
| InProgressSnapshotEvent
| StateChangeEvent
| ToolInfoEvent
| ApproveRequestEvent
@@ -284,12 +261,6 @@ export function isStreamEndEvent(e: ServerEvent): e is StreamEndEvent {
return e.type === "stream_end";
}
export function isInProgressSnapshotEvent(
e: ServerEvent,
): e is InProgressSnapshotEvent {
return e.type === "in_progress_snapshot";
}
export function isStateChangeEvent(e: ServerEvent): e is StateChangeEvent {
return e.type === "state_change";
}
-25
View File
@@ -26,28 +26,3 @@ def make_chat_session(**overrides: Any) -> Any:
}
defaults.update(overrides)
return ChatSession(**defaults)
def patch_session_storage(
monkeypatch: Any,
*,
active: bool = True,
raise_on_is_active: bool = False,
) -> list[str]:
"""Patch ``session.get_storage`` to a stub whose ``is_watch_active``
returns *active* (or raises if *raise_on_is_active*). Returns the
list of ``watch_id``s the predicate was called with.
"""
from turnstone.core import session as session_mod
calls: list[str] = []
class _Stub:
def is_watch_active(self, watch_id: str) -> bool:
calls.append(watch_id)
if raise_on_is_active:
raise RuntimeError("storage down")
return active
monkeypatch.setattr(session_mod, "get_storage", lambda: _Stub())
return calls
-45
View File
@@ -1,45 +0,0 @@
"""Shared session-test helpers.
Two reasoning-test modules (``test_session_replay_reasoning.py`` and
``test_session_synth_reasoning_block.py``) need the same minimal
``ChatSession`` factory + a ``SessionUIBase`` no-op subclass. Hoisting
keeps a future third caller from drifting on the defaults the third
existing ``_make_session`` (``test_model_registry.py``) deliberately
takes a different signature (registry / model_alias / reasoning_effort
+ ``_FakeUI``) and is NOT a candidate for sharing this helper.
Module is named with a leading underscore so pytest doesn't try to
collect it as a test file it's an importable utility, not a test.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
from turnstone.core.session_ui_base import SessionUIBase
class NullUI(SessionUIBase):
"""Bare-bones UI satisfying the SessionUIBase contract for tests
that don't care about UI side effects."""
def __init__(self) -> None:
super().__init__()
def make_session(**kwargs: Any) -> ChatSession:
"""Build a ChatSession with minimal defaults; tests override
individual fields via kwargs."""
defaults: dict[str, Any] = {
"client": MagicMock(),
"model": "test-model",
"ui": NullUI(),
"instructions": None,
"temperature": 0.5,
"max_tokens": 4096,
"tool_timeout": 30,
}
defaults.update(kwargs)
return ChatSession(**defaults)
-69
View File
@@ -1,79 +1,10 @@
from __future__ import annotations
import os
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock
import pytest
if TYPE_CHECKING:
from turnstone.core.mcp_client import MCPClientManager, StaticServerState
from turnstone.core.mcp_crypto import MCPTokenCipher
from turnstone.core.oidc import OIDCConfig
def make_mcp_token_cipher() -> MCPTokenCipher:
"""Build a single-key MCP token cipher for tests.
Used by test files that need to exercise ``MCPTokenStore`` round-
trips without the lifespan-side configuration loader; centralised
here so the key/material defaults stay aligned across files.
"""
import base64
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import MCPTokenCipher, MCPTokenCipherConfig
raw = base64.urlsafe_b64decode(Fernet.generate_key())
return MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,)))
def _seed_static_state(mgr: MCPClientManager, name: str, **overrides: Any) -> StaticServerState:
"""Get-or-create a ``StaticServerState`` on ``mgr`` and apply ``overrides``.
Shared across MCP test files so the helper stays in one place. Imported
where needed; ``StaticServerState`` is constructed lazily so non-MCP
tests don't pay the import cost.
"""
from turnstone.core.mcp_client import StaticServerState
state = mgr._static_servers.get(name)
if state is None:
state = StaticServerState(name=name)
mgr._static_servers[name] = state
for k, v in overrides.items():
setattr(state, k, v)
return state
def make_oidc_test_config(**overrides: Any) -> OIDCConfig:
"""Build a test ``OIDCConfig`` with sensible defaults.
Shared between ``test_oidc.py`` and ``test_oidc_handlers.py`` so the
defaults (including the now-required ``redirect_base``) stay aligned.
"""
from turnstone.core.oidc import OIDCConfig
defaults: dict[str, Any] = {
"enabled": True,
"issuer": "https://idp.example.com",
"client_id": "my-client",
"client_secret": "my-secret",
"scopes": "openid email profile",
"provider_name": "TestIDP",
"role_claim": "",
"role_map": {},
"password_enabled": True,
"redirect_base": "https://app.example.com",
"authorization_endpoint": "https://idp.example.com/authorize",
"token_endpoint": "https://idp.example.com/token",
"userinfo_endpoint": "https://idp.example.com/userinfo",
"jwks_uri": "https://idp.example.com/.well-known/jwks.json",
}
defaults.update(overrides)
return OIDCConfig(**defaults)
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption(
-391
View File
@@ -1,391 +0,0 @@
"""Spike 1 — validate MCP SDK behavior for the per-(user, server) session pool.
Three scenarios:
1. N=20 concurrent ClientSession instances to the same URL.
Verifies: no FD blow-up, no shared transport state, each session's
tools/list returns independently.
2. Two concurrent tools/call on a shared ClientSession with interleaving
payloads. Verifies: request_id demux works under contention.
3. Per-session Authorization header isolation. Verifies: different Bearer
tokens per ClientSession reach the server with the expected
Authorization header i.e. httpx connection pooling does not cross
headers between sessions.
Run: uv run python tests/spike_sdk_concurrency.py
Outcome gates Phase 5's pool architecture; if any scenario fails, fall
back to per-call header injection (Alternative F in the OAuth-MCP RFC).
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
import os
import socket
import sys
import threading
import time
from collections import defaultdict
from typing import TYPE_CHECKING
import uvicorn
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
# Reduce uvicorn / mcp log noise so spike output is readable.
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
# Records (auth_header, tool_name) per request — populated by the
# AuthHeaderRecorder middleware below. Indexed by call sequence.
SERVER_OBSERVATIONS: list[tuple[str | None, str | None]] = []
# Tool-call payloads observed (for request_id demux verification).
TOOL_CALL_PAYLOADS: list[str] = []
class AuthHeaderRecorder(BaseHTTPMiddleware):
"""Records the Authorization header on every request the server sees."""
async def dispatch(self, request: Request, call_next: Callable) -> Response:
auth = request.headers.get("authorization")
# We only record the auth header here; tool name comes from the
# body payload which we can't read non-destructively. The tool
# handler logs the payload it received.
SERVER_OBSERVATIONS.append((auth, None))
return await call_next(request)
def find_free_port() -> int:
"""Bind to port 0, return the assigned port."""
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def build_server(port: int) -> uvicorn.Server:
"""Create a minimal FastMCP server with one echo tool."""
mcp = FastMCP(name="spike-target", streamable_http_path="/mcp")
@mcp.tool()
async def echo(payload: str) -> str:
"""Echo the payload back. Records the payload server-side."""
TOOL_CALL_PAYLOADS.append(payload)
# Add a small await so two concurrent calls can interleave
# on the wire if the SDK pools the requests.
await asyncio.sleep(0.05)
return f"echoed:{payload}"
app = mcp.streamable_http_app()
app.add_middleware(AuthHeaderRecorder)
config = uvicorn.Config(
app,
host="127.0.0.1",
port=port,
log_level="warning",
access_log=False,
)
return uvicorn.Server(config)
def run_server_in_thread(server: uvicorn.Server) -> threading.Thread:
"""Boot the server in a background thread on its own asyncio loop."""
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="spike-server")
t.start()
return t
async def wait_for_server_ready(url: str, timeout: float = 5.0) -> None:
"""Poll the server until it accepts connections."""
import urllib.parse
parsed = urllib.parse.urlparse(url)
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
reader, writer = await asyncio.open_connection(parsed.hostname, parsed.port)
writer.close()
await writer.wait_closed()
return
except OSError:
await asyncio.sleep(0.05)
raise TimeoutError(f"server at {url} not ready within {timeout}s")
def fd_count() -> int:
"""Count open file descriptors for the current process."""
try:
return len(os.listdir(f"/proc/{os.getpid()}/fd"))
except OSError:
return -1
# ---------------------------------------------------------------------------
# Scenario 1: N=20 concurrent ClientSession instances
# ---------------------------------------------------------------------------
async def scenario_1_concurrent_sessions(url: str, n: int = 20) -> dict:
"""Open N concurrent ClientSession instances and call tools/list on each."""
print(f"\n=== Scenario 1: {n} concurrent ClientSession instances ===")
fd_before = fd_count()
async def one_session(idx: int) -> dict:
headers = {"Authorization": f"Bearer test-token-{idx}"}
async with (
streamablehttp_client(url=url, headers=headers) as (read, write, _),
ClientSession(read, write) as session,
):
await session.initialize()
tools = await session.list_tools()
return {
"idx": idx,
"tool_count": len(tools.tools),
"tool_names": [t.name for t in tools.tools],
}
start = time.monotonic()
results = await asyncio.gather(*[one_session(i) for i in range(n)], return_exceptions=True)
elapsed = time.monotonic() - start
fd_after = fd_count()
# Allow some settling time for FDs to release.
await asyncio.sleep(0.5)
fd_settled = fd_count()
successes = [r for r in results if isinstance(r, dict)]
failures = [r for r in results if isinstance(r, Exception)]
# Verify every session got the same tool catalog.
catalog_consistent = (
len(successes) == n and len({tuple(r["tool_names"]) for r in successes}) == 1
)
return {
"scenario": "concurrent_sessions",
"n": n,
"successes": len(successes),
"failures": len(failures),
"elapsed_seconds": round(elapsed, 3),
"fd_before": fd_before,
"fd_during_peak": fd_after,
"fd_settled": fd_settled,
"fd_growth_during": fd_after - fd_before,
"fd_growth_settled": fd_settled - fd_before,
"catalog_consistent": catalog_consistent,
"first_failure": str(failures[0]) if failures else None,
}
# ---------------------------------------------------------------------------
# Scenario 2: 2 concurrent tools/call on a shared session
# ---------------------------------------------------------------------------
async def scenario_2_concurrent_calls_shared_session(url: str) -> dict:
"""Two concurrent tools/call on one ClientSession with interleaving payloads.
The echo tool sleeps 50ms, so concurrent calls overlap on the wire.
Each call passes a distinct payload (~10KB) to make request bodies
spannable across multiple stream frames.
"""
print("\n=== Scenario 2: 2 concurrent tools/call on shared session ===")
# Generous-size payloads so both bodies live during the await.
payload_a = "A" * 10000
payload_b = "B" * 10000
headers = {"Authorization": "Bearer shared-session-token"}
async with (
streamablehttp_client(url=url, headers=headers) as (read, write, _),
ClientSession(read, write) as session,
):
await session.initialize()
TOOL_CALL_PAYLOADS.clear()
start = time.monotonic()
results = await asyncio.gather(
session.call_tool("echo", {"payload": payload_a}),
session.call_tool("echo", {"payload": payload_b}),
return_exceptions=True,
)
elapsed = time.monotonic() - start
successes = [r for r in results if not isinstance(r, Exception)]
failures = [r for r in results if isinstance(r, Exception)]
# Each result.content[0].text should be "echoed:{payload}".
response_payloads: list[str] = []
if len(successes) == 2:
for r in successes:
text = r.content[0].text if r.content else ""
response_payloads.append(text)
# Order may not match call order — what matters is both payloads echo.
expected = {f"echoed:{payload_a}", f"echoed:{payload_b}"}
received = set(response_payloads)
demux_ok = received == expected
# Did both calls actually overlap? If sequential, elapsed ~= 0.1+s;
# if concurrent, ~0.05s.
concurrent_observed = elapsed < 0.09
return {
"scenario": "concurrent_calls_shared_session",
"successes": len(successes),
"failures": len(failures),
"elapsed_seconds": round(elapsed, 3),
"demux_ok": demux_ok,
"expected_payloads_received": list(received) if demux_ok else None,
"actual_payloads_received_count": len(received),
"appears_concurrent_on_wire": concurrent_observed,
"first_failure": str(failures[0]) if failures else None,
}
# ---------------------------------------------------------------------------
# Scenario 3: per-session header isolation
# ---------------------------------------------------------------------------
async def scenario_3_header_isolation(url: str, n: int = 5) -> dict:
"""Open N sessions with distinct Authorization headers, call echo on each.
Verifies the server sees each session's own header — i.e. httpx
connection pooling does not cross headers between concurrent
ClientSession instances against the same URL.
"""
print(f"\n=== Scenario 3: {n}-session Authorization-header isolation ===")
SERVER_OBSERVATIONS.clear()
async def one_session(idx: int) -> str | None:
headers = {"Authorization": f"Bearer iso-token-{idx}"}
async with (
streamablehttp_client(url=url, headers=headers) as (read, write, _),
ClientSession(read, write) as session,
):
await session.initialize()
# One call per session.
await session.call_tool("echo", {"payload": f"session-{idx}"})
return f"Bearer iso-token-{idx}"
start = time.monotonic()
expected_tokens = await asyncio.gather(*[one_session(i) for i in range(n)])
elapsed = time.monotonic() - start
# Tally observed Authorization headers, ignoring None entries (initial
# handshake sometimes lacks auth).
observed_auth = [auth for auth, _ in SERVER_OBSERVATIONS if auth]
expected_set = set(expected_tokens)
observed_set = set(observed_auth)
# Every expected token must show up at least once on the server.
all_present = expected_set.issubset(observed_set)
# No spurious tokens.
no_extras = observed_set.issubset(expected_set)
# Frequency: at least one observation per token.
counts = defaultdict(int)
for a in observed_auth:
counts[a] += 1
each_seen = all(counts[t] >= 1 for t in expected_tokens)
return {
"scenario": "header_isolation",
"n": n,
"elapsed_seconds": round(elapsed, 3),
"expected_tokens": sorted(expected_set),
"observed_tokens": sorted(observed_set),
"all_expected_present": all_present,
"no_extra_tokens_observed": no_extras,
"each_token_seen_at_least_once": each_seen,
"header_counts_per_token": dict(counts),
"total_requests_observed": len(observed_auth),
}
# ---------------------------------------------------------------------------
# Driver
# ---------------------------------------------------------------------------
async def main() -> None:
port = find_free_port()
url = f"http://127.0.0.1:{port}/mcp"
server = build_server(port)
server_thread = run_server_in_thread(server)
try:
await wait_for_server_ready(url)
print(f"server up at {url}\n")
result_1 = await scenario_1_concurrent_sessions(url, n=20)
print_scenario_result(result_1)
result_2 = await scenario_2_concurrent_calls_shared_session(url)
print_scenario_result(result_2)
result_3 = await scenario_3_header_isolation(url, n=5)
print_scenario_result(result_3)
# Final verdict
verdict_1 = (
result_1["successes"] == result_1["n"]
and result_1["catalog_consistent"]
and result_1["fd_growth_settled"] < 30 # 20 sessions, generous bound
)
verdict_2 = result_2["demux_ok"] and result_2["successes"] == 2
verdict_3 = (
result_3["all_expected_present"]
and result_3["no_extra_tokens_observed"]
and result_3["each_token_seen_at_least_once"]
)
print("\n=== VERDICT ===")
print(f" Scenario 1 (concurrent sessions): {'PASS' if verdict_1 else 'FAIL'}")
print(f" Scenario 2 (concurrent calls shared): {'PASS' if verdict_2 else 'FAIL'}")
print(f" Scenario 3 (header isolation): {'PASS' if verdict_3 else 'FAIL'}")
all_pass = verdict_1 and verdict_2 and verdict_3
print(
f"\n Phase 5 per-(user, server) pool architecture: "
f"{'VIABLE' if all_pass else 'NEEDS REWORK (Alternative F fallback)'}"
)
sys.exit(0 if all_pass else 1)
finally:
server.should_exit = True
server_thread.join(timeout=5)
def print_scenario_result(result: dict) -> None:
print(f"\nresult[{result['scenario']}]:")
for k, v in result.items():
if k == "scenario":
continue
print(f" {k}: {v}")
if __name__ == "__main__":
with contextlib.suppress(KeyboardInterrupt):
asyncio.run(main())
-220
View File
@@ -1,220 +0,0 @@
"""Tests for turnstone-admin DB configuration precedence.
Locks in the alignment with turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded default
The motivation is to keep DB secrets in config.toml (see
feedback_secrets_not_in_env) rather than forcing operators to export
TURNSTONE_DB_URL before every admin invocation.
"""
from __future__ import annotations
import argparse
from typing import TYPE_CHECKING
from unittest.mock import patch
import pytest
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
import turnstone.core.config as config_mod
from turnstone.admin import _get_storage
def _reset_cache() -> None:
config_mod._cache = None
config_mod._config_path = None
def _build_args(config_path: str | None) -> argparse.Namespace:
"""Build an args namespace the way admin.main() does.
Skips ``add_config_arg`` (which reads ``sys.argv``) the test
constructs the args programmatically instead.
"""
config_mod.set_config_path(config_path or "/nonexistent/turnstone-admin-test.toml")
parser = argparse.ArgumentParser()
config_mod.apply_config(parser, ["database"])
sub = parser.add_subparsers(dest="command")
sub.add_parser("list-users")
return parser.parse_args(["list-users"])
@pytest.fixture(autouse=True)
def _clear_db_env(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
"""Clean slate: no TURNSTONE_DB_* env vars unless a test sets them."""
for var in (
"TURNSTONE_DB_BACKEND",
"TURNSTONE_DB_URL",
"TURNSTONE_DB_PATH",
"TURNSTONE_DB_POOL_SIZE",
"TURNSTONE_DB_SSLMODE",
"TURNSTONE_DB_SSLROOTCERT",
"TURNSTONE_DB_SSLCERT",
"TURNSTONE_DB_SSLKEY",
"TURNSTONE_CONFIG",
):
monkeypatch.delenv(var, raising=False)
_reset_cache()
yield
_reset_cache()
def test_defaults_to_sqlite_when_neither_config_nor_env_set() -> None:
args = _build_args(None)
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("sqlite",)
assert init.call_args.kwargs["url"] == ""
assert init.call_args.kwargs["path"] == ""
assert init.call_args.kwargs["pool_size"] == 2
def test_config_toml_database_section_drives_init_storage(tmp_path: Path) -> None:
cfg = tmp_path / "config.toml"
cfg.write_text(
"[database]\n"
'backend = "postgresql"\n'
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
"pool_size = 5\n"
'sslmode = "verify-full"\n'
'sslrootcert = "/etc/ssl/ca.pem"\n'
'sslcert = "/etc/ssl/client.pem"\n'
'sslkey = "/etc/ssl/client.key"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["pool_size"] == 5
assert kw["sslmode"] == "verify-full"
assert kw["sslrootcert"] == "/etc/ssl/ca.pem"
assert kw["sslcert"] == "/etc/ssl/client.pem"
assert kw["sslkey"] == "/etc/ssl/client.key"
def test_env_used_as_fallback_when_config_absent(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "postgresql")
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "7")
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
args = _build_args(None)
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromenv:x@host/db"
assert kw["pool_size"] == 7
assert kw["sslmode"] == "require"
def test_config_toml_wins_over_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""config.toml beats env — operators should put secrets in TOML."""
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "sqlite")
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
cfg = tmp_path / "config.toml"
cfg.write_text(
"[database]\n"
'backend = "postgresql"\n'
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
'sslmode = "verify-full"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["sslmode"] == "verify-full"
def test_partial_config_falls_through_to_env_per_key(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""A key missing from [database] should fall back to its env var."""
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "9")
cfg = tmp_path / "config.toml"
cfg.write_text(
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromconfig:x@host/db"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["sslmode"] == "require"
assert kw["pool_size"] == 9
def test_empty_string_in_config_beats_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""`url = ""` in config.toml beats an env var.
Locks in the `is not None` guard a falsy-but-present TOML value
should NOT silently fall through to the env fallback.
"""
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nbackend = "sqlite"\nurl = ""\n')
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.kwargs["url"] == ""
def test_main_threads_config_toml_through_real_argv(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""End-to-end: ``turnstone-admin --config <toml> list-users`` honors TOML.
Covers the ``add_config_arg`` -> ``apply_config`` -> ``_get_storage``
chain that the programmatic ``_build_args`` helper skips.
"""
cfg = tmp_path / "config.toml"
cfg.write_text(
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromcli:x@host/db"\n'
)
monkeypatch.setattr("sys.argv", ["turnstone-admin", "--config", str(cfg), "list-users"])
fake_storage = patch("turnstone.core.storage.init_storage").start()
fake_storage.return_value.list_users.return_value = []
try:
from turnstone.admin import main
main()
finally:
patch.stopall()
assert fake_storage.call_args.args == ("postgresql",)
assert fake_storage.call_args.kwargs["url"] == "postgresql+psycopg://fromcli:x@host/db"
def test_get_storage_initializes_real_sqlite_backend(tmp_path: Path) -> None:
"""Drives the real ``init_storage`` boundary on a fresh sqlite file.
Mock-only tests would miss a kwarg-name typo (sslmode -> ssl_mode).
This test trips on any such drift because Alembic + the backend
actually run.
"""
from turnstone.core.storage import reset_storage
db_file = tmp_path / "admin.db"
cfg = tmp_path / "config.toml"
cfg.write_text(f'[database]\nbackend = "sqlite"\npath = "{db_file}"\n')
args = _build_args(str(cfg))
reset_storage()
try:
storage = _get_storage(args)
assert storage.list_users() == []
finally:
reset_storage()
-631
View File
@@ -22,8 +22,6 @@ to lock in:
from __future__ import annotations
import threading
from types import SimpleNamespace
from typing import Any
from unittest.mock import MagicMock
@@ -35,7 +33,6 @@ from starlette.testclient import TestClient
from tests._coord_test_helpers import _AuthMiddleware
from turnstone.console.server import (
_maybe_bootstrap_coord_subsystem,
_refresh_coord_registry,
admin_create_model_definition,
admin_delete_model_definition,
@@ -45,29 +42,6 @@ from turnstone.console.server import (
from turnstone.core.model_registry import ModelConfig, ModelRegistry
from turnstone.core.storage._sqlite import SQLiteBackend
def _bootstrap_app(**overrides: Any) -> Any:
"""Build a fake ``app`` with the ``state`` attrs the bootstrap helper
inspects. Defaults match a freshly-installed console (no coord
subsystem yet) with all required prereqs (collector, console_metrics,
config_store) populated as MagicMocks. Tests pass overrides to
suppress individual prereqs or pre-set ``coord_mgr`` etc.
"""
state_kwargs: dict[str, Any] = {
"coord_mgr": None,
"coord_adapter": None,
"coord_registry": None,
"coord_registry_error": "",
"coord_state_writer": None,
"coord_idle_observer": None,
"config_store": MagicMock(),
"collector": MagicMock(),
"console_metrics": MagicMock(),
}
state_kwargs.update(overrides)
return SimpleNamespace(state=SimpleNamespace(**state_kwargs))
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@@ -239,538 +213,6 @@ def test_helper_preserves_registry_when_no_enabled_rows(storage: SQLiteBackend)
assert state.coord_registry.get_config("local").model == "cached-model"
# ---------------------------------------------------------------------------
# First-row bootstrap tests — ``_maybe_bootstrap_coord_subsystem`` semantics.
# A console booted with no model rows leaves coord_mgr = None; the operator
# adding the first row at runtime must promote the subsystem to ready
# without a console restart.
# ---------------------------------------------------------------------------
def test_bootstrap_noop_when_coord_mgr_already_built(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Idempotent fast-path — already-bootstrapped subsystem must not
re-stand-up a second SessionManager / StateWriter pair."""
from turnstone.console import server as server_module
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
app = _bootstrap_app(coord_mgr=MagicMock()) # subsystem already built
calls: list[Any] = []
monkeypatch.setattr(
server_module,
"_bootstrap_coord_subsystem",
lambda *a, **kw: calls.append(a),
)
_maybe_bootstrap_coord_subsystem(app, storage)
assert calls == []
@pytest.mark.parametrize("missing_attr", ["config_store", "collector", "console_metrics"])
def test_bootstrap_noop_when_prerequisites_missing(
storage: SQLiteBackend,
monkeypatch: pytest.MonkeyPatch,
missing_attr: str,
) -> None:
"""Each strictly-required ``app.state`` attr (config_store, collector,
console_metrics) must individually short-circuit the bootstrap to a
no-op partial init / test harnesses don't have the full set, and a
CRUD write that already landed mustn't 500 on a missing prereq."""
from turnstone.console import server as server_module
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
app = _bootstrap_app(**{missing_attr: None})
calls: list[Any] = []
monkeypatch.setattr(
server_module,
"_bootstrap_coord_subsystem",
lambda *a, **kw: calls.append(a),
)
_maybe_bootstrap_coord_subsystem(app, storage)
assert calls == []
assert app.state.coord_mgr is None
def test_bootstrap_records_error_when_no_rows(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""All rows disabled (or none seeded) — load_model_registry raises
ValueError. Helper records the message on app.state so the
coord-endpoint 503 surfaces a current diagnosis instead of a stale
one from boot."""
from turnstone.console import server as server_module
app = _bootstrap_app()
calls: list[Any] = []
monkeypatch.setattr(
server_module,
"_bootstrap_coord_subsystem",
lambda *a, **kw: calls.append(a),
)
_maybe_bootstrap_coord_subsystem(app, storage)
assert calls == []
assert "No model definitions found" in app.state.coord_registry_error
def test_bootstrap_calls_subsystem_builder_on_first_row(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A row exists ⇒ helper loads the registry, hands it to the
subsystem builder, and the builder stamps it on app.state. Mirrors
the post-build invariant the real ``_bootstrap_coord_subsystem``
establishes (coord_registry set iff coord_mgr set) so the stale
boot-time error string clears as part of the same commit step."""
from turnstone.console import server as server_module
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
app = _bootstrap_app(coord_registry_error="stale boot-time message")
captured: dict[str, Any] = {}
def _fake_build(app_arg: Any, _storage: Any, _cfg: Any, registry_arg: Any) -> None:
captured["app"] = app_arg
captured["registry"] = registry_arg
# Simulate the real builder's final commit step: stamp registry
# + clear stale error + set coord_mgr atomically.
app_arg.state.coord_registry = registry_arg
app_arg.state.coord_registry_error = ""
app_arg.state.coord_mgr = MagicMock()
monkeypatch.setattr(server_module, "_bootstrap_coord_subsystem", _fake_build)
_maybe_bootstrap_coord_subsystem(app, storage)
assert captured["app"] is app
assert captured["registry"].has_alias("local")
assert app.state.coord_registry is captured["registry"]
assert app.state.coord_registry_error == ""
def test_bootstrap_replaces_stale_error_on_builder_failure(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A builder failure after a successful registry load must not leave
the stale "no model definitions" message on app.state that
diagnosis is demonstrably wrong (rows ARE present, the build failed
for a different reason). Replacement message must surface the
actual exception type so operators can correlate with logs."""
from turnstone.console import server as server_module
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
app = _bootstrap_app(
coord_registry_error=(
"No model definitions found. Provide --model, configure [models.*] "
"in config.toml, or add model definitions in the admin panel."
)
)
def _boom(*_a: Any, **_kw: Any) -> None:
raise RuntimeError("simulated builder failure")
monkeypatch.setattr(server_module, "_bootstrap_coord_subsystem", _boom)
_maybe_bootstrap_coord_subsystem(app, storage) # must not raise
assert app.state.coord_mgr is None
# Stale "no models" message replaced.
assert "No model definitions found" not in app.state.coord_registry_error
# New message mentions the actual failure class so the 503 banner
# gives operators something actionable beyond "look at logs".
assert "RuntimeError" in app.state.coord_registry_error
assert "failed to initialise" in app.state.coord_registry_error
def test_bootstrap_tears_down_partial_state_on_builder_failure(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""If the builder partially stamps handles on app.state and then
raises, the helper must call the teardown path so a subsequent
retry doesn't leak a StateWriter daemon / observer subscription."""
from turnstone.console import server as server_module
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
app = _bootstrap_app()
state_writer = MagicMock()
idle_observer = MagicMock()
coord_adapter = MagicMock()
def _partial_then_boom(app_arg: Any, *_a: Any, **_kw: Any) -> None:
# Mirror the real builder's stamp-immediately-after-start order:
# StateWriter spawned + stamped before SessionManager validates.
app_arg.state.coord_state_writer = state_writer
app_arg.state.coord_idle_observer = idle_observer
app_arg.state.coord_adapter = coord_adapter
raise RuntimeError("simulated mid-build failure")
monkeypatch.setattr(server_module, "_bootstrap_coord_subsystem", _partial_then_boom)
_maybe_bootstrap_coord_subsystem(app, storage)
# Teardown ran for each partially-stamped handle.
state_writer.shutdown.assert_called_once()
idle_observer.shutdown.assert_called_once()
coord_adapter.shutdown.assert_called_once()
# And the app.state slots are reset so a retry sees a clean field.
assert app.state.coord_state_writer is None
assert app.state.coord_idle_observer is None
assert app.state.coord_adapter is None
assert app.state.coord_mgr is None
assert app.state.coord_registry is None
def test_real_bootstrap_stands_up_subsystem_end_to_end(
storage: SQLiteBackend,
) -> None:
"""End-to-end: the real ``_bootstrap_coord_subsystem`` constructs a
working ``SessionManager`` against a real ``ConfigStore`` + real
``ClusterCollector`` when an operator adds the first model row to
a freshly-installed console.
This is the test that reproduces the user-reported bug without it,
all the bootstrap helper-level tests can pass even if the real
builder never actually completes (the helper-level tests
monkeypatch the builder out). Asserts the post-bootstrap invariant
that ``_require_coord_mgr`` relies on: ``coord_mgr`` is a real
SessionManager and ``coord_registry_error`` has been cleared.
"""
from turnstone.console import server as server_module
from turnstone.console.collector import ClusterCollector
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.console.metrics import ConsoleMetrics
from turnstone.core.config_store import ConfigStore
from turnstone.core.session_manager import SessionManager
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
config_store = ConfigStore(storage)
# Disable the idle-cleanup daemon for this test — it has no
# stop_event hook in the bootstrap (the loop runs until process
# termination) so leaving the default 120-minute timeout would
# leak a daemon thread across every test run.
config_store.set("server.workstream_idle_timeout", 0)
# ClusterCollector is constructed but NOT started — start() spawns
# network discovery + SSE manager threads we don't need for this
# test. ensure_console_pseudo_node() (called by the bootstrap via
# start_child_event_fanout) operates on the in-memory snapshot map
# without requiring the discovery loop to be live.
collector = ClusterCollector(storage=storage)
# Snapshot ConsoleCoordinatorUI's class attrs so the test can
# restore them on teardown — the bootstrap mutates them and they
# persist across tests at process scope.
saved_coord_mgr = ConsoleCoordinatorUI._coord_mgr
saved_collector = ConsoleCoordinatorUI._collector
saved_metrics = ConsoleCoordinatorUI._console_metrics
app = SimpleNamespace(
state=SimpleNamespace(
coord_mgr=None,
coord_adapter=None,
coord_registry=None,
coord_registry_error=(
"No model definitions found. Provide --model, configure [models.*] "
"in config.toml, or add model definitions in the admin panel."
),
coord_state_writer=None,
coord_idle_observer=None,
config_store=config_store,
collector=collector,
console_metrics=ConsoleMetrics(),
jwt_secret="x" * 32,
console_url="http://127.0.0.1:8001",
)
)
try:
_maybe_bootstrap_coord_subsystem(app, storage)
# The real builder ran and produced a working SessionManager.
assert isinstance(app.state.coord_mgr, SessionManager)
assert app.state.coord_adapter is not None
# Registry stamped with the seeded alias.
assert app.state.coord_registry is not None
assert app.state.coord_registry.has_alias("local")
# Stale boot-time error string cleared as part of the commit.
assert app.state.coord_registry_error == ""
# StateWriter daemon is alive — it's the load-bearing async
# persistence layer for SessionManager state transitions.
assert app.state.coord_state_writer is not None
# Class-level wiring on ConsoleCoordinatorUI is the path
# on_state_change / on_rename use to fan out to the dashboard.
assert ConsoleCoordinatorUI._coord_mgr is app.state.coord_mgr
assert ConsoleCoordinatorUI._collector is collector
finally:
# Tear down threads + subscriptions spawned by the bootstrap.
# ``_teardown_partial_coord_subsystem`` does the same work the
# runtime-bootstrap failure path does, so reusing it here also
# exercises that helper end-to-end.
server_module._teardown_partial_coord_subsystem(app)
# Restore ConsoleCoordinatorUI class attrs so other tests in
# the suite see them as they were before this test ran.
ConsoleCoordinatorUI._coord_mgr = saved_coord_mgr
ConsoleCoordinatorUI._collector = saved_collector
ConsoleCoordinatorUI._console_metrics = saved_metrics
def test_real_bootstrap_rolls_back_partial_state_on_side_effect_failure(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The real ``_bootstrap_coord_subsystem`` must roll back from
locally-held handles when a side-effect step fails mid-build, so
``app.state`` is never stamped (no half-built subsystem visible)
and the started ``StateWriter`` daemon is shut down (no leaked
thread across retries).
Exercises the bug-2 fix end-to-end: monkeypatches
``install_idle_nudge_watcher`` to raise, drives the real builder,
and asserts (a) the exception propagates, (b) ``app.state`` shows
a clean fresh-install state, (c) the started ``StateWriter`` is
no longer alive.
"""
from turnstone.console import server as server_module
from turnstone.console.collector import ClusterCollector
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.console.metrics import ConsoleMetrics
from turnstone.core.config_store import ConfigStore
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
config_store = ConfigStore(storage)
config_store.set("server.workstream_idle_timeout", 0)
collector = ClusterCollector(storage=storage)
saved_coord_mgr = ConsoleCoordinatorUI._coord_mgr
saved_collector = ConsoleCoordinatorUI._collector
saved_metrics = ConsoleCoordinatorUI._console_metrics
app = SimpleNamespace(
state=SimpleNamespace(
coord_mgr=None,
coord_adapter=None,
coord_registry=None,
coord_registry_error="boot-time stale message",
coord_state_writer=None,
coord_idle_observer=None,
config_store=config_store,
collector=collector,
console_metrics=ConsoleMetrics(),
jwt_secret="x" * 32,
console_url="http://127.0.0.1:8001",
)
)
# Monkeypatch a mid-build side-effect to fail AFTER StateWriter +
# observer have started but BEFORE the atomic commit. This is the
# exact failure shape the new local-rollback path is designed to
# handle cleanly.
def _boom(*_a: Any, **_kw: Any) -> Any:
raise RuntimeError("simulated mid-build subscription failure")
monkeypatch.setattr("turnstone.console.server.install_idle_nudge_watcher", _boom, raising=False)
# The bootstrap helper imports install_idle_nudge_watcher locally
# at call time (inside the function), so we need to patch the
# source module too — server.py's import is a name lookup against
# the module each call.
monkeypatch.setattr(
"turnstone.core.idle_nudge_watcher.install_idle_nudge_watcher",
_boom,
)
try:
# ``_maybe_bootstrap_coord_subsystem`` swallows the exception,
# logs it, and replaces the stale boot-time error string with
# a builder-failure-specific one — but the underlying invariant
# we're testing here is that the real builder cleaned up its
# own partial side-effects so ``app.state`` is left clean.
_maybe_bootstrap_coord_subsystem(app, storage)
# No state stamped — atomic commit never reached.
assert app.state.coord_mgr is None
assert app.state.coord_registry is None
assert app.state.coord_state_writer is None
assert app.state.coord_idle_observer is None
assert app.state.coord_adapter is None
# ConsoleCoordinatorUI class attrs were never stamped because
# they sit AFTER the side-effect phase — local-rollback never
# had to touch them, but the post-failure state still matches
# the lifespan's clean state.
assert ConsoleCoordinatorUI._coord_mgr is None
# The error string surfaces the actual failure cause, not the
# stale boot-time "no models" message.
assert "RuntimeError" in app.state.coord_registry_error
assert "failed to initialise" in app.state.coord_registry_error
finally:
# Defensive — _maybe_bootstrap should already have torn down,
# but call once more in case future drift introduces a leak.
server_module._teardown_partial_coord_subsystem(app)
ConsoleCoordinatorUI._coord_mgr = saved_coord_mgr
ConsoleCoordinatorUI._collector = saved_collector
ConsoleCoordinatorUI._console_metrics = saved_metrics
def test_bootstrap_atomic_commit_no_partial_visibility(
storage: SQLiteBackend,
) -> None:
"""A concurrent reader scanning ``app.state`` while the bootstrap
runs must never observe ``coord_mgr`` set with ``coord_registry``
still ``None`` that combination would surface a misleading
"Restart the console after adding a model definition" 503 from
:func:`_require_coord_mgr` even though the operator just
successfully added a model.
Drives the real builder while a separate thread polls
``coord_mgr`` / ``coord_registry`` in tight loops; if the bootstrap
ever stamps ``coord_mgr`` before ``coord_registry``, the polling
thread will catch it.
"""
from turnstone.console import server as server_module
from turnstone.console.collector import ClusterCollector
from turnstone.console.coordinator_ui import ConsoleCoordinatorUI
from turnstone.console.metrics import ConsoleMetrics
from turnstone.core.config_store import ConfigStore
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
config_store = ConfigStore(storage)
config_store.set("server.workstream_idle_timeout", 0)
collector = ClusterCollector(storage=storage)
saved_coord_mgr = ConsoleCoordinatorUI._coord_mgr
saved_collector = ConsoleCoordinatorUI._collector
saved_metrics = ConsoleCoordinatorUI._console_metrics
app = SimpleNamespace(
state=SimpleNamespace(
coord_mgr=None,
coord_adapter=None,
coord_registry=None,
coord_registry_error="",
coord_state_writer=None,
coord_idle_observer=None,
config_store=config_store,
collector=collector,
console_metrics=ConsoleMetrics(),
jwt_secret="x" * 32,
console_url="http://127.0.0.1:8001",
)
)
stop_polling = threading.Event()
violations: list[str] = []
def _poll_for_partial_state() -> None:
# Tight loop emulating ``_require_coord_mgr``'s read pattern
# (coord_mgr first, then coord_registry). Any iteration that
# observes coord_mgr set with coord_registry still None is the
# exact bug Copilot's first finding pointed at.
while not stop_polling.is_set():
mgr = app.state.coord_mgr
reg = app.state.coord_registry
if mgr is not None and reg is None:
violations.append(f"mgr={mgr!r} reg={reg!r}")
return
poller = threading.Thread(target=_poll_for_partial_state, name="partial-state-poller")
poller.start()
try:
_maybe_bootstrap_coord_subsystem(app, storage)
finally:
stop_polling.set()
poller.join(timeout=2.0)
server_module._teardown_partial_coord_subsystem(app)
ConsoleCoordinatorUI._coord_mgr = saved_coord_mgr
ConsoleCoordinatorUI._collector = saved_collector
ConsoleCoordinatorUI._console_metrics = saved_metrics
assert violations == [], (
"concurrent reader observed coord_mgr set with coord_registry still None — "
f"atomic commit invariant violated: {violations}"
)
def test_bootstrap_lock_serialises_concurrent_calls(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Two simultaneous CRUD writes both seeing ``coord_mgr is None``
must serialise via ``_COORD_BOOTSTRAP_LOCK`` and the second caller
must observe the post-build state on its inside-the-lock re-check
so the builder runs exactly once. Without the lock + double-check,
both threads enter the build and stamp duplicate SessionManager /
StateWriter / observer triples on app.state.
The synchronisation is deterministic, not wall-clock-based: an
instrumented lock wrapper signals when a second acquirer arrives,
so the test fails fast and reproducibly on slow CI rather than
relying on a sleep long enough to "probably" let thread 2 reach
the lock a dependence the previous version was rightly criticised
for.
"""
from turnstone.console import server as server_module
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
app = _bootstrap_app()
build_count = 0
count_lock = threading.Lock()
in_build = threading.Event()
release_build = threading.Event()
def _slow_build(app_arg: Any, *_a: Any, **_kw: Any) -> None:
nonlocal build_count
with count_lock:
build_count += 1
is_first = build_count == 1
if is_first:
# Hold inside the build so the second thread is forced to
# queue at the lock — without the lock it would race ahead
# and increment build_count to 2.
in_build.set()
release_build.wait(timeout=2.0)
# Mirror the real builder's commit step.
app_arg.state.coord_mgr = MagicMock()
app_arg.state.coord_registry = MagicMock()
monkeypatch.setattr(server_module, "_bootstrap_coord_subsystem", _slow_build)
# Instrumented wrapper: delegates to a real ``threading.Lock`` so
# the production ``with _COORD_BOOTSTRAP_LOCK:`` block keeps doing
# genuine serialisation work, but counts arrivals so the main
# thread can wait deterministically until thread 2 is at the lock
# before releasing thread 1. If the production code drops the
# ``with`` block entirely, the wrapper is never entered, the
# arrival event never fires, and the assertion below times out
# with a clear error rather than the subtler false-pass a sleep
# would allow.
real_lock = threading.Lock()
arrivals_lock = threading.Lock()
arrivals = 0
second_waiter_arrived = threading.Event()
class _InstrumentedLock:
def __enter__(self) -> Any:
nonlocal arrivals
with arrivals_lock:
arrivals += 1
arrival_index = arrivals
if arrival_index >= 2:
second_waiter_arrived.set()
real_lock.acquire()
return self
def __exit__(self, *_exc: Any) -> None:
real_lock.release()
monkeypatch.setattr(server_module, "_COORD_BOOTSTRAP_LOCK", _InstrumentedLock())
def _run() -> None:
_maybe_bootstrap_coord_subsystem(app, storage)
t1 = threading.Thread(target=_run, name="bootstrap-thread-1")
t2 = threading.Thread(target=_run, name="bootstrap-thread-2")
t1.start()
assert in_build.wait(timeout=2.0), "thread 1 never entered the builder"
t2.start()
# Deterministic: block here until thread 2 has reached the lock
# (or the wait times out, signalling the lock was bypassed entirely).
assert second_waiter_arrived.wait(timeout=2.0), (
"thread 2 never reached the lock — concurrency was not exercised, "
"production code may be skipping the lock"
)
release_build.set()
t1.join(timeout=5.0)
t2.join(timeout=5.0)
assert not t1.is_alive() and not t2.is_alive()
assert build_count == 1, (
f"builder ran {build_count} times — lock failed to serialise concurrent calls"
)
def test_helper_preserves_registry_on_reload_validation_error(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
@@ -867,79 +309,6 @@ def test_create_endpoint_refreshes_registry(storage: SQLiteBackend) -> None:
assert registry.get_config("fast").model == "fast-model"
def test_create_endpoint_bootstraps_subsystem_on_fresh_install(
storage: SQLiteBackend, monkeypatch: pytest.MonkeyPatch
) -> None:
"""User-visible regression: a console booted with no model rows leaves
coord_mgr unbuilt; the operator adding their first model via the
admin panel must promote the subsystem to ready (no console restart).
Before the fix, ``_refresh_coord_registry`` short-circuited on
``coord_registry is None`` and the dashboard's 503 banner persisted
until the user restarted.
"""
from turnstone.console import server as server_module
# Fresh-install state: registry=None, coord_mgr=None, boot-time
# error string set by the lifespan's ValueError catch. Build the
# app explicitly so the test can inspect ``app.state`` after the
# request completes (TestClient's ``.app`` attribute is typed as
# ASGIApp, which loses the ``.state`` accessor).
app = Starlette(
routes=[
Route(
"/v1/api/admin/model-definitions",
admin_create_model_definition,
methods=["POST"],
),
],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.auth_storage = storage
app.state.coord_registry = None
app.state.coord_mgr = None
app.state.coord_registry_error = (
"No model definitions found. Provide --model, configure [models.*] "
"in config.toml, or add model definitions in the admin panel."
)
app.state.collector = MagicMock()
app.state.collector.get_all_nodes.return_value = []
app.state.config_store = MagicMock()
app.state.console_metrics = MagicMock()
client = TestClient(app)
client.headers.update({"X-Test-User": "admin", "X-Test-Perms": "admin.models"})
captured: dict[str, Any] = {}
def _fake_build(app_arg: Any, _storage: Any, _cfg: Any, registry_arg: Any) -> None:
captured["registry"] = registry_arg
# Mirror the real builder's commit step so the post-call asserts
# see the same invariant a successful real bootstrap establishes.
app_arg.state.coord_registry = registry_arg
app_arg.state.coord_registry_error = ""
app_arg.state.coord_mgr = MagicMock()
monkeypatch.setattr(server_module, "_bootstrap_coord_subsystem", _fake_build)
resp = client.post(
"/v1/api/admin/model-definitions",
json={
"alias": "first",
"model": "first-model",
"provider": "openai-compatible",
"base_url": "http://localhost:9000/v1",
"api_key": "sk-x",
},
)
assert resp.status_code == 200, resp.text
# Bootstrap fired with a registry holding the just-added alias.
assert "registry" in captured and captured["registry"].has_alias("first")
# coord_mgr is now non-None (bootstrap completed) and the stale
# boot-time error message has been cleared so subsequent 503s
# don't lie about current state.
assert app.state.coord_mgr is not None
assert app.state.coord_registry_error == ""
def test_update_endpoint_refreshes_registry(storage: SQLiteBackend) -> None:
"""PUT swaps the underlying model name behind a stable alias — the
user's reported regression."""
+5 -1050
View File
File diff suppressed because it is too large Load Diff
-57
View File
@@ -83,39 +83,6 @@ class TestIsPublicPath:
def test_shared_static_public(self):
assert is_public_path("/shared/base.css") is True
# Console proxy: a public proxied path must still be public, otherwise
# the login modal can never re-authenticate from inside a ``/node/{id}/``
# proxied page once the cookie expires.
def test_proxy_v1_login_public(self):
assert is_public_path("/node/node-a/v1/api/auth/login") is True
def test_proxy_no_v1_login_public(self):
assert is_public_path("/node/node-a/api/auth/login") is True
def test_proxy_v1_status_public(self):
assert is_public_path("/node/node-a/v1/api/auth/status") is True
def test_proxy_v1_setup_public(self):
assert is_public_path("/node/node-a/v1/api/auth/setup") is True
def test_proxy_v1_logout_public(self):
assert is_public_path("/node/node-a/v1/api/auth/logout") is True
def test_proxy_v1_oidc_authorize_public(self):
assert is_public_path("/node/node-a/v1/api/auth/oidc/authorize") is True
def test_proxy_v1_oidc_callback_public(self):
assert is_public_path("/node/node-a/v1/api/auth/oidc/callback") is True
def test_proxy_v1_workstreams_still_not_public(self):
"""Proxy prefix must not turn protected paths into public ones."""
assert is_public_path("/node/node-a/v1/api/workstreams") is False
def test_proxy_v1_refresh_still_requires_auth(self):
"""Refresh isn't in PUBLIC_PATHS — the caller must already have
a valid cookie. Proxy-prefix shouldn't change that."""
assert is_public_path("/node/node-a/v1/api/auth/refresh") is False
# ---------------------------------------------------------------------------
# TestRequiredRole
@@ -240,30 +207,6 @@ class TestRequiredScope:
"""Only POST is elevated — GET falls through to read."""
assert required_scope("GET", "/api/_internal/mcp-reload") == "read"
def test_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/api/_internal/mcp-refresh/srv") == "approve"
def test_v1_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/v1/api/_internal/mcp-refresh/srv") == "approve"
def test_proxy_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/node/n1/v1/api/_internal/mcp-refresh/srv") == "approve"
def test_proxy_no_v1_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/node/n1/api/_internal/mcp-refresh/srv") == "approve"
def test_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/api/_internal/mcp-reconnect/srv") == "approve"
def test_v1_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/v1/api/_internal/mcp-reconnect/srv") == "approve"
def test_proxy_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/node/n1/v1/api/_internal/mcp-reconnect/srv") == "approve"
def test_proxy_no_v1_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/node/n1/api/_internal/mcp-reconnect/srv") == "approve"
# Workstream sub-resource mutations (parametric paths)
def test_ws_delete_needs_write(self):
assert required_scope("POST", "/api/workstreams/abc123/delete") == "write"
-266
View File
@@ -1,266 +0,0 @@
"""Tests for ``turnstone.server._build_history`` reminder + source surfacing.
The replay path (``_build_history``) projects the ``_source`` and
``_reminders`` side-channels onto the wire entry the frontend
consumes. Persisted via migration 050 (Commit 1) so multi-tab /
multi-device replay sees the same metacognitive bubble shape the
originating tab saw live.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
from unittest.mock import patch
from turnstone.server import _build_history
def _make_stub_session(messages: list[dict[str, Any]]) -> Any:
"""Minimal ChatSession-shaped stub. ``_build_history`` only reads
``session.messages`` plus calls ``_load_verdict_indexes(ws_id)``
the latter we patch out below.
"""
return SimpleNamespace(messages=messages, _ws_id="ws-test")
def _build(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Run ``_build_history`` against a stub session, bypassing the
verdicts / output-assessment storage round-trip (no tool_calls in
these tests, so the indexes are unused anyway).
"""
session = _make_stub_session(messages)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
return _build_history(session)
class TestSourceSurfacing:
def test_source_surfaces_when_set(self) -> None:
msg = {
"role": "user",
"content": "",
"_source": "system_nudge",
}
history = _build([msg])
assert len(history) == 1
assert history[0]["source"] == "system_nudge"
def test_source_absent_when_unset(self) -> None:
msg = {"role": "user", "content": "hello"}
history = _build([msg])
assert "source" not in history[0]
class TestRemindersWidening:
def test_watch_triggered_optional_fields_propagate(self) -> None:
"""The widened payload (Commit 2) carries watch_name / command /
poll_count / max_polls / is_final on each ``watch_triggered``
reminder so the frontend renders ``.msg.watch-result``.
"""
msg = {
"role": "user",
"content": "",
"_source": "system_nudge",
"_reminders": [
{
"type": "watch_triggered",
"text": "$ ls\nfile.txt",
"watch_name": "w1",
"command": "ls",
"poll_count": 2,
"max_polls": 100,
"is_final": False,
}
],
}
history = _build([msg])
assert history[0]["source"] == "system_nudge"
assert history[0]["reminders"] == [
{
"type": "watch_triggered",
"text": "$ ls\nfile.txt",
"watch_name": "w1",
"command": "ls",
"poll_count": 2,
"max_polls": 100,
"is_final": False,
}
]
def test_legacy_two_field_reminders_still_work(self) -> None:
"""Producers without optional fields (correction / denial /
idle_children) keep the legacy ``{type, text}`` shape the
widened filter just doesn't add anything beyond that."""
msg = {
"role": "user",
"content": "noted",
"_reminders": [{"type": "correction", "text": "watch out"}],
}
history = _build([msg])
assert history[0]["reminders"] == [{"type": "correction", "text": "watch out"}]
def test_unknown_keys_are_dropped(self) -> None:
"""The wire-layer filter projects on a known set of keys so a
future producer accidentally stuffing arbitrary fields can't
leak them through replay.
"""
msg = {
"role": "user",
"content": "x",
"_reminders": [
{
"type": "correction",
"text": "hi",
"secret": "leak-me",
"internal_id": 42,
}
],
}
history = _build([msg])
clean = history[0]["reminders"][0]
assert "secret" not in clean
assert "internal_id" not in clean
assert clean == {"type": "correction", "text": "hi"}
def test_malformed_reminder_skipped(self) -> None:
"""A non-dict / empty entry is filtered out instead of breaking
the rest of the list (mirrors the defensive filter in
``_apply_reminders_for_provider``).
"""
msg = {
"role": "user",
"content": "x",
"_reminders": [
"garbage string",
{"type": "", "text": ""}, # empty type + text → drop
{"type": "denial", "text": "ok"},
],
}
history = _build([msg])
assert history[0]["reminders"] == [{"type": "denial", "text": "ok"}]
class _StubRegistry:
"""Minimal model registry — only ``get_config`` is read by
``_build_history``."""
def __init__(self, surface_persisted_reasoning: bool = True) -> None:
self._cfg = SimpleNamespace(surface_persisted_reasoning=surface_persisted_reasoning)
def get_config(self, alias: str) -> Any:
return self._cfg
def _build_with_registry(
messages: list[dict[str, Any]],
surface_persisted_reasoning: bool = True,
) -> list[dict[str, Any]]:
session = SimpleNamespace(
messages=messages,
_ws_id="ws-test",
_registry=_StubRegistry(surface_persisted_reasoning=surface_persisted_reasoning),
_model_alias="claude-opus-4-7",
)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
return _build_history(session)
class TestReasoningSurfacing:
"""Phase 1 — surface stored Anthropic thinking blocks on the
history payload so refresh-the-page rehydrates the reasoning bubble.
Drives through the real ``AnthropicProvider`` extractor (no mock-of-
extractor) only the model registry is stubbed.
"""
def test_reasoning_surfaces_for_anthropic_thinking_msg(self) -> None:
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "s"},
{"type": "text", "text": "Final answer."},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert len(history) == 1
assert history[0]["reasoning"] == "let me think"
def test_reasoning_empty_when_persist_flag_false(self) -> None:
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "hidden", "signature": "s"},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=False)
assert "reasoning" not in history[0]
def test_provider_content_never_in_wire_entry(self) -> None:
# The build path does not copy ``_provider_content`` into the
# entry dict regardless of flag — wire payload stays tight.
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "x", "signature": "s"},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert "_provider_content" not in history[0]
def test_no_reasoning_field_when_provider_content_missing(self) -> None:
msg = {"role": "assistant", "content": "plain answer"}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert "reasoning" not in history[0]
def test_no_reasoning_field_for_non_assistant_messages(self) -> None:
# Defensive — user/tool messages with a stray _provider_content
# do not get the reasoning field stamped.
msgs: list[dict[str, Any]] = [
{"role": "user", "content": "hi"},
{
"role": "tool",
"tool_call_id": "c1",
"content": "out",
"_provider_content": [{"type": "thinking", "thinking": "leak", "signature": "s"}],
},
]
history = _build_with_registry(msgs, surface_persisted_reasoning=True)
assert "reasoning" not in history[0]
assert "reasoning" not in history[1]
def test_default_true_when_registry_lookup_raises(self) -> None:
# Conservative default — Phase 1 spec mandates rehydration on
# refresh. A registry/alias mismatch must not silently kill the
# bubble.
class BrokenRegistry:
def get_config(self, alias: str) -> Any:
raise KeyError(alias)
session = SimpleNamespace(
messages=[
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "still works", "signature": "s"}
],
}
],
_ws_id="ws-test",
_registry=BrokenRegistry(),
_model_alias="missing-alias",
)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
history = _build_history(session)
assert history[0]["reasoning"] == "still works"
-112
View File
@@ -19,12 +19,6 @@ class NullUI:
self.infos = []
self.stream_ends = 0
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
@@ -826,109 +820,3 @@ class TestForceCancelThreaded:
assert "idle" in ui.states
assistant_msgs = [m for m in session.messages if m["role"] == "assistant"]
assert any("Fresh response" in m.get("content", "") for m in assistant_msgs)
class TestSynthesizeCancelledResults:
"""Regression coverage for ``_synthesize_cancelled_results`` — must
fire ``on_tool_result`` for each synthesized cancellation so live
SSE listeners (e.g. coord's ``--running`` indicator added by
tool_info) can complete the in-DOM tool batch. Without this, the
coord JS would spin the running indicator forever on cancelled
batches because ``state_change`` doesn't strip ``--running`` from
individual batches."""
def _ui_with_tool_result_tracking(self):
class _TrackingUI(NullUI):
def __init__(self) -> None:
super().__init__()
self.tool_results: list[tuple[str, str, str, bool]] = []
def on_tool_result(self, call_id, name, output, **kwargs):
self.tool_results.append(
(call_id, name, output, bool(kwargs.get("is_error", False))),
)
return _TrackingUI()
def test_synthesizes_tool_result_for_unanswered_calls(self, tmp_db):
ui = self._ui_with_tool_result_tracking()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"content": "calling tools",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
session._synthesize_cancelled_results("Cancelled by user.")
# Both unanswered calls fired ``on_tool_result``.
assert len(ui.tool_results) == 2
ids = {tr[0] for tr in ui.tool_results}
assert ids == {"call_a", "call_b"}
# All emitted as errors so the live UI renders them as
# ``coord-tool-row-result--error``.
assert all(tr[3] is True for tr in ui.tool_results)
# Reason text propagates as the synthetic tool output.
assert all(tr[2] == "Cancelled by user." for tr in ui.tool_results)
# And the message list has the synthesized tool entries
# (preserves the prior contract).
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
assert len(tool_msgs) == 2
def test_skips_calls_already_answered(self, tmp_db):
ui = self._ui_with_tool_result_tracking()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
# call_a already answered.
session.messages.append(
{"role": "tool", "tool_call_id": "call_a", "content": "result"},
)
session._msg_tokens.append(1)
session._synthesize_cancelled_results("Cancelled by user.")
# Only call_b synthesized.
assert len(ui.tool_results) == 1
assert ui.tool_results[0][0] == "call_b"
def test_ui_emit_failure_does_not_break_synthesis(self, tmp_db):
"""The UI hook is wrapped in try/except — a hook failure
during cancel must NOT compound the problem. Synthesis still
appends to messages + storage."""
class _ExplodingUI(NullUI):
def on_tool_result(self, call_id, name, output, **kwargs):
raise RuntimeError("ui hook blew up")
ui = _ExplodingUI()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
# Must not raise.
session._synthesize_cancelled_results("Cancelled by user.")
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
assert len(tool_msgs) == 1
@@ -1,66 +0,0 @@
"""ChatSession interactivity flag tests (Phase 9).
Validates that ``ChatSession._is_interactive_for_consent`` is computed
correctly from ``client_type`` on construction. This is the front of
the Phase 9 plumb-through: the flag flows from here to
``_dispatch_pool_sync`` to the structured-error pending-consent
write path.
"""
from __future__ import annotations
from tests._session_helpers import make_session
from turnstone.prompts import INTERACTIVE_CONSENT_CLIENT_TYPES, ClientType
def test_web_is_interactive() -> None:
s = make_session(client_type=ClientType.WEB)
assert s._is_interactive_for_consent is True
def test_cli_is_interactive() -> None:
s = make_session(client_type=ClientType.CLI)
assert s._is_interactive_for_consent is True
def test_chat_is_not_interactive() -> None:
# Discord / Slack adapters cannot drive a browser redirect from
# inside the channel — consent prompts must be deferred to the
# dashboard badge.
s = make_session(client_type=ClientType.CHAT)
assert s._is_interactive_for_consent is False
def test_scheduled_is_not_interactive() -> None:
# The scheduler runs autonomously; the user isn't online to
# complete the OAuth redirect.
s = make_session(client_type=ClientType.SCHEDULED)
assert s._is_interactive_for_consent is False
def test_interactive_set_matches_module_constant() -> None:
# Pin the module-level frozenset against the flag computation —
# a future reorganisation that drifts the set vs the per-session
# logic would silently break the gating.
for ct in ClientType:
s = make_session(client_type=ct)
assert s._is_interactive_for_consent == (ct in INTERACTIVE_CONSENT_CLIENT_TYPES), ct
def test_default_client_type_is_cli_interactive() -> None:
# Defaults preserved — make_session uses ChatSession's default
# which is CLI. Sanity check that the default user experience
# stays interactive-for-consent.
s = make_session()
assert s._client_type == ClientType.CLI
assert s._is_interactive_for_consent is True
def test_scheduled_env_file_exists() -> None:
"""The SCHEDULED env module must exist; otherwise
``compose_system_message`` for a scheduled session would 500."""
from turnstone.prompts import _load
text = _load("env/scheduled.md")
assert "Output Environment" in text
assert "consent" in text.lower()
-215
View File
@@ -1,215 +0,0 @@
"""Unit tests for :class:`turnstone.core.child_event_bus.ChildEventBus`.
The bus is the in-process wakeup primitive for ``wait_for_workstream``
(see :mod:`turnstone.console.coordinator_client`). It's a small dict
of ws_id set[threading.Event] under a lock focused tests for
register/notify symmetry, no-subscriber notify, multi-waiter fan-out,
multi-child waiter, and concurrent register/notify (smoke). End-to-end
integration with the dispatch sink lives in
``test_coordinator_adapter.py`` and ``test_coordinator_client.py``.
"""
from __future__ import annotations
import threading
import time
import pytest
from turnstone.core.child_event_bus import ChildEventBus
def test_register_returns_event_that_starts_unset() -> None:
"""A waiter must not see leftover state from before it registered —
a fresh wait should always block until the first notify."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
assert isinstance(event, threading.Event)
assert not event.is_set()
def test_notify_wakes_waiter_on_matching_ws_id() -> None:
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify("ws-1")
assert event.is_set()
def test_notify_does_not_wake_waiter_on_unrelated_ws_id() -> None:
"""Different ws_ids must keep independent waiter sets — a notify on
a stranger ws can't wake the wait or the bus stops being keyed."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify("ws-other")
assert not event.is_set()
def test_notify_with_no_subscribers_is_noop() -> None:
"""The dispatch sink calls notify on every translated event; the
steady state has no wait tool active. Must not raise."""
bus = ChildEventBus()
bus.notify("ws-nobody-cares") # no exception
def test_multi_waiter_each_gets_independent_event() -> None:
"""Two waits on the same ws_id must wake independently — clearing
one Event must not silence the other."""
bus = ChildEventBus()
e1 = bus.register_waiter(["ws-1"])
e2 = bus.register_waiter(["ws-1"])
assert e1 is not e2
bus.notify("ws-1")
assert e1.is_set()
assert e2.is_set()
def test_multi_child_waiter_fires_on_any_listed_ws_id() -> None:
"""A wait on [A, B, C] returns a single Event registered against
all three. Notify on ANY of A/B/C must wake the wait the
caller's snapshot re-read disambiguates which one changed."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-a", "ws-b", "ws-c"])
bus.notify("ws-b")
assert event.is_set()
def test_unregister_removes_event_from_all_listed_ws_ids() -> None:
"""After unregister, notify on any of the previously-watched ws_ids
must NOT wake the Event leaks would mean every future notify on
that ws_id wakes a long-dead wait."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-a", "ws-b"])
bus.unregister_waiter(["ws-a", "ws-b"], event)
bus.notify("ws-a")
bus.notify("ws-b")
assert not event.is_set()
def test_unregister_is_idempotent() -> None:
"""A double-unregister must silently no-op — finally blocks may
run twice in odd shutdown paths, the bus must not raise."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.unregister_waiter(["ws-1"], event)
bus.unregister_waiter(["ws-1"], event) # no exception
def test_unregister_pops_empty_buckets() -> None:
"""Empty per-ws_id buckets must be popped so a long-lived bus
doesn't accumulate dead keys after many waits have churned through.
Reaches into the private state the property is structural, not
behavioral, so the assertion is also."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
assert "ws-1" in bus._waiters
bus.unregister_waiter(["ws-1"], event)
assert "ws-1" not in bus._waiters
def test_unregister_keeps_bucket_with_remaining_waiters() -> None:
"""Removing one waiter from a multi-waiter bucket must not drop
the others popping the bucket would silently disable notifies
for every concurrent wait on the same ws_id."""
bus = ChildEventBus()
e1 = bus.register_waiter(["ws-1"])
e2 = bus.register_waiter(["ws-1"])
bus.unregister_waiter(["ws-1"], e1)
bus.notify("ws-1")
assert not e1.is_set()
assert e2.is_set()
def test_empty_and_falsy_ws_ids_are_skipped_on_register() -> None:
"""Defensive: ``wait_for_workstream`` cleans its inputs but the bus
is reachable from other callers in future use; falsy ids should be
silently dropped, not registered against an empty-string key."""
bus = ChildEventBus()
event = bus.register_waiter(["", "ws-1", ""])
# Only the real ws_id should bucket the waiter.
assert list(bus._waiters.keys()) == ["ws-1"]
bus.notify("") # no crash, no spurious wake
assert not event.is_set()
bus.notify("ws-1")
assert event.is_set()
def test_notify_wakes_waiter_blocking_on_event_wait() -> None:
"""End-to-end wake-up latency: a wait blocked on ``Event.wait``
must return promptly after a notify on a watched ws_id. This is
the property that retires the 0.5s polling cadence."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
woken_at = [0.0]
def _waiter() -> None:
event.wait(timeout=2.0)
woken_at[0] = time.monotonic()
t = threading.Thread(target=_waiter, daemon=True)
t.start()
# Give the waiter a beat to enter Event.wait, then notify.
time.sleep(0.05)
notified_at = time.monotonic()
bus.notify("ws-1")
t.join(timeout=1.0)
assert not t.is_alive(), "waiter did not wake within 1s of notify"
# Latency budget is generous; the contract is "well under the legacy
# 0.5s poll cadence", not microsecond timing.
assert woken_at[0] - notified_at < 0.2
def test_clear_before_check_race_does_not_lose_wake() -> None:
"""The wait-loop pattern is ``clear(); snapshot(); ...; wait()``.
A notify between clear and wait must leave the Event set, so the
next wait returns immediately and the loop re-snapshots. Same
standard subscribe/check race the wait loop guards against."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
# Simulate wait-loop ordering: clear, then notify "between" clear
# and the next wait.
event.clear()
bus.notify("ws-1")
# The next wait must return True immediately (set is sticky until
# the next clear).
assert event.wait(timeout=0.1) is True
def test_concurrent_register_and_notify_is_safe() -> None:
"""Smoke test: many threads registering / notifying / unregistering
in parallel must not raise or deadlock. Doesn't assert specific
interleavings only structural safety of the lock discipline."""
bus = ChildEventBus()
stop = threading.Event()
errors: list[BaseException] = []
def _worker(ws_id: str) -> None:
try:
for _ in range(200):
if stop.is_set():
return
ev = bus.register_waiter([ws_id])
bus.notify(ws_id)
bus.unregister_waiter([ws_id], ev)
except BaseException as e: # noqa: BLE001
errors.append(e)
threads = [threading.Thread(target=_worker, args=(f"ws-{i}",), daemon=True) for i in range(8)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5.0)
stop.set()
assert not errors, f"worker threads raised: {errors!r}"
# All buckets should have been popped (every register paired with
# unregister).
assert bus._waiters == {}
@pytest.mark.parametrize("ws_id", ["", None])
def test_notify_silently_ignores_falsy_ws_id(ws_id: object) -> None:
"""Defensive: the dispatch sink already guards against empty
ws_ids, but a falsy slip-through must not raise."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify(ws_id) # type: ignore[arg-type]
assert not event.is_set()
-35
View File
@@ -50,41 +50,6 @@ def test_load_config_invalid_toml(tmp_path):
assert load_config() == {}
def test_load_config_warns_when_world_readable(tmp_path, caplog):
"""Secrets in config.toml — warn if anyone but the owner can read it."""
import logging
import os
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
os.chmod(cfg, 0o644)
set_config_path(str(cfg))
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
load_config()
messages = [r.getMessage() for r in caplog.records]
assert any("group/world-readable" in m for m in messages)
def test_load_config_quiet_when_mode_0600(tmp_path, caplog):
import logging
import os
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
os.chmod(cfg, 0o600)
set_config_path(str(cfg))
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
load_config()
messages = [r.getMessage() for r in caplog.records]
assert not any("group/world-readable" in m for m in messages)
def test_load_config_caches(tmp_path):
_reset_cache()
cfg = tmp_path / "config.toml"
-210
View File
@@ -3,13 +3,11 @@
import asyncio
import json
import queue
from typing import Any
from unittest.mock import MagicMock
import pytest
from turnstone.console.collector import ClusterCollector, NodeSnapshot
from turnstone.console.server import _PROXY_AUTH_LOCAL_HANDLERS
# Shared test auth — JWT-based
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
@@ -151,78 +149,6 @@ class TestCollectorDiscovery:
assert c._nodes["node-a"].started == 1234567890.0
class TestCollectorNotifyWireIn:
"""NotifyDispatcher-driven discovery — reactive node visibility."""
def test_start_subscribes_to_services_channel(self):
# Stub dispatcher records subscriptions without spawning threads.
class _StubDispatcher:
def __init__(self):
self.subscriptions: list[tuple[str, Any]] = []
def subscribe(self, channel, handler):
self.subscriptions.append((channel, handler))
return lambda: None
stub = _StubDispatcher()
storage = MockStorage()
c = ClusterCollector(
storage=storage,
discovery_interval=999,
notify_dispatcher=stub,
)
try:
c.start()
assert len(stub.subscriptions) == 1
channel, handler = stub.subscriptions[0]
assert channel == "services"
assert handler == c._on_services_notify
finally:
c.stop()
def test_no_dispatcher_means_no_subscribe(self):
# Collector without a dispatcher (single-node / SQLite dev) just
# falls back to the 60 s discovery-loop polling — no error.
c = _make_collector(MockStorage())
try:
c.start()
assert c._notify_unsubscribe is None
finally:
c.stop()
def test_on_notify_runs_discovery(self):
# Construct a synthetic Notify and invoke the handler directly —
# asserts the wire-in delegates back to ``_discover_nodes``.
from turnstone.core.storage._notify import Notify
storage = MockStorage()
c = _make_collector(storage)
c._running = True # bypass start() so we don't spawn threads
q: queue.Queue[dict[str, Any]] = queue.Queue()
c.register_listener(q)
storage.services = [
{"service_id": "node-z", "url": "http://z:8080", "metadata": "{}"},
]
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
event = q.get_nowait()
assert event["type"] == "node_joined"
assert event["node_id"] == "node-z"
def test_on_notify_when_not_running_is_noop(self):
# If a stray notify arrives after stop, the handler doesn't run
# discovery on a half-torn-down collector.
from turnstone.core.storage._notify import Notify
storage = MockStorage()
storage.services = [{"service_id": "node-y", "url": "http://y:8080", "metadata": "{}"}]
c = _make_collector(storage)
# _running stays False (never called start()).
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
assert c.get_overview()["nodes"] == 0
class TestCollectorSnapshot:
"""Applying node_snapshot SSE events."""
@@ -1563,142 +1489,6 @@ class TestConsoleProxy:
assert sse_mock.await_count == 1
assert sse_mock.await_args.kwargs.get("use_service_auth") is False
# -------------------------------------------------------------------
# Proxied auth endpoints — handled locally by the console, not
# forwarded to the upstream node. Cases derive directly from
# ``_PROXY_AUTH_LOCAL_HANDLERS`` so a new dispatch entry can't be
# added without a matching test (or vice versa). See proxy_api's
# docstring for the JWT-audience reasoning.
# -------------------------------------------------------------------
@pytest.mark.parametrize(
("method", "path", "handler_name"),
[
(method, path, handler_name)
for (method, path), handler_name in sorted(_PROXY_AUTH_LOCAL_HANDLERS.items())
],
)
def test_proxy_auth_endpoint_dispatches_to_local_handler(
self, client, method, path, handler_name
):
"""Every entry in ``_PROXY_AUTH_LOCAL_HANDLERS`` must route to its
local console handler and never reach the upstream proxy. The
lockout class of bug this dispatch was added to fix is exactly
what a regression here would reintroduce silently covering all
eight branches keeps each path tied to its handler."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
with (
patch(
f"turnstone.console.server.{handler_name}",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "ok"}),
) as local_mock,
patch(
"turnstone.console.server._proxy_post",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as post_mock,
patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as get_mock,
):
resp = client.request(method, f"/node/node-a/v1/api/{path}")
assert resp.status_code == 200
assert local_mock.await_count == 1
assert post_mock.await_count == 0
assert get_mock.await_count == 0
def test_proxy_auth_login_works_without_cookie(self, mock_collector):
"""Without this fix the AuthMiddleware 401s before any handler
runs the user is locked out of the proxied UI once the cookie
expires. Test bypasses _TEST_AUTH_HEADERS to reproduce."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
from starlette.testclient import TestClient
from turnstone.console.server import _load_static, create_app
_load_static()
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
unauth_client = TestClient(app, raise_server_exceptions=False)
try:
with patch(
"turnstone.console.server.auth_login",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "ok"}),
) as local_mock:
resp = unauth_client.post(
"/node/node-a/v1/api/auth/login",
json={"username": "x", "password": "y"},
)
# AuthMiddleware must classify the proxied login path as
# public (is_public_path change) AND proxy_api must
# dispatch to the local handler (proxy_api change).
assert resp.status_code == 200, (
f"login locked out: got {resp.status_code}, body={resp.text}"
)
assert local_mock.await_count == 1
finally:
unauth_client.close()
def test_proxy_auth_wrong_method_returns_405_not_forwarded(self, client):
"""A non-canonical method on an auth path (e.g. PUT on auth/login)
must short-circuit with 405 instead of falling through to the
upstream proxy falling through would forward the request
authenticated as the console's service token (``_proxy_auth_headers``
fallback)."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
with (
patch(
"turnstone.console.server._proxy_post",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as post_mock,
patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as get_mock,
):
# PUT on a POST-only auth path → 405
put_resp = client.put("/node/node-a/v1/api/auth/login")
assert put_resp.status_code == 405
# POST on a GET-only auth path → 405
post_resp = client.post("/node/node-a/v1/api/auth/status")
assert post_resp.status_code == 405
assert post_mock.await_count == 0
assert get_mock.await_count == 0
def test_proxy_non_auth_endpoint_still_forwarded(self, client, mock_collector):
"""Sanity: only auth/* paths intercept. Other API paths still
forward to the upstream node."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
mock_collector.get_node_detail.return_value = {
"node_id": "node-a",
"server_url": "http://a:8080",
"reachable": True,
}
with patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"ok": True}),
) as proxy_mock:
resp = client.get("/node/node-a/v1/api/workstreams")
assert resp.status_code == 200
assert proxy_mock.await_count == 1
# ---------------------------------------------------------------------------
# Proxy URL rewriting unit tests (no HTTP needed)
-345
View File
@@ -1,345 +0,0 @@
"""``GET /v1/api/models`` resolution-chain coverage.
The console handler resolves four defaults from settings + the enabled
model list:
* ``default_alias`` ``model.default_alias``
* ``channel_default_alias`` ``channels.default_model_alias``
* ``coordinator_default_alias`` ``coordinator.model_alias``, falling
back to ``default_alias`` when empty *or* pointing at a disabled /
removed alias (mirrors :mod:`turnstone.console.session_factory`).
* ``judge_default_alias`` ``judge.model``, falling back to the
resolved coordinator alias when empty *or* pointing at a value that
isn't an enabled alias. ``judge.model`` is alias-only — same
contract as the other model roles and
:class:`turnstone.core.judge.IntentJudge` silently inherits the
session model when an unknown value is configured, so the API
surfaces the resolved coordinator alias rather than echoing the
misconfigured string.
These tests pin each branch so the home composer's resolved-alias
placeholder stays correct as the precedence rules evolve.
"""
from __future__ import annotations
from typing import Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route
from starlette.testclient import TestClient
from tests._coord_test_helpers import _AuthMiddleware, _FakeConfigStore
from turnstone.console.server import list_available_models
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "available_models.db"))
def _seed_model(
storage: SQLiteBackend,
*,
definition_id: str,
alias: str,
model: str = "model-x",
enabled: bool = True,
) -> None:
storage.create_model_definition(
definition_id=definition_id,
alias=alias,
model=model,
provider="openai-compatible",
base_url="http://localhost:8000/v1",
api_key="sk-test",
context_window=8192,
capabilities="{}",
enabled=enabled,
created_by="admin",
)
class _StubRegistry:
"""Mimics the surface ``resolve_coordinator_alias`` reads from
``coord_registry``: ``.default`` and ``.has_alias()``.
Production wires this through ``ModelRegistry``, which in turn
pulls aliases from both DB rows and config.toml. The fixture
mirrors the storage's enabled-row set so ``has_alias()`` agrees
with what the placeholder's enabled-row filter would accept —
without that alignment the helper rejects every tier-2 candidate
and the placeholder goes blank in cases that production handles
fine."""
def __init__(self, *, default: str, known: set[str]) -> None:
self.default = default
self._known = known
def has_alias(self, alias: str) -> bool:
return alias in self._known
def _make_client(
storage: SQLiteBackend,
*,
settings: dict[str, str] | None = None,
registry_default: str = "",
config_store: bool = True,
) -> TestClient:
app = Starlette(
routes=[Route("/v1/api/models", list_available_models)],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.auth_storage = storage
if config_store:
app.state.config_store = _FakeConfigStore(dict(settings or {}))
# ``coord_registry`` is always set in production after lifespan
# startup; mirror that here. ``has_alias`` answers from the same
# enabled-rows set the handler filters against.
enabled = {r["alias"] for r in storage.list_model_definitions(enabled_only=True)}
app.state.coord_registry = _StubRegistry(default=registry_default, known=enabled)
client = TestClient(app)
client.headers.update({"X-Test-User": "admin", "X-Test-Perms": ""})
return client
def _get_models(client: TestClient) -> dict[str, Any]:
resp = client.get("/v1/api/models")
assert resp.status_code == 200, resp.text
return resp.json()
# ---------------------------------------------------------------------------
# Coordinator resolution
# ---------------------------------------------------------------------------
def test_no_settings_leaves_all_defaults_blank(storage: SQLiteBackend) -> None:
"""No model.default_alias, no per-role overrides → every default
field is empty and ``models`` is an empty list."""
body = _get_models(_make_client(storage))
assert body == {
"models": [],
"default_alias": "",
"channel_default_alias": "",
"coordinator_default_alias": "",
"judge_default_alias": "",
}
def test_coordinator_inherits_default_alias_when_unset(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, settings={"model.default_alias": "primary"}))
assert body["default_alias"] == "primary"
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_explicit_enabled_alias_passes_through(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "fast",
},
)
)
assert body["coordinator_default_alias"] == "fast"
def test_coordinator_set_to_disabled_alias_falls_back_to_default(
storage: SQLiteBackend,
) -> None:
"""Operator disabled the alias the coordinator was pinned to —
fall back to the registry default rather than advertising a model
that workstream creation would refuse to use."""
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="legacy", enabled=False)
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "legacy",
},
)
)
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_set_to_unknown_alias_falls_back_to_default(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "ghost",
},
)
)
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_falls_back_to_registry_default_when_config_store_empty(
storage: SQLiteBackend,
) -> None:
"""Match ``console/session_factory.py:109-110``: when both
``coordinator.model_alias`` and ``model.default_alias`` are unset, new
coordinator sessions run on ``registry.default`` (loaded from
config.toml ``[model].default``). The placeholder must report the
same alias rather than going blank otherwise the home composer
advertises "Default model" while sessions actually launch on a
concrete alias."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, registry_default="primary"))
assert body["default_alias"] == ""
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
def test_coordinator_skips_registry_default_when_alias_disabled(
storage: SQLiteBackend,
) -> None:
"""Registry default points at an alias that's been disabled in the DB
the placeholder stays blank rather than advertising a model that
workstream creation would refuse to use."""
_seed_model(storage, definition_id="m1", alias="legacy", enabled=False)
body = _get_models(_make_client(storage, registry_default="legacy"))
assert body["coordinator_default_alias"] == ""
def test_coordinator_falls_back_to_registry_default_when_config_store_missing(
storage: SQLiteBackend,
) -> None:
"""Edge case from PR #500 review: lifespan can leave
``app.state.config_store`` as None (e.g. a startup exception) while
``coord_registry`` still binds successfully. The placeholder must
still advertise ``registry.default`` (filtered against enabled rows)
rather than going blank otherwise the home composer is uselessly
empty in a degraded-but-recoverable state."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, registry_default="primary", config_store=False))
assert body["default_alias"] == ""
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
# ---------------------------------------------------------------------------
# Judge resolution
# ---------------------------------------------------------------------------
def test_judge_empty_inherits_resolved_coordinator_alias(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "fast",
},
)
)
assert body["coordinator_default_alias"] == "fast"
assert body["judge_default_alias"] == "fast"
def test_judge_explicit_enabled_alias_passes_through(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="judge-fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "judge-fast",
},
)
)
assert body["judge_default_alias"] == "judge-fast"
def test_judge_set_to_unknown_value_inherits_coordinator(
storage: SQLiteBackend,
) -> None:
"""``judge.model`` is alias-only — same contract as the other model
roles. An unknown value silently inherits the session model in
:class:`IntentJudge`, so the API surfaces the resolved coordinator
alias rather than echoing the misconfigured string."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "anthropic/claude-haiku-4-5", # raw, not an alias
},
)
)
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
def test_judge_set_to_disabled_alias_inherits_coordinator(
storage: SQLiteBackend,
) -> None:
"""Disabled-alias case is handled identically to the unknown-value
case both trip the alias-not-resolved path."""
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="judge-old", enabled=False)
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "judge-old",
},
)
)
assert body["judge_default_alias"] == "primary"
# ---------------------------------------------------------------------------
# Pre-existing fields stay correct under the new resolution code
# ---------------------------------------------------------------------------
def test_channel_default_alias_blanked_when_disabled(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary", enabled=False)
body = _get_models(
_make_client(
storage,
settings={"channels.default_model_alias": "primary"},
)
)
assert body["channel_default_alias"] == ""
def test_models_payload_strips_secret_fields(storage: SQLiteBackend) -> None:
"""Regression guard: only alias/model/provider land in the response,
never api_key / base_url / context_window / capabilities."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage))
assert body["models"] == [
{"alias": "primary", "model": "model-x", "provider": "openai-compatible"}
]
+29 -229
View File
@@ -5,15 +5,11 @@ lifting is in ``SessionManager.close_idle`` (covered in
``test_session_manager.py``) and ``bulk_close_stale_orphans`` (covered
in ``test_storage_sqlite.py``). These tests verify the glue:
- the helper runs an initial sweep BEFORE its first wait (cold-start
- the helper runs an initial sweep BEFORE its first sleep (cold-start
cleanup without blocking the lifespan),
- the helper swallows exceptions so a transient DB blip can't kill the
daemon thread,
- the helper exits cleanly when ``stop_event`` is set,
- the helper subscribes to ``mgr.subscribe_to_state`` and a state-change
event wakes the next sweep early (event-driven, not polling),
- the helper unsubscribes when the thread exits so the subscriber
doesn't leak past one cleanup-thread lifetime.
- the helper exits cleanly when ``stop_event`` is set.
The ``stop_event`` parameter is exclusively for tests production
callers pass ``None`` and the daemon runs for process lifetime.
@@ -21,36 +17,28 @@ callers pass ``None`` and the daemon runs for process lifetime.
from __future__ import annotations
import contextlib
import threading
import time
from typing import TYPE_CHECKING
from unittest.mock import patch
from turnstone.console.server import _coord_idle_cleanup_thread
if TYPE_CHECKING:
from collections.abc import Callable
class _StubMgr:
"""Minimal SessionManager substitute exposing only what the cleanup
thread touches: ``close_idle``, ``subscribe_to_state``,
``unsubscribe_from_state``. Records call ordering for assertions
and lets the test fire state-change events manually via
:meth:`fire_state_change`.
"""
def __init__(
self, *, stop_event: threading.Event, expected_calls: int, raise_after: int = -1
) -> None:
self.calls: list[float] = []
self.sleep_calls_at_each_close: list[int] = []
self._stop_event = stop_event
self._expected = expected_calls
self._raise_after = raise_after
self._subscribers: list[Callable[[str, object], None]] = []
self._sub_lock = threading.Lock()
self._sleep_count = 0
def close_idle(self, timeout_sec: float) -> list[str]:
# Snapshot how many sleeps preceded this close — lets the
# "initial sweep" test verify the first close_idle ran with
# zero preceding sleeps.
self.sleep_calls_at_each_close.append(self._sleep_count)
self.calls.append(timeout_sec)
try:
if 0 <= self._raise_after < len(self.calls):
@@ -62,78 +50,39 @@ class _StubMgr:
self._stop_event.set()
return []
def subscribe_to_state(self, callback: Callable[[str, object], None]) -> None:
with self._sub_lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Callable[[str, object], None]) -> None:
with self._sub_lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
@property
def subscribers_count(self) -> int:
with self._sub_lock:
return len(self._subscribers)
def fire_state_change(self, ws_id: str = "ws-x", state: object = "idle") -> None:
with self._sub_lock:
snapshot = list(self._subscribers)
for cb in snapshot:
cb(ws_id, state)
def record_sleep(self, _seconds: float) -> None:
self._sleep_count += 1
def _run_until_done(mgr: _StubMgr, stop_event: threading.Event, timeout_sec: float) -> None:
# ``min_sweep_interval=0.0`` disables the production cadence floor
# (default 5 s) so tests can fire many close_idle calls back-to-back
# without waiting real time between them. The floor is exercised
# in its own dedicated test below.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, timeout_sec, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "helper failed to exit on stop_event"
with patch("turnstone.console.server.time.sleep", mgr.record_sleep):
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, timeout_sec, stop_event),
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "helper failed to exit on stop_event"
def test_coord_idle_cleanup_runs_initial_sweep_before_wait() -> None:
"""The first close_idle call must happen BEFORE the first wait
def test_coord_idle_cleanup_runs_initial_sweep_before_sleep() -> None:
"""The first close_idle call must happen BEFORE the first time.sleep
otherwise cold-start orphans wait one ``check_every`` interval (~30 min
on default 2h timeout) for the first reap. Crucial because the
lifespan no longer does a synchronous initial sweep.
Verified structurally: a single ``expected_calls=1`` run completes
in well under one ``check_every`` (here 0.04 s timeout 0.01 s
check_every), so the initial sweep must have happened before any
real wait could have blocked it.
"""
lifespan no longer does a synchronous initial sweep."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
started = time.monotonic()
_run_until_done(mgr, stop_event, timeout_sec=0.04)
elapsed = time.monotonic() - started
assert len(mgr.calls) == 1
# check_every = min(300.0, 0.04/4) = 0.01 s. An initial sweep
# gated behind one full wait would have taken ~0.01+ s anyway, so
# the upper bound here is "much less than one check_every plus
# process noise" — the explicit 1.0 s gives generous CI headroom
# while still asserting the test is testing the right thing.
assert elapsed < 1.0
_run_until_done(mgr, stop_event, timeout_sec=120.0)
assert mgr.sleep_calls_at_each_close == [0], "first close_idle should run before any sleep"
def test_coord_idle_cleanup_calls_close_idle_each_tick() -> None:
"""Heartbeat path: with no state-change events, close_idle fires
each ``check_every`` interval. Test uses a tiny timeout so the
test runs fast the contract under test is "the loop iterates",
not the production cadence.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
assert len(mgr.calls) == 3
assert all(t == 0.04 for t in mgr.calls)
assert all(t == 120.0 for t in mgr.calls)
def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
@@ -142,7 +91,7 @@ def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
blip would silently leak orphans forever."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=4, raise_after=1)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
# All four calls must have fired despite calls 2-4 raising.
assert len(mgr.calls) == 4
@@ -153,154 +102,5 @@ def test_coord_idle_cleanup_exits_cleanly_on_stop_event() -> None:
daemon-process termination."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
assert stop_event.is_set()
def test_state_change_wakes_close_idle_before_heartbeat() -> None:
"""The event-driven path is the whole point of the refactor: a
workstream state-change must wake the cleanup sweep without
waiting one ``check_every`` interval. Tested with a long
timeout_sec so the heartbeat would NOT have fired in the test
window the close_idle call past the initial sweep must come
from a state-change wake.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
# check_every = min(300.0, 120.0/4) = 30 s — well outside the test
# window. Any close_idle call past the initial sweep must come
# from a fire_state_change-driven wake-up.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
# Wait for the initial sweep to complete AND the thread to enter
# its first ``tick_now.wait`` (signalled here by the subscriber
# being registered + calls advancing to 1).
deadline = time.monotonic() + 1.0
while time.monotonic() < deadline:
if mgr.subscribers_count == 1 and len(mgr.calls) >= 1:
break
time.sleep(0.01)
assert mgr.subscribers_count == 1, "thread didn't subscribe to state"
assert len(mgr.calls) == 1, "initial sweep didn't fire"
# One state-change fire wakes the first ``wait`` → close_idle runs
# again → stop_event is set (expected_calls=2) → thread exits.
mgr.fire_state_change()
thread.join(timeout=2.0)
assert not thread.is_alive(), "thread didn't exit after state-change-driven sweep"
# 2 = initial + state-change-driven. If the state change weren't
# being honoured, close_idle would have stalled on the 30 s wait
# and the thread.join would have timed out.
assert len(mgr.calls) == 2
def test_subscriber_unregisters_when_thread_exits() -> None:
"""The cleanup thread's state-change subscriber must be removed
when the thread exits otherwise long-running processes that
restart their cleanup threads (admin model-CRUD path, tests) leak
subscribers and every state change fires N stale callbacks.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert mgr.subscribers_count == 0, "subscriber leaked past thread exit"
def test_state_change_during_close_idle_triggers_followup_sweep() -> None:
"""A state-change fired during the initial sweep (e.g. close_idle's
own ``close()`` calls firing subscribers) must wake the next
``tick_now.wait`` rather than being lost to the clear-before-sweep
ordering. The clear runs INSIDE the loop just before close_idle,
so a fire during the initial sweep which precedes the loop
arrives at an already-set event that the first wait sees set and
returns on immediately.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
real_close_idle = mgr.close_idle
# One-shot fire during the initial sweep, mirroring what
# close_idle's own close() calls do in production (set_state →
# state-change subscribers).
fired = [False]
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
result = real_close_idle(timeout_sec)
if not fired[0]:
fired[0] = True
mgr.fire_state_change()
return result
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "thread blocked on the next wait — mid-sweep wake was lost"
# 2 = initial sweep + state-change-driven follow-up. Without the
# event surviving the clear-before-sweep ordering, the thread
# would have blocked on the 30 s ``wait`` and the test would have
# timed out at thread.join.
assert len(mgr.calls) == 2
def test_min_sweep_interval_floors_close_idle_cadence_under_sustained_wakes() -> None:
"""Cadence floor: even when state-change events keep firing
``tick_now.set()``, ``close_idle`` must not run more often than
``min_sweep_interval`` otherwise the loop tight-spins close_idle
at the rate of its own DB latency, doing 600-1500x more DB work
than the pre-refactor fixed-30 s cadence.
Wires a state-change subscriber that fires another state change
from inside close_idle, so the bus would tick forever if not
floored. Asserts the elapsed-between-sweeps is at least
``min_sweep_interval`` modulo small wall-clock noise.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
real_close_idle = mgr.close_idle
sweep_times: list[float] = []
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
sweep_times.append(time.monotonic())
result = real_close_idle(timeout_sec)
# Always fire another state-change to simulate sustained
# activity (each turn fires thinking/running/attention/idle).
# If the floor were absent, the next wake would race the next
# close_idle immediately and ``sweep_times`` deltas would be
# bounded by close_idle latency (microseconds), not the floor.
mgr.fire_state_change()
return result
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
# 0.15 s floor keeps the test fast (~0.3 s total) while still
# representing a meaningful gap relative to close_idle's
# near-zero stub latency.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.15},
daemon=True,
)
thread.start()
thread.join(timeout=3.0)
assert not thread.is_alive(), "thread didn't exit"
assert len(sweep_times) >= 2, "fewer than two sweeps fired"
# Gap between sweep 1 (post-initial) and sweep 2 must respect
# the floor. Initial sweep at sweep_times[0] is unfloored
# (no prior sweep to compare against), so the meaningful
# assertion is on sweep_times[1] - sweep_times[0].
gap = sweep_times[1] - sweep_times[0]
assert gap >= 0.12, f"floor breached: gap {gap:.3f}s < min_sweep_interval 0.15s"
-219
View File
@@ -1,219 +0,0 @@
"""``console/session_factory.py`` alias-resolution coverage.
The console session factory resolves the coordinator alias through a
three-tier chain that must stay in lockstep with the placeholder logic
in ``console/server.py:list_available_models`` otherwise the home
composer advertises one alias while sessions launch on another.
Tier order (highest priority first):
1. Per-call ``model_alias`` arg, or the ``coordinator.model_alias``
ConfigStore setting (admin-pinned coordinator-specific override).
2. ``model.default_alias`` ConfigStore setting (admin-managed system
default surfaced in the Models tab).
3. ``registry.default`` (config.toml ``[model].default``, the boot-time
fallback).
These tests pin each branch by intercepting ``registry.resolve``
they short-circuit before ChatSession construction so the test never
has to satisfy ChatSession's full kwarg contract.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
import pytest
from tests._coord_test_helpers import _FakeConfigStore
from turnstone.console.session_factory import build_console_session_factory
class _StopBeforeChatSessionError(Exception):
"""Sentinel raised by the capturing registry to short-circuit
factory execution after alias resolution but before ChatSession is
built. The factory's outer code path is irrelevant to alias
resolution and would force the test to satisfy a long kwarg
contract for no extra coverage."""
class _CapturingRegistry:
"""Records the alias passed to ``resolve()`` and short-circuits.
``has_alias`` answers from the configured known set so the
``model.default_alias`` validation tier behaves realistically.
Mirrors the public surface ``ModelRegistry`` exposes to
session_factory: ``has_alias``, ``resolve``, and ``default``.
"""
def __init__(self, *, default: str, known: set[str]) -> None:
self.default = default
self._known = known
self.captured_alias: str | None = None
def has_alias(self, alias: str) -> bool:
return alias in self._known
def resolve(self, alias: str) -> Any:
self.captured_alias = alias
raise _StopBeforeChatSessionError()
def _build_factory(
*,
registry_default: str = "registry-default",
known_aliases: set[str] | None = None,
settings: dict[str, Any] | None = None,
) -> tuple[Any, _CapturingRegistry]:
"""Construct the factory with stub deps. Returns ``(factory_callable,
registry)`` so tests can read back ``registry.captured_alias``."""
registry = _CapturingRegistry(
default=registry_default,
known=known_aliases if known_aliases is not None else {registry_default},
)
config_store = _FakeConfigStore(dict(settings or {}))
factory = build_console_session_factory(
registry=registry, # type: ignore[arg-type]
config_store=config_store, # type: ignore[arg-type]
node_id="console",
coord_client_factory=lambda ws_id, uid: MagicMock(),
)
return factory, registry
def _invoke(factory: Any, **factory_kwargs: Any) -> None:
"""Call the factory with a stub UI and absorb the sentinel.
Forwards ``factory_kwargs`` to the factory so per-call overrides
(e.g. ``model_alias``) can flow through. Raises if any other
exception comes out the test should fail loudly when alias
resolution itself errors rather than swallowing it.
"""
ui = MagicMock()
ui._user_id = "" # skip storage-backed username lookup branch
with pytest.raises(_StopBeforeChatSessionError):
factory(ui, **factory_kwargs)
# ---------------------------------------------------------------------------
# Tier 1 — explicit pin (per-call arg or coordinator.model_alias)
# ---------------------------------------------------------------------------
def test_per_call_model_alias_arg_wins_over_everything() -> None:
"""The ``model_alias`` kwarg on the factory call (e.g. body field on
POST /workstreams/new) wins over both ConfigStore tiers and the
registry default."""
factory, registry = _build_factory(
known_aliases={"per-call", "coord-pin", "admin-default", "registry-default"},
settings={
"coordinator.model_alias": "coord-pin",
"model.default_alias": "admin-default",
},
)
_invoke(factory, model_alias="per-call")
assert registry.captured_alias == "per-call"
def test_coordinator_model_alias_wins_when_no_per_call_override() -> None:
factory, registry = _build_factory(
known_aliases={"coord-pin", "admin-default", "registry-default"},
settings={
"coordinator.model_alias": "coord-pin",
"model.default_alias": "admin-default",
},
)
_invoke(factory)
assert registry.captured_alias == "coord-pin"
def test_coordinator_model_alias_passed_through_unvalidated() -> None:
"""Tier 1 is an *explicit* operator pin — when it's stale or typoed
we deliberately pass it through to ``registry.resolve`` so the
request layer turns it into a 503 with the alias surfaced in the
error. Falling through silently would mask the misconfiguration."""
factory, registry = _build_factory(
known_aliases={"admin-default", "registry-default"},
settings={
"coordinator.model_alias": "ghost", # unknown
"model.default_alias": "admin-default",
},
)
_invoke(factory)
assert registry.captured_alias == "ghost"
def test_per_call_model_alias_arg_passed_through_unvalidated() -> None:
"""The per-call ``model_alias`` kwarg (POST body field — the more
common production trigger) is the same kind of explicit pin as the
ConfigStore setting, so a stale value passes through to
``registry.resolve`` rather than silently falling through to the
system default."""
factory, registry = _build_factory(
known_aliases={"registry-default"},
settings={"model.default_alias": "registry-default"},
)
_invoke(factory, model_alias="ghost")
assert registry.captured_alias == "ghost"
# ---------------------------------------------------------------------------
# Tier 2 — model.default_alias (admin-managed system default)
# ---------------------------------------------------------------------------
def test_model_default_alias_used_when_coordinator_unset() -> None:
"""Regression for the historical drift: admin sets the system
default in the Models tab, the home composer advertises it, and new
coordinator sessions must launch on the same alias rather than
silently falling through to ``registry.default``."""
factory, registry = _build_factory(
known_aliases={"admin-default", "registry-default"},
settings={"model.default_alias": "admin-default"},
)
_invoke(factory)
assert registry.captured_alias == "admin-default"
def test_unknown_model_default_alias_falls_through_to_registry_default() -> None:
"""Tier 2 is *not* an explicit pin — operators set
``model.default_alias`` once in the UI and forget about it; an alias
that's later disabled or typo'd should not 503 the coordinator,
since tier 3 (``registry.default``) is guaranteed to resolve."""
factory, registry = _build_factory(
known_aliases={"registry-default"}, # admin-default got removed
settings={"model.default_alias": "admin-default"},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
def test_blank_model_default_alias_falls_through_to_registry_default() -> None:
factory, registry = _build_factory(
settings={"model.default_alias": ""},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
# ---------------------------------------------------------------------------
# Tier 3 — registry.default (config.toml [model].default)
# ---------------------------------------------------------------------------
def test_no_settings_uses_registry_default() -> None:
factory, registry = _build_factory()
_invoke(factory)
assert registry.captured_alias == "registry-default"
def test_whitespace_only_coord_alias_falls_through() -> None:
"""``" "`` is not an explicit pin — ``.strip()`` reduces it to
"", which the chain should treat as unset."""
factory, registry = _build_factory(
settings={"coordinator.model_alias": " "},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
-79
View File
@@ -716,82 +716,3 @@ class TestCoordinatorAdapterDispatchChildEvent:
},
)
assert recorder.enqueued == []
def test_dispatch_notifies_child_event_bus_on_state_event(self) -> None:
"""Every translated state-class event must call
``ChildEventBus.notify(ws_id)`` so a registered
``wait_for_workstream`` waiter wakes promptly. Notify fires
AFTER the UI enqueue so the SSE fan-out keeps priority the
order assertion here is structural (one notify call, matching
ws_id) since the bus side-effect lookup is what guards against
regressions, not the relative event ordering.
"""
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-a1"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "idle",
}
)
assert event.is_set(), "bus notify did not fire on cluster_state dispatch"
def test_dispatch_notifies_for_all_state_class_event_types(self) -> None:
"""The dispatch sink translates six event types into the
``child_ws_*`` SSE shape; all six must also fire the bus so
a wait on any of them wakes. ``ws_created`` is intentionally
NOT in this set waiters register against ws_ids they already
know exist (the wait tool takes a pre-known list)."""
for etype, extra in [
("cluster_state", {"state": "running"}),
("ws_closed", {"reason": "evicted"}),
("ws_rename", {"name": "renamed"}),
("intent_verdict", {"verdict": {"call_id": "c1"}}),
("approval_resolved", {"approved": True}),
("approve_request", {"detail": {}}),
]:
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-a1"])
adapter._dispatch_child_event(
{"type": etype, "ws_id": "child-a1", **extra},
)
assert event.is_set(), f"bus notify did not fire on {etype} dispatch"
def test_dispatch_does_not_notify_for_unrelated_ws_id(self) -> None:
"""Bus is keyed by ws_id — a dispatch for ws X must not wake a
waiter registered against ws Y, or every state change anywhere
in the system would shake every concurrent wait."""
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-other"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "idle",
}
)
assert not event.is_set(), "bus notify spuriously fired on unrelated ws_id"
def test_dispatch_does_not_notify_for_unknown_child(self) -> None:
"""Events whose ws_id isn't in any coord's registry are dropped
BEFORE the bus notify (early return at ``coord_id is None``).
Notify only fires for events the dispatch sink fully translated,
keeping the bus side-effect aligned with the UI enqueue."""
adapter, _, _ = self._setup()
bus = adapter.child_event_bus
event = bus.register_waiter(["ws-orphan"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "ws-orphan",
"state": "idle",
}
)
assert not event.is_set(), "bus notify fired for ws_id the dispatch dropped"
+9 -649
View File
@@ -9,7 +9,6 @@ storage-call path.
from __future__ import annotations
import json
import time
from typing import TYPE_CHECKING, Any
import httpx
@@ -21,7 +20,6 @@ from turnstone.console.coordinator_client import (
CoordinatorTokenManager,
)
from turnstone.core.auth import JWT_AUD_CONSOLE, validate_jwt
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
@@ -147,7 +145,6 @@ def _mock_client(
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
return client, captured
@@ -473,7 +470,6 @@ def _make_read_client(storage: SQLiteBackend) -> CoordinatorClient:
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
@@ -667,7 +663,6 @@ def _make_client_with_cluster_response(
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
@@ -1228,49 +1223,6 @@ def test_list_skills_hides_interactive_only_skills(tmp_path):
assert skill["kind"] in {"coordinator", "any"}
def test_list_skills_omits_allowed_tools_when_empty(tmp_path):
"""``allowed_tools`` is the auto-approve allowlist (tools exempt
from the operator approval gate), NOT the set of tools the skill
can use. An empty list reads as "no tool access" to a model
that doesn't know the semantics — real misdiagnosis source: a
code-review skill with no auto-approve allowlist looked like it
had been spawned with zero tools. Dropping the key when empty
removes the ambiguity at the source; absence of the field carries
the unambiguous meaning "no tool is pre-approved for this skill"
while a tool list reads as "these specific tools bypass the prompt".
"""
st = SQLiteBackend(str(tmp_path / "skills_empty.db"))
st.create_prompt_template(
template_id="s-empty",
name="empty-skill",
category="ops",
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags="[]",
allowed_tools="[]",
)
st.create_prompt_template(
template_id="s-nonempty",
name="nonempty-skill",
category="ops",
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags="[]",
allowed_tools='["read_file"]',
)
client = _make_read_client(st)
result = client.list_skills()
by_name = {s["name"]: s for s in result["skills"]}
assert "allowed_tools" not in by_name["empty-skill"]
assert by_name["nonempty-skill"]["allowed_tools"] == ["read_file"]
def test_list_skills_projects_allowed_tools_capped_with_sentinel(tmp_path):
"""Each row carries the skill's allowed_tools (capped at the projection
cap with a +N more sentinel) so coordinators can pick a skill without
@@ -1491,23 +1443,6 @@ def test_wait_for_workstream_denies_foreign_ws_id(populated_storage):
assert result["elapsed"] < 1.0
def test_wait_for_workstream_denies_cross_tenant_child(populated_storage):
"""Defense-in-depth (Copilot #506): a row whose ``parent_ws_id``
matches the coordinator but whose ``user_id`` belongs to a
different tenant must collapse to ``denied`` otherwise a
forged / migration-era / pre-tenant-gate row would let a
coordinator's LLM observe foreign-tenant state through
``wait_for_workstream``. The ``populated_storage`` fixture's
``cross-tenant-child`` row has exactly this shape
(parent_ws_id="coord-1", user_id="user-2").
"""
client = _make_read_client(populated_storage)
result = client.wait_for_workstream(["cross-tenant-child"], timeout=5, mode="any")
assert result["results"]["cross-tenant-child"]["state"] == "denied"
assert result["complete"] is False
assert result["elapsed"] < 1.0
def test_wait_for_workstream_missing_ws_id_indistinguishable_from_denied(populated_storage):
"""A ws_id that doesn't exist collapses into the same 'denied'
shape as a foreign ws_id so wait can't be used as an existence
@@ -1596,22 +1531,10 @@ def test_wait_for_workstream_dedupes_ws_ids(populated_storage):
assert list(result["results"].keys()) == ["child-a"]
def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
populated_storage, monkeypatch
):
"""All storage reads issued by ``wait_for_workstream`` must go
through the batched paths. At the documented cap (32 ws_ids over
a 600 s wait) the naive per-id shape produced ~38k row reads, so
a regression to per-id is the meaningful failure mode this test
guards against.
The primary safety net is the ``pytest.fail`` mock on the per-id
``get_workstream`` / ``sum_workstream_tokens`` paths any call
there blows up loudly with the regression message. The
additional ``batch_calls`` / ``sum_calls`` assertions cover the
subtler regression where the call IS batched but only covers a
subset of ws_ids (e.g. one ws_id per call in a loop).
"""
def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monkeypatch):
"""Per-tick polling must issue batched storage calls — at the
documented cap (32 ws_ids over a 600s wait) the naive per-id
shape produced ~38k row reads. Guard against regression."""
client = _make_read_client(populated_storage)
batch_calls: list[list[str]] = []
sum_calls: list[list[str]] = []
@@ -1642,16 +1565,11 @@ def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
result = client.wait_for_workstream(["child-a", "child-b"], timeout=5, mode="any")
assert result["complete"] is True
# Every batched call carried the full ws_id set. The exact count
# (currently 2: one pre-loop ownership filter + one snapshot tick)
# is incidental; if either gains another batched read it stays
# batched, which is the property under test.
assert batch_calls, "no batched get_workstreams_batch call observed"
assert sum_calls, "no batched sum_workstream_tokens_batch call observed"
first_batch = set(batch_calls[0])
first_sum = set(sum_calls[0])
assert first_batch == {"child-a", "child-b"}
assert first_sum == {"child-a", "child-b"}
# One tick is enough since child-a is already idle (terminal).
assert len(batch_calls) == 1
assert len(sum_calls) == 1
assert set(batch_calls[0]) == {"child-a", "child-b"}
assert set(sum_calls[0]) == {"child-a", "child-b"}
def test_wait_for_workstream_handles_non_string_mode(populated_storage):
@@ -1663,205 +1581,6 @@ def test_wait_for_workstream_handles_non_string_mode(populated_storage):
assert "invalid mode" in result["error"]
# ---------------------------------------------------------------------------
# wait_for_workstream — event-driven (ChildEventBus wired in)
# ---------------------------------------------------------------------------
#
# When the coord adapter wires its ``child_event_bus`` into the client,
# the wait loop blocks on a per-call ``threading.Event`` keyed by ws_id
# and only re-snapshots storage on state-change wakes or the heartbeat
# cap. The legacy ``time.sleep`` poll path remains intact for tests
# that don't wire the bus (above), so this section adds focused
# coverage of the bus-driven behaviour without re-running the full
# matrix of mode / since / cross-tenant cases.
def _make_read_client_with_bus(storage, bus) -> CoordinatorClient:
"""Like ``_make_read_client`` but wires a real ``ChildEventBus``.
Caller owns the bus so the test can call ``bus.notify(ws_id)`` to
simulate the dispatch-sink wake-up.
"""
transport = httpx.MockTransport(lambda r: httpx.Response(200))
http = httpx.Client(transport=transport)
return CoordinatorClient(
console_base_url="http://x",
storage=storage,
token_factory=lambda: "t",
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=bus,
)
def test_wait_with_bus_returns_immediately_when_already_terminal(populated_storage):
"""Subscribe-after-terminal race: the wait registers its waiter
BEFORE the first snapshot, then re-snapshots an already-terminal
child must return at once without spinning the heartbeat cap.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
result = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
assert result["complete"] is True
assert result["results"]["child-a"]["state"] == "idle"
assert result["elapsed"] < 1.0
# Waiter must be unregistered on exit so a long-lived bus doesn't
# accumulate dead keys across many waits.
assert "child-a" not in bus._waiters
def test_wait_with_bus_wakes_on_notify(populated_storage):
"""The core property of the refactor: a state-change ``notify``
must wake the wait promptly well under the legacy 0.5 s poll
cadence AND the 2 s heartbeat cap. Test fires a state update
+ notify after a short delay and asserts the wait returns quickly.
"""
import threading as _t
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# child-b starts running; flip to idle + notify after the wait
# blocks. 100 ms is enough that the wait is parked in event.wait()
# but short enough that the test runs fast.
timer = _t.Timer(
0.1,
lambda: (
populated_storage.update_workstream_state("child-b", "idle"),
bus.notify("child-b"),
),
)
timer.start()
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=5.0, mode="any")
elapsed = time.monotonic() - start
assert result["complete"] is True
assert result["results"]["child-b"]["state"] == "idle"
# Bus-driven wake should fire well under 1 s; legacy poll would
# take ~0.5 s but bus-driven should be ~0.1 s (the timer delay)
# plus a few ms. Generous 0.6 s budget for CI noise.
assert elapsed < 0.6, f"wake-up too slow: {elapsed}s"
def test_wait_with_bus_unrelated_notify_does_not_wake(populated_storage):
"""A notify on a ws_id the wait isn't watching must NOT wake it —
otherwise every state change anywhere on the system would shake
every concurrent wait into a redundant storage snapshot.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# child-b is running indefinitely; mode='all' will time out unless
# a relevant notify fires. Fire only unrelated notifies — wait
# should still hit the full timeout.
import threading as _t
def _fire_unrelated() -> None:
for _ in range(5):
bus.notify("ws-unrelated-1")
bus.notify("ws-unrelated-2")
time.sleep(0.05)
t = _t.Thread(target=_fire_unrelated, daemon=True)
t.start()
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=0.5, mode="all")
elapsed = time.monotonic() - start
assert result["complete"] is False, "unrelated notify falsely satisfied wait"
# Wait should burn its full timeout (give or take heartbeat
# granularity). The bus path doesn't have a 0.5 s poll, so the
# bound is "approximately timeout".
assert elapsed >= 0.5
t.join(timeout=1.0)
def test_wait_with_bus_heartbeat_still_progresses_without_notify(populated_storage):
"""Without any notify, the wait must still progress through ticks
via the heartbeat cap so ``progress_callback`` keeps firing for
the sidebar UI. Verified by counting callback firings over an
interval longer than the heartbeat.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# Shrink the heartbeat for test speed via the ClassVar seam —
# instance attribute shadows the class-level default. Production
# stays at 2.0 s; the test exercises the heartbeat-fires-without-
# notify property in well under 1 s.
client._WAIT_HEARTBEAT_INTERVAL = 0.1 # type: ignore[misc]
snapshots: list[dict[str, dict[str, object]]] = []
def _cb(snap: dict[str, dict[str, object]], _elapsed: float) -> None:
snapshots.append(snap)
# child-b is running indefinitely; wait will time out at 0.4 s.
# With heartbeat = 0.1 s, we expect ~3-5 callback firings
# (initial tick + ~3-4 heartbeats). Loose lower bound to avoid
# CI flakiness.
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=0.4, mode="all", progress_callback=_cb)
elapsed = time.monotonic() - start
assert result["complete"] is False
assert elapsed >= 0.4
# At least 2 callback firings: the initial snapshot plus at least
# one heartbeat-driven re-tick. Tight upper bound would be
# ~ceil(0.4/0.1) + 1 = 5 firings.
assert len(snapshots) >= 2, f"heartbeat didn't fire: {len(snapshots)} snapshots"
def test_wait_with_bus_unregisters_waiter_on_exit(populated_storage):
"""Both the success path and the timeout path must unregister the
waiter otherwise a long-lived bus accumulates dead
``threading.Event`` instances forever.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# Success path (already-terminal child).
client.wait_for_workstream(["child-a"], timeout=5, mode="any")
assert bus._waiters == {}, "success path leaked waiter"
# Timeout path (running child, mode='all' that times out).
client.wait_for_workstream(["child-a", "child-b"], timeout=0.3, mode="all")
assert bus._waiters == {}, "timeout path leaked waiter"
def test_wait_with_bus_multi_waiter_independence(populated_storage):
"""Two concurrent waits on the same ws_id must be independent —
one wait completing must not affect the other's wake-up state.
Smoke-tests the multi-Event-per-bucket bus behaviour against the
real wait-loop.
"""
import threading as _t
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
results: dict[str, dict[str, object]] = {}
def _do_wait(label: str) -> None:
results[label] = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
threads = [_t.Thread(target=_do_wait, args=(f"t{i}",), daemon=True) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5.0)
for label in ("t0", "t1", "t2"):
assert results[label]["complete"] is True
assert results[label]["results"]["child-a"]["state"] == "idle"
# All waiters must be unregistered after exit.
assert bus._waiters == {}
# ---------------------------------------------------------------------------
# wait_for_workstream — last-message bundling
# ---------------------------------------------------------------------------
@@ -2659,362 +2378,3 @@ def test_cleanup_dead_task_child_refs_storage_batch_failure_swallows(populated_s
populated_storage.get_workstreams_batch = _boom # type: ignore[method-assign]
assert client.cleanup_dead_task_child_refs("coord-1") == 0
# ---------------------------------------------------------------------------
# inspect_workstream — three-tier output compression
# ---------------------------------------------------------------------------
#
# A coord doing a fan-out wave against tool-heavy children would
# otherwise blow the context budget on raw output alone. Mirrors the
# search tool's Tier-1/Tier-2/Tier-3 ladder.
def _make_inspect_result(
*, ws_id: str = "ws-test", state: str = "running", n_messages: int = 5
) -> dict[str, Any]:
"""Build an inspect-result dict shaped like ``coordinator_client.inspect()``.
Production output keys (``ws_id``, ``skill_id``) mirror the storage
row that ``inspect()`` spreads from ``get_workstream``. Tests that
synthesize an inspect result must match these keys otherwise a
formatter that looks at the production keys silently emits null
values against a fixture that uses different ones (real bug-1
regression source: skeleton tier read ``skill`` from a fixture
that wrote ``skill`` while production wrote ``skill_id``).
"""
return {
"ws_id": ws_id,
"state": state,
"title": "test workstream",
"skill_id": "researcher",
"messages": [
{"role": "user" if i % 2 == 0 else "assistant", "content": f"msg {i} content"}
for i in range(n_messages)
],
"verdicts": [],
}
def test_format_inspect_tiered_full_fits_returns_full_tier():
"""Small payloads pass through with `_tier='full'` — no compression."""
from turnstone.console.coordinator_client import _format_inspect_tiered
result = _make_inspect_result(n_messages=3)
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "full"
# Every message verbatim.
assert len(parsed["messages"]) == 3
assert parsed["messages"][0]["content"] == "msg 0 content"
def test_format_inspect_tiered_compact_when_full_exceeds_budget():
"""Large messages trigger the compact tier — head/tail-snipped
content with the rest of the row intact."""
from turnstone.console.coordinator_client import (
_INSPECT_MSG_CONTENT_HEAD,
_INSPECT_MSG_CONTENT_TAIL,
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# Each message ~5KB; with 20 messages, full tier blows the 32KB budget.
fat = "X" * 5000
result = {
"id": "ws-fat",
"state": "running",
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
# Every message preserved (compact keeps the count, just snips content).
assert len(parsed["messages"]) == 20
# Head/tail snip kicked in.
msg_content = parsed["messages"][0]["content"]
assert msg_content.startswith("X" * _INSPECT_MSG_CONTENT_HEAD)
assert msg_content.endswith("X" * _INSPECT_MSG_CONTENT_TAIL)
assert "chars elided" in msg_content
# Budget invariant — the load-bearing contract of the formatter.
# Without this assertion, a future change to ``_tier_note`` or
# ``_compact_message`` could push the output over budget and the
# ``_truncate_output`` head+tail safety net would silently mask
# the regression, re-introducing the middle-message-drop pathology.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_compact_when_content_below_snip_threshold():
"""When per-message content is below the snip threshold but the
message COUNT alone overflows the budget, compact tier must still
stay within budget by trimming the message list (head + tail of
messages) rather than degrading straight to skeleton. Bug-3
regression cover: with 400 × 100-char messages, the original
formatter fell through to skeleton because adding ``_tier_note``
to an un-snipped tier-2 produced output strictly larger than
tier-1 (both over budget). The fix preserves messages from both
ends of the list and inserts an ``_omitted`` sentinel."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# 400 × ~100 chars → Tier-1 ~53 KB (over budget), per-message
# content under the 964-char snip threshold so content-snipping
# saves nothing. Without the list-trim rung the formatter would
# fall to skeleton and drop all 400 messages.
smallish = "S" * 100
result = {
"ws_id": "ws-many-small",
"state": "running",
"messages": [
{"role": "assistant" if i % 2 == 0 else "user", "content": smallish} for i in range(400)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
# Should NOT fall through to skeleton — message-list trim preserves
# head + tail of the conversation.
assert parsed["_tier"] == "compact"
assert "messages" in parsed
# Some messages must survive; the trim shape is head + tail with an
# ``_omitted`` sentinel between them.
assert len(parsed["messages"]) > 0
assert len(parsed["messages"]) < 400
# Budget invariant.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_skeleton_when_compact_also_exceeds_budget():
"""Tier 3 fallback: counts + last assistant preview only. Trigger by
flooding with messages whose content is a multi-block list the
snipper correctly leaves non-string content unchanged (mirrors
Anthropic/OpenAI multi-block content shape), so even after the
(5, 10) message-list trim the surviving 15 messages don't fit in
the 32 KB budget."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# 50 messages × multi-block content (~30 KB each — list-shape
# content bypasses the head/tail string snipper because lists
# aren't strings). Even (5, 10) trim leaves 15 × 30 KB which
# blows the 32 KB budget — forces skeleton.
fat_block = {"type": "text", "text": "Y" * 3000}
result = {
"ws_id": "ws-flood",
"state": "running",
"title": "flood",
"skill_id": "researcher",
"messages": [
{
"role": "assistant" if i % 2 == 0 else "user",
"content": [fat_block] * 10,
}
for i in range(50)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "skeleton"
assert parsed["message_count"] == 50
# Role distribution surfaces — the "what shape of activity" signal.
assert parsed["roles"]["assistant"] == 25
assert parsed["roles"]["user"] == 25
# No `messages` field at skeleton tier — only the aggregate signal.
assert "messages" not in parsed
# Budget invariant.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_skeleton_keeps_terminal_state_fields():
"""``close_reason`` / ``last_error`` survive the skeleton fall — they're
small, load-bearing, and the operator needs them to understand WHY
a terminal child landed in its state."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# Same flood pattern as the bare-skeleton test (multi-block content
# bypasses the string snipper) — paired with terminal-state fields
# that must survive the skeleton fall.
fat_block = {"type": "text", "text": "Z" * 3000}
result = {
"ws_id": "ws-closed",
"state": "closed",
"title": "done",
"skill_id": "researcher",
"messages": [{"role": "user", "content": [fat_block] * 10} for _ in range(50)],
"verdicts": [],
"close_reason": "task complete: report attached",
"live": None, # filtered by truthy check
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "skeleton"
assert parsed["close_reason"] == "task complete: report attached"
# Falsy ``live`` doesn't bleed through.
assert "live" not in parsed
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_error_shapes_bypass_tiering():
"""Cross-tenant / not-found responses keep their original shape — they
carry no messages, are already tiny, and changing them would break
callers that key on the ``error`` field."""
from turnstone.console.coordinator_client import _format_inspect_tiered
result = {"error": "workstream not found", "ws_id": "ws-foreign"}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed == {"error": "workstream not found", "ws_id": "ws-foreign"}
# No `_tier` annotation — error shapes are self-describing.
assert "_tier" not in parsed
def test_format_inspect_tiered_compact_preserves_tool_call_linkage():
"""Compact tier keeps ``tool_name`` / ``tool_call_id`` / ``name`` so a
model reading the snipped trace can still pair a tool call to its
response the linkage is load-bearing for "what happened" signal."""
from turnstone.console.coordinator_client import _format_inspect_tiered
fat = "Q" * 5000
result = {
"ws_id": "ws-tools",
"state": "running",
"messages": [
{
"role": "assistant",
"content": fat,
"tool_name": "bash",
"tool_call_id": "call-1",
}
for _ in range(20)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
first = parsed["messages"][0]
assert first["tool_name"] == "bash"
assert first["tool_call_id"] == "call-1"
def test_format_inspect_tiered_compact_preserves_assistant_tool_calls():
"""Compact tier must preserve the assistant-side ``tool_calls`` list
(OpenAI shape: ``[{id, type, function: {name, arguments}}]``) so a
model reading the snipped trace can see WHICH tool was called and
pair it with the corresponding result row via ``id`` ``tool_call_id``.
Bug-2 regression cover: the pre-fix compactor stripped ``tool_calls``,
leaving the audit reader with a tool-result orphan against an
invisible call.
``function.arguments`` strings are snipped head/tail (analogous to
content) because they can be multi-KB JSON; ``id`` and
``function.name`` are preserved verbatim they're the linkage."""
from turnstone.console.coordinator_client import (
_INSPECT_TOOL_ARG_HEAD,
_INSPECT_TOOL_ARG_TAIL,
_format_inspect_tiered,
)
fat_content = "C" * 5000 # forces compact tier
fat_args = "A" * 5000 # forces argument snipping
tool_calls = [
{
"id": "call-abc-123",
"type": "function",
"function": {"name": "bash", "arguments": fat_args},
},
{
"id": "call-def-456",
"type": "function",
"function": {"name": "read_file", "arguments": fat_args},
},
]
result = {
"ws_id": "ws-tool-calls",
"state": "running",
"messages": [
{"role": "assistant", "content": fat_content, "tool_calls": tool_calls}
for _ in range(20)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
first = parsed["messages"][0]
# tool_calls survives compaction.
assert "tool_calls" in first
assert len(first["tool_calls"]) == 2
# Linkage fields verbatim.
assert first["tool_calls"][0]["id"] == "call-abc-123"
assert first["tool_calls"][0]["function"]["name"] == "bash"
assert first["tool_calls"][1]["id"] == "call-def-456"
assert first["tool_calls"][1]["function"]["name"] == "read_file"
# arguments snipped head/tail — both prefix and suffix preserved.
snipped_args = first["tool_calls"][0]["function"]["arguments"]
assert snipped_args.startswith("A" * _INSPECT_TOOL_ARG_HEAD)
assert snipped_args.endswith("A" * _INSPECT_TOOL_ARG_TAIL)
assert "chars elided" in snipped_args
def test_format_inspect_tiered_compact_passes_small_messages_through_unsnipped():
"""Messages under the snip threshold pass through verbatim at compact
tier snipping a 100-byte message costs more bytes (the elision
marker) than it saves."""
from turnstone.console.coordinator_client import _format_inspect_tiered
# Mix: a few large messages force compact tier; small messages must
# not be snipped.
big = "B" * 5000
small = "S" * 50
result = {
"id": "ws-mixed",
"state": "running",
"messages": [{"role": "assistant", "content": big} for _ in range(15)]
+ [{"role": "user", "content": small}],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
# The trailing small message is exact, not snipped.
assert parsed["messages"][-1]["content"] == small
def test_format_inspect_tiered_emits_tier_note_when_compressed():
"""The ``_tier_note`` advisory tells the LLM how to ask for a tighter
or fuller view next time actionable feedback rather than a bare
"we compressed your output" signal."""
from turnstone.console.coordinator_client import _format_inspect_tiered
fat = "F" * 5000
result = {
"id": "ws-noted",
"state": "running",
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert "_tier_note" in parsed
assert "message_limit" in parsed["_tier_note"]
def test_format_inspect_tiered_full_tier_omits_tier_note():
"""When the full tier fits, no note is emitted — the absence of a
note is the signal that nothing was compressed."""
from turnstone.console.coordinator_client import _format_inspect_tiered
out = _format_inspect_tiered(_make_inspect_result(n_messages=2))
parsed = json.loads(out)
assert parsed["_tier"] == "full"
assert "_tier_note" not in parsed
-3
View File
@@ -40,7 +40,6 @@ from turnstone.console.server import (
_require_coord_mgr,
)
from turnstone.core.auth import AuthResult
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.session_manager import SessionManager
from turnstone.core.session_routes import (
SessionEndpointConfig,
@@ -287,7 +286,6 @@ def test_coordinator_client_spawn_close_delete(tmp_path):
coord_ws_id="coord-42",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
# spawn ---------------------------------------------------------------
@@ -389,7 +387,6 @@ def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
coord_ws_id="coord-root",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
-489
View File
@@ -1,489 +0,0 @@
"""Unit tests for :class:`CoordinatorIdleObserver`.
Drives a fake :class:`SessionManager` that mirrors the real one's
``subscribe_to_state`` / ``get`` contract, plus a fake storage with the
``list_workstreams`` slice the observer queries.
"""
from __future__ import annotations
import contextlib
import threading
from typing import Any
from unittest.mock import MagicMock
import pytest
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
from turnstone.core.nudge_queue import NudgeQueue
from turnstone.core.workstream import WorkstreamKind, WorkstreamState
class _FakeRow:
"""SQLAlchemy-Row-like wrapper exposing ``_mapping``."""
def __init__(self, **kwargs: Any) -> None:
self._mapping = kwargs
class _FakeStorage:
def __init__(self) -> None:
self.children: list[dict[str, Any]] = []
self.list_calls: list[dict[str, Any]] = []
self.count_calls: list[dict[str, Any]] = []
self.list_raises: bool = False
self.count_raises: bool = False
def list_workstreams(
self,
node_id: str | None = None,
limit: int = 100,
*,
parent_ws_id: str | None = None,
kind: WorkstreamKind | str | None = None,
user_id: str | None = None,
) -> list[Any]:
self.list_calls.append(
{
"limit": limit,
"parent_ws_id": parent_ws_id,
"kind": kind,
"user_id": user_id,
}
)
if self.list_raises:
raise RuntimeError("storage forced failure")
return [_FakeRow(**c) for c in self.children]
def count_workstreams_by_state(
self,
*,
parent_ws_id: str | None = None,
user_id: str | None = None,
) -> dict[str, int]:
self.count_calls.append({"parent_ws_id": parent_ws_id, "user_id": user_id})
if self.count_raises:
raise RuntimeError("count forced failure")
counts: dict[str, int] = {}
for c in self.children:
counts[c["state"]] = counts.get(c["state"], 0) + 1
return counts
class _FakeSession:
def __init__(self) -> None:
self._nudge_queue = NudgeQueue()
self.messages: list[dict[str, Any]] = []
self._wake_source_tag: str = ""
self._metacog_state: dict[str, float] = {}
self._mem_cfg = MagicMock(nudge_cooldown=300)
def _visible_memory_count(self) -> int:
return 0
class _FakeWorkstream:
def __init__(
self,
ws_id: str = "ws-coord",
kind: WorkstreamKind = WorkstreamKind.COORDINATOR,
user_id: str = "u1",
) -> None:
self.id = ws_id
self.kind = kind
self.user_id = user_id
self.session: _FakeSession | None = _FakeSession()
class _FakeManager:
def __init__(self) -> None:
self._workstreams: dict[str, _FakeWorkstream] = {}
self._subscribers: list[Any] = []
self._lock = threading.Lock()
def add_ws(self, ws: _FakeWorkstream) -> None:
self._workstreams[ws.id] = ws
def remove_ws(self, ws_id: str) -> None:
self._workstreams.pop(ws_id, None)
def get(self, ws_id: str) -> _FakeWorkstream | None:
return self._workstreams.get(ws_id)
def subscribe_to_state(self, callback: Any) -> None:
with self._lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Any) -> None:
with self._lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
def fire_state(self, ws_id: str, state: WorkstreamState) -> None:
with self._lock:
subs = list(self._subscribers)
for cb in subs:
with contextlib.suppress(Exception):
cb(ws_id, state)
@pytest.fixture
def coord_setup() -> tuple[_FakeManager, _FakeStorage, _FakeWorkstream]:
mgr = _FakeManager()
storage = _FakeStorage()
ws = _FakeWorkstream()
mgr.add_ws(ws)
return mgr, storage, ws
def _add_active_child(storage: _FakeStorage, **overrides: Any) -> None:
storage.children.append(
{
"ws_id": overrides.get("ws_id", "child-1"),
"name": overrides.get("name", "research"),
"state": overrides.get("state", "running"),
}
)
class TestEnqueueOnIdle:
def test_idle_with_active_children_enqueues(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage, ws_id="child-a", state="running")
_add_active_child(storage, ws_id="child-b", state="thinking")
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
snap = ws.session._nudge_queue.pending("any")
assert len(snap) == 1
nudge_type, text = snap[0]
assert nudge_type == "idle_children"
assert "child-a" in text
assert "child-b" in text
def test_idle_with_no_active_children_no_enqueue(self, coord_setup):
mgr, storage, ws = coord_setup
# storage.children is empty
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
def test_idle_only_idle_state_children_no_enqueue(self, coord_setup):
mgr, storage, ws = coord_setup
# All children "idle" — terminal-from-coord-perspective; not active.
_add_active_child(storage, state="idle")
_add_active_child(storage, state="closed")
_add_active_child(storage, state="error")
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
def test_non_idle_state_no_enqueue(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
for state in (
WorkstreamState.RUNNING,
WorkstreamState.THINKING,
WorkstreamState.ATTENTION,
WorkstreamState.ERROR,
):
mgr.fire_state(ws.id, state)
assert len(ws.session._nudge_queue) == 0
class TestKindFilter:
def test_interactive_workstream_skipped(self):
mgr = _FakeManager()
storage = _FakeStorage()
_add_active_child(storage)
ws = _FakeWorkstream(kind=WorkstreamKind.INTERACTIVE)
mgr.add_ws(ws)
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Observer ignored the non-coord workstream entirely.
assert len(ws.session._nudge_queue) == 0
# Storage was NOT queried — kind check happens before list_workstreams.
assert storage.list_calls == []
class TestWaitForWorkstreamSkip:
def test_skips_when_last_assistant_used_wait(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
ws.session.messages = [
{"role": "user", "content": "kick off"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call-1",
"function": {"name": "wait_for_workstream", "arguments": "{}"},
}
],
},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Don't pile on — model is already using the right tool.
assert len(ws.session._nudge_queue) == 0
def test_fires_when_last_assistant_used_different_tool(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
ws.session.messages = [
{"role": "user", "content": "go"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{"id": "call-1", "function": {"name": "spawn_workstream", "arguments": "{}"}}
],
},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 1
class TestHardCap:
def test_hard_cap_blocks_after_n_fires(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Bypass cooldown for this test: each call burns a per-type slot
# in ``_metacog_state`` so we need to clear it between fires.
for _ in range(3):
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Cap = 3 fires. Even with cooldown bypassed, the 4th doesn't fire.
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# We enqueued 3 entries total; cap blocked the 4th.
snap = ws.session._nudge_queue.pending("any")
assert len(snap) == 3
def test_cap_resets_when_state_leaves_idle_without_wake(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Burn the cap.
for _ in range(3):
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 3
# Drain the queue (simulate the watcher delivering them).
ws.session._nudge_queue.drain({"any"})
# Real (non-wake) leave-IDLE: tag is empty. Cap resets.
ws.session._wake_source_tag = ""
mgr.fire_state(ws.id, WorkstreamState.RUNNING)
# New IDLE — cap is fresh, fires again.
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 1
def test_cap_does_not_reset_during_wake_driven_exit(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Burn the cap.
for _ in range(3):
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
ws.session._nudge_queue.drain({"any"})
# Wake-driven leave-IDLE: tag is set during the wake send.
ws.session._wake_source_tag = "system_nudge"
mgr.fire_state(ws.id, WorkstreamState.RUNNING)
ws.session._wake_source_tag = "" # tag cleared at end of wake send
# Cap should NOT have reset — re-IDLE shouldn't fire.
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 0
class TestCooldown:
def test_cooldown_blocks_within_window(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 1
# Drain so the queue isn't the gate.
ws.session._nudge_queue.drain({"any"})
# Second fire within the cooldown window → should_nudge returns False.
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 0
class TestStorageFailure:
def test_storage_exception_is_swallowed(self, coord_setup):
mgr, storage, ws = coord_setup
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
storage.list_raises = True
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Must not raise / propagate.
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
class TestValidUntilPredicate:
def test_predicate_drops_when_children_finish_before_drain(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage, ws_id="child-a", state="running")
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 1
# Children now complete (storage shows none active).
storage.children.clear()
# Drain at the user seam — predicate re-queries, finds 0 active,
# drops the entry without delivering.
from turnstone.core.nudge_queue import USER_DRAIN
delivered = ws.session._nudge_queue.drain(USER_DRAIN)
assert delivered == []
assert len(ws.session._nudge_queue) == 0
def test_predicate_delivers_when_children_still_active(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage, ws_id="child-a", state="running")
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Children still active → predicate returns True → entry delivers.
from turnstone.core.nudge_queue import USER_DRAIN
delivered = ws.session._nudge_queue.drain(USER_DRAIN)
assert len(delivered) == 1
assert delivered[0][0] == "idle_children"
def test_predicate_drops_on_storage_failure(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Storage failure at drain time. Predicate treats raises as
# "no longer valid" (drop) — see NudgeQueue.drain's predicate
# exception handling.
storage.count_raises = True
from turnstone.core.nudge_queue import USER_DRAIN
delivered = ws.session._nudge_queue.drain(USER_DRAIN)
assert delivered == []
class TestLifecycle:
def test_start_idempotent(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
observer.start() # no-op
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Double-subscribe would have produced 2 entries.
assert len(ws.session._nudge_queue.pending("any")) == 1
def test_shutdown_unsubscribes(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
observer.shutdown()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
def test_shutdown_idempotent(self, coord_setup):
mgr, _storage, _ws = coord_setup
observer = CoordinatorIdleObserver(mgr, _storage)
observer.start()
observer.shutdown()
observer.shutdown() # no error
-62
View File
@@ -167,28 +167,6 @@ def test_coordinator_js_exposes_inline_approval_helpers():
# direction.
assert "function appendUserMessageWithAttachments" in body
assert "msg-user-attach" in body
# PR #487 — whitespace-only assistant content (Qwen3 with vLLM
# ``--reasoning-parser`` strips ``<think>…</think>`` and emits only
# ``"\n\n"`` as content before a tool call) must be skipped on
# history replay or the empty ``.msg.assistant`` card surfaces as
# a phantom row. The literal substring ``content.trim()`` is the
# single-line guard the rendering branch uses; a refactor that
# drops the trim() (e.g. simplifies to ``if (!content)``) silently
# regresses the phantom-card fix on the multi-node coord path.
# Mirrors ``test_app_js.py``'s same-shape pin on ``app.js``.
assert "content.trim()" in body
# PR #487 — coord history replay must render the assistant content
# card BEFORE the tool batch, not after, so DOM order matches the
# chronological order the model emitted (text → dispatch → results).
# Pre-fix the tool_calls branch sat at the role-agnostic top of the
# loop and rendered ahead of the assistant text that announced the
# batch, putting parallel fan-outs visually above their narrating
# message. The fix hoisted the synthesis into ``renderAssistantToolBatch``
# called from inside the assistant branch AFTER the content card —
# asserting the helper name lets a refactor that re-inlines or
# renames it surface here instead of via manual reload testing.
assert "function renderAssistantToolBatch" in body
assert "renderAssistantToolBatch(m)" in body
def test_coordinator_js_handle_child_state_no_longer_reads_sse_pending_approval_detail():
@@ -302,43 +280,3 @@ def test_coordinator_js_handle_child_state_no_longer_reads_sse_pending_approval_
"cycles) — without this, the second bulk-poll after an SSE "
"transition silently clobbers."
)
def test_coord_history_renders_user_interjection_advisory_after_tool_block():
"""Queued user messages spliced into the last tool-result envelope
of a batch (Seam 1) persist on the tool DB row as a wrapped
``<tool_output>`` envelope. ``decorate_history_messages`` extracts
the advisory back out and the wire layer projects it onto
``m.advisories``; the coord history loop must invoke the shared
``replayAdvisoriesAfterTool`` helper (defined in
``shared/utils.js``) so each ``user_interjection`` renders through
``appendUserMessageWithAttachments`` and the bubble looks identical
to a Seam 2/3 user row.
This test pins the call site so a refactor that drops the helper
invocation regresses the queued-during-batch replay shape silently.
Mirrors ``test_app_js.py``'s same-shape pin on interactive's
``replayHistory``."""
import re
from pathlib import Path
coord_js = Path(__file__).resolve().parent.parent / (
"turnstone/console/static/coordinator/coordinator.js"
)
body = coord_js.read_text(encoding="utf-8")
assert "replayAdvisoriesAfterTool(m.advisories" in body, (
"Coord history loop must invoke replayAdvisoriesAfterTool with "
"m.advisories so queued messages spliced into the tool envelope "
"render as user bubbles after the tool block."
)
# The renderer callback routes through appendUserMessageWithAttachments
# so the bubble matches a normal user-row replay.
assert re.search(
r"appendUserMessageWithAttachments\(\s*text",
body,
), (
"Coord history loop's renderer callback must route the extracted "
"advisory text through appendUserMessageWithAttachments so the "
"rendered bubble matches a normal user-row replay."
)
+4 -33
View File
@@ -215,12 +215,7 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
summary tempted callers to write ``if result["status"] == "idle"``
which silently never matched. The summary now omits the field
entirely; lifecycle state lives on the workstream row and is read
via inspect_workstream.
Also asserts the return key is ``child_ws_id`` (not ``ws_id``) so
the coordinator LLM doesn't recency-bias toward feeding the spawn
output back into another ``spawn_workstream(ws_id=...)`` call.
"""
via inspect_workstream."""
sess, coord, _ui = coord_session
coord.spawn.return_value = {
"ws_id": "child-7",
@@ -232,9 +227,8 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
_call_id, output = sess._exec_spawn_workstream(item)
body = json.loads(output)
assert "status" not in body
assert "ws_id" not in body
# The substantive fields are still here.
assert body["child_ws_id"] == "child-7"
assert body["ws_id"] == "child-7"
assert body["node_id"] == "node-1"
@@ -254,10 +248,6 @@ def test_spawn_batch_exec_does_not_surface_misleading_status_field(coord_session
body = json.loads(output)
assert "0" in body["results"]
assert "status" not in body["results"]["0"]
# Per-result entries surface ``child_ws_id``, not ``ws_id`` — same
# recency-bias rationale as the spawn_workstream test above.
assert body["results"]["0"]["child_ws_id"] == "c-x"
assert "ws_id" not in body["results"]["0"]
def test_spawn_exec_surfaces_client_error(coord_session):
@@ -270,21 +260,6 @@ def test_spawn_exec_surfaces_client_error(coord_session):
assert ui.tool_results[-1][3] is True # is_error
def test_spawn_exec_treats_missing_ws_id_on_success_path_as_error(coord_session):
"""A malformed upstream response (200-success-shape with no
``ws_id``) used to emit ``{"child_ws_id": null}`` to the LLM,
which then chased a null id through follow-up tools. Now matches
the matching guard in ``_exec_spawn_batch``: surface as a tool
error so the model retries instead of acting on garbage."""
sess, coord, ui = coord_session
# No ``error`` field, but ``ws_id`` is missing — the silent-null path.
coord.spawn.return_value = {"name": "c", "node_id": "node-1", "status": 200}
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
_call_id, output = sess._exec_spawn_workstream(item)
assert "no ws_id" in output
assert ui.tool_results[-1][3] is True # is_error
# ---------------------------------------------------------------------------
# inspect_workstream
# ---------------------------------------------------------------------------
@@ -1435,13 +1410,9 @@ def test_spawn_batch_exec_serialises_spawns_and_returns_results(coord_session):
assert body["denied"] == []
# Keyed by input index (stringified).
assert set(body["results"].keys()) == {"0", "1", "2"}
assert body["results"]["0"]["child_ws_id"] == "child-0"
assert body["results"]["0"]["ws_id"] == "child-0"
assert body["results"]["1"]["node_id"] == "n-1"
assert body["results"]["2"]["child_ws_id"] == "child-2"
# Confirm we don't leak the old ``ws_id`` key alongside the new
# ``child_ws_id`` — see test_spawn_exec_does_not_surface_misleading_status_field
# for the rationale on the rename.
assert "ws_id" not in body["results"]["0"]
assert body["results"]["2"]["ws_id"] == "child-2"
def test_spawn_batch_exec_surfaces_per_item_errors_in_denied(coord_session):
-8
View File
@@ -8,7 +8,6 @@ from __future__ import annotations
from datetime import UTC, datetime
import pytest
import sqlalchemy as sa
# ---------------------------------------------------------------------------
@@ -318,13 +317,6 @@ class TestPromptTemplateCRUD:
def test_get_prompt_template_nonexistent(self, db):
assert db.get_prompt_template("missing") is None
def test_create_prompt_template_duplicate_id_raises_conflict(self, db):
from turnstone.core.storage._protocol import StorageConflictError
db.create_prompt_template("dup", "first", "general", "A")
with pytest.raises(StorageConflictError, match="prompt_template conflict"):
db.create_prompt_template("dup", "second", "general", "B")
def test_list_prompt_templates_ordered_by_name(self, db):
db.create_prompt_template("t2", "beta", "general", "B")
db.create_prompt_template("t1", "alpha", "general", "A")
+12 -632
View File
@@ -167,7 +167,7 @@ class TestDecorateHistoryMessages:
"""End-to-end mutation of a /history-shaped message list — covers
the full transform applied by ``make_history_handler``."""
def test_decorates_tool_calls_with_verdict_and_assessment(self) -> None:
def test_decorates_tool_calls_and_marks_truncated(self) -> None:
verdicts = {
"call_a": {
"risk_level": "high",
@@ -181,6 +181,13 @@ class TestDecorateHistoryMessages:
assessments = {
"call_a": {"risk_level": "high", "flags": '["secret"]', "redacted": 1},
}
# Tool result content of exactly TOOL_RESULT_STORAGE_CAP chars
# hits the storage cap (longer is impossible — storage clamps
# at the cap). Reference the constant rather than a literal so
# this test stays correct if the cap moves again.
from turnstone.core.history_decoration import TOOL_RESULT_STORAGE_CAP
truncated_content = "x" * TOOL_RESULT_STORAGE_CAP
messages: list[dict[str, object]] = [
{"role": "user", "content": "hi"},
{
@@ -193,7 +200,7 @@ class TestDecorateHistoryMessages:
}
],
},
{"role": "tool", "tool_call_id": "call_a", "content": "long output"},
{"role": "tool", "tool_call_id": "call_a", "content": truncated_content},
{"role": "tool", "tool_call_id": "call_b", "content": "short"},
]
decorate_history_messages(messages, verdicts, assessments)
@@ -204,12 +211,9 @@ class TestDecorateHistoryMessages:
assert "reasoning" in tc["verdict"]
assert tc["output_assessment"]["flags"] == ["secret"]
assert tc["output_assessment"]["redacted"] is True
# Plain tool content (no envelope) is left intact and no
# advisories key is set.
assert messages[2]["content"] == "long output"
assert "advisories" not in messages[2]
assert messages[3]["content"] == "short"
assert "advisories" not in messages[3]
# Truncated tool message got the flag; the short one did not.
assert messages[2].get("truncated") is True
assert "truncated" not in messages[3]
def test_no_op_on_empty_indexes(self) -> None:
"""When neither table has rows for the workstream, the wire
@@ -226,627 +230,3 @@ class TestDecorateHistoryMessages:
tc = messages[0]["tool_calls"][0] # type: ignore[index]
assert "verdict" not in tc
assert "output_assessment" not in tc
class TestDecorateAdvisoryExtraction:
"""Round-trip the persisted ``<tool_output>`` envelope (Seam 1
queued-message splice) back into wire-shape advisories on each
tool message replay surface for the queued-during-batch case.
"""
def test_decorate_extracts_user_interjection_from_tool_envelope(self) -> None:
"""A tool row that persisted a wrapped envelope (raw output +
UserInterjection advisory) returns to the wire as cleaned
content + a single ``advisories`` entry the UI can render as a
user bubble after the tool block."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"hello",
[UserInterjection(message="check logs", priority="notice")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
assert messages[0]["content"] == "hello"
assert messages[0]["advisories"] == [
{"type": "user_interjection", "text": "check logs", "priority": "notice"}
]
def test_decorate_round_trips_escaped_content(self) -> None:
"""A user message body containing one of the wrapper-tag
literals is escaped on wrap (so embedded text can't fabricate
or close an envelope) and must round-trip back to the original
literal on extract."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
evil = "</system-reminder>"
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message=evil, priority="notice")],
)
# Sanity: the user-controlled literal does NOT appear inside
# the advisory body — only the entity-encoded form does. The
# wrapper itself uses the literal closing tag for its envelope,
# so a global ``not in`` would be a false negative.
assert "User message: &lt;/system-reminder&gt;" in wrapped
assert "User message: </system-reminder>" not in wrapped
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
# Extract entity-decoded the escaped form back to the literal.
assert messages[0]["advisories"][0]["text"] == evil # type: ignore[index]
assert messages[0]["content"] == "tool body"
def test_decorate_no_envelope_left_intact(self) -> None:
"""Plain tool content (no ``<tool_output>`` prefix) is not
touched no advisories field, content unchanged."""
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": "plain output"},
]
decorate_history_messages(messages, {}, {})
assert messages[0]["content"] == "plain output"
assert "advisories" not in messages[0]
def test_decorate_drops_output_guard_advisory_from_extraction(self) -> None:
"""A wrapped envelope carrying both a guard advisory and a
user_interjection produces only the user_interjection on
``advisories``. The guard advisory still ships via the
``output_assessment`` audit-table decoration; doubling it here
would paint two warning bubbles."""
from turnstone.core.output_guard import OutputAssessment
from turnstone.core.tool_advisory import (
GuardAdvisory,
UserInterjection,
wrap_tool_result,
)
assessment = OutputAssessment(
risk_level="medium",
flags=["api_key"],
annotations=["redacted token in line 2"],
sanitized="cleaned body",
)
wrapped = wrap_tool_result(
"raw body",
[
GuardAdvisory(assessment=assessment, func_name="bash"),
UserInterjection(message="and here", priority="notice"),
],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
adv = messages[0]["advisories"]
assert len(adv) == 1 # type: ignore[arg-type]
assert adv[0]["type"] == "user_interjection" # type: ignore[index]
def test_decorate_handles_important_priority(self) -> None:
"""The MUST-address preamble round-trips to ``priority=important``."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"out",
[UserInterjection(message="urgent", priority="important")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
adv = messages[0]["advisories"][0] # type: ignore[index]
assert adv["priority"] == "important"
assert adv["text"] == "urgent"
def test_decorate_suppresses_empty_advisory_body(self) -> None:
"""``queue_message`` doesn't reject empty / whitespace-only
text, so an advisory with an empty body can round-trip through
``wrap_tool_result``. ``_classify_advisory`` must filter those
out so replay doesn't paint a featureless empty user bubble.
Removing the ``if not body.strip(): return None`` guard in
``_classify_advisory`` breaks this test."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message="", priority="notice")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
# Envelope is still stripped from content (the cleaning side
# of decoration runs unconditionally), but no advisories
# surface — the empty body is filtered.
assert messages[0]["content"] == "tool body"
assert "advisories" not in messages[0]
def test_decorate_suppresses_whitespace_only_advisory_body(self) -> None:
"""Whitespace-only bodies are similarly suppressed — same
reasoning as the empty-body case."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message=" \n\t ", priority="notice")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
assert messages[0]["content"] == "tool body"
assert "advisories" not in messages[0]
def test_wrap_extract_round_trips_preexisting_entities(self) -> None:
"""A user message body containing literal HTML-entity references
matching the wrapper-escape forms must round-trip identically
through ``wrap_tool_result + extract_advisories_from_tool_envelope``.
Without escaping ``&`` first in the encode step, encodedecode
would produce the bare wrapper tag, fabricating an envelope the
wrapper layer never produced.
"""
from turnstone.core.history_decoration import (
extract_advisories_from_tool_envelope,
)
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
tricky = "I describe XML tags like &lt;tool_output&gt; in my docs."
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message=tricky, priority="notice")],
)
result = extract_advisories_from_tool_envelope(wrapped)
assert result is not None
cleaned, advisories = result
assert cleaned == "tool body"
assert len(advisories) == 1
# The original literal entity-reference text round-trips
# identically — the parser does not silently turn it into a
# bare wrapper tag.
assert advisories[0]["text"] == tricky
def test_save_load_decorate_round_trips_envelope(self, backend) -> None:
"""End-to-end round-trip pinning the persisted-envelope
contract. Persists a wrapped tool-output envelope via
``save_message``, loads via ``load_messages``, runs
``decorate_history_messages``, asserts the wire shape carries
the extracted advisory + cleaned content. Pins the contract
every component in the chain participates in (persistence
layer in-memory replay wire projection) so a schema drift,
an envelope-format change, or a parser regression surfaces
here rather than only in production.
"""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"command output",
[UserInterjection(message="check the logs", priority="notice")],
)
backend.register_workstream("ws_rt_1")
backend.save_message("ws_rt_1", "user", "go")
backend.save_message(
"ws_rt_1",
"assistant",
None,
tool_calls='[{"id":"call_a","type":"function","function":{"name":"bash","arguments":"{}"}}]',
)
backend.save_message(
"ws_rt_1",
"tool",
wrapped,
tool_call_id="call_a",
)
msgs = backend.load_messages("ws_rt_1")
# Persisted shape — content survives the storage layer
# untouched. Symmetry with in-memory ``self.messages[i]['content']``
# is what makes envelope extraction lossless on replay.
tool_msg = next(m for m in msgs if m["role"] == "tool")
assert tool_msg["content"] == wrapped
# Decorate (the /history shared transform) — extracts the
# advisory and strips the envelope.
decorate_history_messages(msgs, {}, {})
tool_msg = next(m for m in msgs if m["role"] == "tool")
assert tool_msg["content"] == "command output"
assert tool_msg["advisories"] == [
{"type": "user_interjection", "text": "check the logs", "priority": "notice"}
]
class TestExtractReasoningForHistory:
"""``extract_reasoning_for_history`` — Phase 1 surfaces stored
Anthropic thinking blocks on assistant messages and strips
``_provider_content`` from the wire payload.
Drives through the real ``AnthropicProvider.extract_reasoning_text``
(no mock-of-extractor) the helper test and the provider unit
test (``tests/test_provider_anthropic_reasoning.py``) together
catch a regression at either layer distinctly.
"""
def _anthropic_thinking_msg(self, text: str = "let me think") -> dict[str, object]:
return {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": text, "signature": "sig"},
{"type": "text", "text": "Final answer."},
],
}
def test_extract_thinking_surfaces_reasoning_field(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("let me think")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "let me think"
def test_strips_provider_content_after_extraction(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("anything")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "_provider_content" not in messages[0]
def test_strips_provider_content_when_flag_false(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("anything")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=False)
# Strip is unconditional; reasoning is the conditional bit.
assert "_provider_content" not in messages[0]
assert "reasoning" not in messages[0]
def test_first_block_thinking_dispatches_to_anthropic(self) -> None:
# Even when text and tool_use blocks follow, the first-block-type
# discriminator routes thinking-prefixed payloads correctly.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "first", "signature": "s"},
{"type": "text", "text": "spoken"},
{"type": "tool_use", "id": "t1", "name": "f", "input": {}},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "first"
def test_first_block_reasoning_dispatches_to_openai_responses(self) -> None:
# Phase 3: dispatcher routes type=="reasoning" to the
# OpenAI Responses extractor, which now returns the
# summary[*].text concatenation. Pre-Phase-3 this asserted
# "" (the stub); the assertion was tightened once the wire
# path landed.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "s"}]}
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "s"
assert "_provider_content" not in messages[0]
def test_unknown_first_block_type_no_op(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [{"type": "text", "text": "no reasoning here"}],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "_provider_content" not in messages[0]
def test_skips_messages_without_provider_content(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [{"role": "assistant", "content": "plain"}]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert messages[0]["content"] == "plain"
def test_user_and_tool_messages_untouched(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages: list[dict[str, object]] = [
{"role": "user", "content": "hi"},
{"role": "tool", "tool_call_id": "c1", "content": "out"},
self._anthropic_thinking_msg("only this one"),
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "reasoning" not in messages[1]
assert messages[2]["reasoning"] == "only this one"
def test_empty_provider_content_no_extraction(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [{"role": "assistant", "content": "x", "_provider_content": []}]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
# Empty-list provider_content is still stripped from the wire.
assert "_provider_content" not in messages[0]
def test_first_block_not_a_dict_skipped(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages: list[dict[str, object]] = [
{
"role": "assistant",
"content": "x",
"_provider_content": ["bogus"],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "_provider_content" not in messages[0]
def test_first_block_reasoning_text_dispatches_to_openai_chat(self) -> None:
# Phase 3 path 3: synthetic ``reasoning_text`` blocks (stamped
# by ChatSession._maybe_synth_reasoning_block for vLLM /
# llama.cpp / Gemini-compat conversations) dispatch to
# OpenAIChatCompletionsProvider.extract_reasoning_text.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "synth thought"
assert "_provider_content" not in messages[0]
def test_dispatcher_scans_past_unrecognized_first_blocks(self) -> None:
# Regression for Copilot finding: dispatcher used to inspect
# only provider_content[0]['type']. OpenAI Responses captures
# EVERY output_item.done event into provider_blocks (not just
# reasoning), so a hypothetical [message, reasoning, ...]
# ordering would have silently dropped the reasoning. Now
# walks the list for the first recognised reasoning-bearing
# type and dispatches the whole list to that provider.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
# First block is a non-reasoning OpenAI Responses item.
{"type": "message", "role": "assistant", "content": "answer"},
# Reasoning sits later in the list.
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "deferred"}],
},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "deferred"
assert "_provider_content" not in messages[0]
def test_first_block_redacted_thinking_dispatches_to_anthropic(self) -> None:
# Anthropic's extended-thinking API documents that
# ``redacted_thinking`` blocks (sealed by the safety system)
# can appear before, after, or interleaved with regular
# ``thinking`` blocks. When the redacted block lands first,
# the dispatcher must still route to AnthropicProvider so the
# surrounding real thinking text surfaces — without this the
# reasoning bubble silently disappears on history rehydration.
# Pinned by registering "redacted_thinking" as a second key
# in _BLOCK_TYPE_PROVIDER_FACTORY pointing at the Anthropic
# factory; Anthropic's extractor's type=="thinking" filter
# already correctly skips the redacted block.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
{"type": "redacted_thinking", "data": "sealed-blob"},
{"type": "thinking", "thinking": "real thought", "signature": "s"},
{"type": "text", "text": "answer"},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "real thought"
assert "_provider_content" not in messages[0]
class TestAttachVllmChatReasoningField:
"""``attach_vllm_chat_reasoning_field`` — Phase 5 surfaces persisted
reasoning as the vLLM-specific ``reasoning`` field on outgoing
assistant messages so vLLM-served reasoning models can thread CoT
across turns.
Drives through the real ``extract_reasoning_text_from_provider_content``
dispatcher no extractor mocks so a regression in either layer
surfaces distinctly. All 3 caller-side gates (provider isinstance,
server_type, operator flag) are exercised by
``test_session_chat_reasoning_replay.py``; this class pins the
helper's projection contract in isolation.
"""
def _assistant_with(self, provider_content: list[dict[str, object]]) -> dict[str, object]:
return {
"role": "assistant",
"content": "Final answer.",
"_provider_content": provider_content,
}
def test_synthetic_reasoning_text_attaches_field(self) -> None:
# Path 3 capture (vLLM --reasoning-parser, llama.cpp
# reasoning_format, Gemini-compat) lands in _provider_content as
# a synthetic reasoning_text block; helper must round-trip it
# back onto the same model on the next turn.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs = [self._assistant_with([{"type": "reasoning_text", "text": "synth thought"}])]
out = attach_vllm_chat_reasoning_field(msgs)
assert out[0]["reasoning"] == "synth thought"
def test_anthropic_thinking_attaches_field(self) -> None:
# Cross-provider switch: workstream started with Anthropic,
# operator flipped model to a vLLM-served reasoning model.
# Helper extracts the thinking text and discards the signature
# (vLLM doesn't validate signatures).
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs = [
self._assistant_with(
[
{"type": "thinking", "thinking": "claude was here", "signature": "sig"},
{"type": "text", "text": "answer"},
]
)
]
out = attach_vllm_chat_reasoning_field(msgs)
assert out[0]["reasoning"] == "claude was here"
# Signature is dropped at extraction; ``reasoning`` field carries
# plain text only.
assert "sig" not in out[0]["reasoning"]
def test_openai_responses_reasoning_attaches_field(self) -> None:
# Cross-provider switch: workstream started on gpt-5, operator
# flipped to a vLLM-served model. Helper extracts the
# summary[*].text concatenation.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs = [
self._assistant_with(
[
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "responses thought"}],
}
]
)
]
out = attach_vllm_chat_reasoning_field(msgs)
assert out[0]["reasoning"] == "responses thought"
def test_no_provider_content_returns_unchanged(self) -> None:
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs: list[dict[str, object]] = [{"role": "assistant", "content": "plain"}]
out = attach_vllm_chat_reasoning_field(msgs)
assert "reasoning" not in out[0]
# No copy made when there's nothing to attach — same object.
assert out[0] is msgs[0]
def test_empty_provider_content_returns_unchanged(self) -> None:
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs: list[dict[str, object]] = [
{"role": "assistant", "content": "x", "_provider_content": []}
]
out = attach_vllm_chat_reasoning_field(msgs)
assert "reasoning" not in out[0]
assert out[0] is msgs[0]
def test_unknown_block_type_returns_unchanged(self) -> None:
# _provider_content has blocks but none are reasoning-bearing.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs: list[dict[str, object]] = [
self._assistant_with([{"type": "text", "text": "no reasoning here"}])
]
out = attach_vllm_chat_reasoning_field(msgs)
assert "reasoning" not in out[0]
assert out[0] is msgs[0]
def test_does_not_touch_user_tool_system_messages(self) -> None:
# Only assistant messages get the reasoning field. User / tool /
# system messages pass through by reference.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
msgs: list[dict[str, object]] = [
{"role": "system", "content": "sys"},
{"role": "user", "content": "hi"},
{"role": "tool", "tool_call_id": "c1", "content": "out"},
# Even an assistant-shaped non-assistant role (defensive — shouldn't happen)
# must not have provider_content read.
]
out = attach_vllm_chat_reasoning_field(msgs)
assert "reasoning" not in out[0]
assert "reasoning" not in out[1]
assert "reasoning" not in out[2]
# All three return by reference (no allocation when no attach).
for original, returned in zip(msgs, out, strict=True):
assert original is returned
def test_preserves_provider_content_for_downstream_sanitize(self) -> None:
# Helper attaches ``reasoning`` but leaves ``_provider_content``
# in place. Downstream ``sanitize_messages`` (in the provider's
# _prepare_messages) strips the ``_``-prefixed sibling key
# before the wire payload leaves. Helper isn't responsible for
# that strip — composition with sanitize is the contract.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
original_content = [{"type": "reasoning_text", "text": "kept"}]
msgs = [self._assistant_with(original_content)]
out = attach_vllm_chat_reasoning_field(msgs)
assert out[0]["reasoning"] == "kept"
# Provider content survives on the helper's output dict.
assert out[0]["_provider_content"] == original_content
def test_does_not_mutate_input_messages(self) -> None:
# Pure transform: input list and input dicts are untouched.
# Callers can keep iterating the original list without surprise.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
original = self._assistant_with([{"type": "reasoning_text", "text": "x"}])
msgs = [original]
attach_vllm_chat_reasoning_field(msgs)
assert "reasoning" not in original
# Original dict untouched even though the function returned a
# modified copy.
def test_mixed_messages_only_attaches_to_assistants_with_reasoning(self) -> None:
# Realistic shape: a workstream with user, assistant-with-reasoning,
# tool, assistant-plain, user. Only the first assistant gets the
# reasoning field; everything else passes through by reference.
from turnstone.core.history_decoration import attach_vllm_chat_reasoning_field
with_reasoning = self._assistant_with([{"type": "reasoning_text", "text": "thinking"}])
plain_assistant: dict[str, object] = {"role": "assistant", "content": "second"}
msgs: list[dict[str, object]] = [
{"role": "user", "content": "q1"},
with_reasoning,
{"role": "tool", "tool_call_id": "c1", "content": "result"},
plain_assistant,
{"role": "user", "content": "q2"},
]
out = attach_vllm_chat_reasoning_field(msgs)
assert out[0] is msgs[0]
assert out[1]["reasoning"] == "thinking"
assert out[1] is not with_reasoning # new dict for the attached one
assert out[2] is msgs[2]
assert out[3] is plain_assistant
assert "reasoning" not in out[3]
assert out[4] is msgs[4]
-391
View File
@@ -1,391 +0,0 @@
"""Boundary-crossing integration test for the wake trigger pipeline.
Drives a *real* :class:`SessionManager` + a *real* :class:`ChatSession`
+ a *real* :class:`IdleNudgeWatcher` end-to-end. The only stub is the
LLM provider (patched ``_create_stream_with_retry``); every other layer
is production code:
* ``SessionManager.set_state`` snapshotting + iterating subscribers
* ``IdleNudgeWatcher._on_state`` peeking the queue
* ``session_worker.send`` atomic-spawn + daemon thread
* ``ChatSession.deliver_wake_nudge_from_queue`` opening / closing
``_wake_source_tag``
* ``ChatSession.send`` chat loop short-circuiting metacog detection
* ``_append_user_turn`` stamping ``_source = "system_nudge"``
* ``_attach_pending_user_reminders`` draining ``USER_DRAIN``
* ``_apply_reminders_for_provider`` splicing the rendered envelope
onto empty content
Per ``feedback_tests_through_boundaries.md``: direct injection tests
that bypass these boundaries silently mask wiring bugs. This test is
the structural integration gate.
"""
from __future__ import annotations
import time
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests.test_session_manager import FakeStorage
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core.session import ChatSession
from turnstone.core.session_manager import SessionManager
from turnstone.core.workstream import Workstream, WorkstreamKind, WorkstreamState
# ---------------------------------------------------------------------------
# Minimal fake adapter / UI for this integration test. Storage reuses
# the canonical FakeStorage from test_session_manager.py to avoid the
# drift risk of a parallel fake.
# ---------------------------------------------------------------------------
class _FakeUI:
"""Minimal UI surface for ChatSession + SessionManager.cleanup_ui."""
def __init__(self) -> None:
self.events: list[tuple[str, Any]] = []
def _unblock(self) -> None: # SessionManager.close calls this
pass
def broadcast_ws_closed(self) -> None:
pass
# ChatSession callbacks (no-op for this test)
def on_turn_start(self) -> None:
pass
def on_turn_committed(self) -> None:
pass
def on_thinking_start(self) -> None:
pass
def on_thinking_end(self) -> None:
pass
def on_state_change(self, state: str) -> None:
self.events.append(("state", state))
def on_user_reminder(self, reminders: Any, source: str | None = None) -> None:
self.events.append(("user_reminder", reminders, source))
def on_error(self, message: str) -> None:
pass
def on_rename(self, name: str) -> None:
pass
def on_output_warning(self, call_id: Any, assessment: Any) -> None:
pass
def __getattr__(self, name: str) -> Any:
# Catch-all for any UI hook not enumerated above so the chat
# loop's ``self.ui.<something>()`` call doesn't blow up.
return MagicMock()
class _BuildRealSessionAdapter:
"""Adapter that returns a real :class:`ChatSession` instead of a stub.
Tracks emit_* events the integration test asserts on. Mirrors the
``SessionKindAdapter`` + ``SessionEventEmitter`` Protocol surface
that production ``WebUI`` / coord adapters expose.
"""
def __init__(self, kind: WorkstreamKind = WorkstreamKind.INTERACTIVE) -> None:
self.kind = kind
self.events: list[str] = []
self.cleaned_up: list[str] = []
def emit_created(self, ws: Workstream) -> None:
self.events.append(f"created:{ws.id}")
def emit_rehydrated(self, ws: Workstream) -> None:
self.events.append(f"rehydrated:{ws.id}")
def emit_state(self, ws: Workstream, state: WorkstreamState) -> None:
self.events.append(f"state:{ws.id}:{state.value}")
def emit_closed(self, ws_id: str, *, reason: str = "closed", name: str = "") -> None:
self.events.append(f"closed:{ws_id}")
def cleanup_ui(self, ws: Workstream) -> None:
# Real production cleanup_ui calls ws.session.cancel() + close().
# We don't need that here — the test exits cleanly via pytest
# teardown without exercising the cleanup path. Just record
# the call for any test that wants to assert on it.
self.cleaned_up.append(ws.id)
def build_ui(self, ws: Workstream) -> Any:
return _FakeUI()
def build_session(
self,
ws: Workstream,
*,
skill: Any = None,
model: Any = None,
client_type: Any = None,
**extra: Any,
) -> Any:
# Mirror SessionManager.create's keyword set so config-threading
# bugs surface here rather than being silently swallowed by
# **kwargs. ``model`` flows to the real ChatSession; the rest
# are accepted but not used by this test.
client = MagicMock()
return ChatSession(
client=client,
model=str(model) if model else "test-model",
ui=ws.ui,
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
# ---------------------------------------------------------------------------
# Test
# ---------------------------------------------------------------------------
@pytest.fixture
def real_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter]:
"""Real SessionManager wired to an adapter that builds real ChatSessions.
No StateWriter is wired so ``set_state`` writes directly to storage
on the calling thread (we want subscriber dispatch to fire in the
same thread the test invokes ``set_state`` on).
"""
adapter = _BuildRealSessionAdapter()
storage = FakeStorage()
mgr = SessionManager(
adapter,
storage=storage,
max_active=5,
event_emitter=adapter,
)
return mgr, adapter
def _wait_for_worker_done(ws: Workstream, timeout: float = 5.0) -> None:
"""Poll ``ws._worker_running`` until it clears or timeout elapses."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
with ws._lock:
if not ws._worker_running:
return
time.sleep(0.01)
raise AssertionError(f"worker thread for ws={ws.id[:8]} didn't exit within {timeout}s")
def test_idle_event_through_real_session_manager_drives_wake_send(real_mgr, tmp_db):
"""The full wake pipeline, no direct-injection shortcuts.
Boundary path under test:
enqueue mgr.set_state(IDLE)
SessionManager._state_subscribers iteration (real)
IdleNudgeWatcher._on_state (real)
session_worker.send (real)
real daemon thread
ChatSession.deliver_wake_nudge_from_queue (real)
ChatSession.send("") (real, with patched LLM stream)
_append_user_turn stamps ``_source``
_attach_pending_user_reminders drains ``{"user","any"}``
_apply_reminders_for_provider splices envelope onto empty content
"""
mgr, _adapter = real_mgr
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
ws = mgr.create(user_id="u1", name="wake-int", skill=None)
assert ws.session is not None
# Patch the LLM-facing surface so send() runs the chat loop end-to-end
# without any real provider. We patch on the just-built ChatSession;
# the patches are reverted by the `with` block.
with (
patch.object(ws.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
ws.session,
"_stream_response",
return_value={"role": "assistant", "content": "ok"},
),
patch.object(ws.session, "_update_token_table"),
patch.object(ws.session, "_print_status_line"),
patch.object(ws.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
# Suppress the auto-title side-thread; orthogonal to wake.
ws.session._title_generated = True
# Enqueue an any-channel nudge — the future ``idle_children`` shape.
ws.session._nudge_queue.enqueue("idle_children", "your kids", "any")
assert len(ws.session._nudge_queue) == 1
# Trigger IDLE. This runs subscriber dispatch synchronously on
# the calling thread → IdleNudgeWatcher._on_state → session_worker.send
# → spawn daemon thread → deliver_wake_nudge_from_queue.
mgr.set_state(ws.id, WorkstreamState.IDLE)
# Wait for the daemon thread to clear ``_worker_running`` so the
# post-conditions are stable.
_wait_for_worker_done(ws)
# Queue fully drained by the wake.
assert len(ws.session._nudge_queue) == 0
# The synthesized empty user message landed in history with the
# ``_source`` audit tag and the reminder side-channel populated.
user_msgs = [m for m in ws.session.messages if m.get("role") == "user"]
assert user_msgs, "expected a synthesized user message from the wake"
wake_msg = user_msgs[-1]
assert wake_msg["content"] == ""
assert wake_msg.get("_source") == "system_nudge"
assert wake_msg.get("_reminders") == [{"type": "idle_children", "text": "your kids"}]
# The wake-source tag is reset post-send so subsequent activity
# behaves normally.
assert ws.session._wake_source_tag == ""
finally:
watcher.shutdown()
def test_idle_event_with_empty_queue_does_not_dispatch_wake(real_mgr, tmp_db):
"""Non-empty queue is the gate. An IDLE event on a workstream with
nothing queued must NOT call ``session_worker.send``.
Patches the dispatch primitive directly rather than racing a
``time.sleep`` against an erroneous spawn the question is
whether the watcher's gate fired, which is a deterministic
decision the patch captures.
"""
mgr, _adapter = real_mgr
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
ws = mgr.create(user_id="u1", name="empty-int", skill=None)
# No enqueue.
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.set_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0, "wake must not dispatch for an empty queue"
finally:
watcher.shutdown()
@pytest.fixture
def coord_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter, FakeStorage]:
"""Real coord-side SessionManager with the adapter's kind set to
COORDINATOR. Same shape as ``real_mgr`` but for the coord half of
the lifespan. No StateWriter wired so subscriber dispatch fires
synchronously on the test thread.
"""
adapter = _BuildRealSessionAdapter(kind=WorkstreamKind.COORDINATOR)
storage = FakeStorage()
mgr = SessionManager(
adapter,
storage=storage,
max_active=5,
event_emitter=adapter,
)
return mgr, adapter, storage
def test_coord_idle_with_active_children_emits_envelope_via_real_managers(coord_mgr, tmp_db):
"""Full coord-path integration test (matches design doc §7.4).
Drives the production install order ``CoordinatorIdleObserver``
registered FIRST, then ``IdleNudgeWatcher`` and asserts the
full chain: observer enqueues on IDLE watcher peeks wake
spawns a worker ``deliver_wake_nudge_from_queue`` drains and
runs the synthetic empty-user turn reminder envelope reaches
the synthesized user message via the side-channel.
The boundary-crossing path tested here mirrors what
``console/server.py``'s lifespan does at production startup; if
the install order is ever reversed, this test fails.
"""
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
from turnstone.core.workstream import WorkstreamKind as _Kind
mgr, adapter, storage = coord_mgr
# Observer FIRST, then watcher. Same order as
# ``console/server.py:4435-4443`` — production correctness depends
# on subscribers firing in registration order on the same IDLE.
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
coord = mgr.create(user_id="u1", name="parent-coord", skill=None)
assert coord.session is not None
# Two interactive children of the coord, both running. Use
# the storage's register_workstream API so the rows match
# production shape (the observer queries via list_workstreams).
storage.register_workstream(
"child-a",
user_id="u1",
name="research-pricing",
kind=_Kind.INTERACTIVE,
parent_ws_id=coord.id,
state="running",
)
storage.register_workstream(
"child-b",
user_id="u1",
name="draft-rfc",
kind=_Kind.INTERACTIVE,
parent_ws_id=coord.id,
state="thinking",
)
# Pretend the coord has already had a real conversation so
# ``should_nudge``'s message_count > 1 gate passes.
coord.session.messages.append({"role": "user", "content": "spawn 2"})
coord.session.messages.append({"role": "assistant", "content": "ok"})
with (
patch.object(coord.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
coord.session,
"_stream_response",
return_value={"role": "assistant", "content": "ack"},
),
patch.object(coord.session, "_full_messages", return_value=[]),
patch.object(coord.session, "_update_token_table"),
patch.object(coord.session, "_print_status_line"),
patch.object(coord.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
coord.session._title_generated = True
mgr.set_state(coord.id, WorkstreamState.IDLE)
_wait_for_worker_done(coord)
# Queue drained — the wake delivered the observer's enqueue.
assert len(coord.session._nudge_queue) == 0
# The synthetic empty-user turn landed with a reminder containing
# both children.
user_msgs = [m for m in coord.session.messages if m.get("role") == "user"]
# Two real msgs (user + assistant context above) plus the wake.
wake_msg = user_msgs[-1]
assert wake_msg["content"] == ""
assert wake_msg.get("_source") == "system_nudge"
reminders = wake_msg.get("_reminders") or []
assert len(reminders) == 1
assert reminders[0]["type"] == "idle_children"
text = reminders[0]["text"]
assert "research-pricing" in text
assert "draft-rfc" in text
assert "child-a" in text
assert "child-b" in text
assert "wait_for_workstream" in text
finally:
watcher.shutdown()
observer.shutdown()
-165
View File
@@ -1,165 +0,0 @@
"""Unit tests for :class:`IdleNudgeWatcher`.
Drives a fake :class:`SessionManager` that mimics the real one's
``subscribe_to_state`` / ``get`` contract. The watcher itself
dispatches via ``turnstone.core.session_worker.send``; we patch that
module-level function to capture calls without spawning real threads.
"""
from __future__ import annotations
import contextlib
import threading
from typing import Any
from unittest.mock import patch
import pytest
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core.nudge_queue import NudgeQueue
from turnstone.core.workstream import WorkstreamState
class _FakeSession:
def __init__(self) -> None:
self._nudge_queue = NudgeQueue()
self.deliver_wake_nudge_from_queue_called = 0
def deliver_wake_nudge_from_queue(self) -> None:
self.deliver_wake_nudge_from_queue_called += 1
class _FakeWorkstream:
def __init__(self, ws_id: str = "ws-test") -> None:
self.id = ws_id
self.session: _FakeSession | None = _FakeSession()
self._lock = threading.Lock()
self._worker_running = False
self._closed = False
self.worker_thread: Any = None
class _FakeManager:
"""Mimics SessionManager's subscribe-to-state surface without a DB."""
def __init__(self) -> None:
self._workstreams: dict[str, _FakeWorkstream] = {}
self._subscribers: list[Any] = []
self._subscribers_lock = threading.Lock()
def add_ws(self, ws: _FakeWorkstream) -> None:
self._workstreams[ws.id] = ws
def get(self, ws_id: str) -> _FakeWorkstream | None:
return self._workstreams.get(ws_id)
def subscribe_to_state(self, callback: Any) -> None:
with self._subscribers_lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Any) -> None:
with self._subscribers_lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
def fire_state(self, ws_id: str, state: WorkstreamState) -> None:
"""Mirror SessionManager.set_state's subscriber-fan-out behaviour."""
with self._subscribers_lock:
subs = list(self._subscribers)
for cb in subs:
# Match contextlib.suppress(Exception) in real SessionManager.
with contextlib.suppress(Exception):
cb(ws_id, state)
@pytest.fixture
def fake_mgr_and_ws() -> tuple[_FakeManager, _FakeWorkstream]:
mgr = _FakeManager()
ws = _FakeWorkstream()
mgr.add_ws(ws)
return mgr, ws
class TestIdleNudgeWatcher:
def test_idle_event_with_empty_queue_no_op(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_idle_event_with_pending_nudge_dispatches(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("idle_children", "your kids", "any")
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 1
kwargs = mock_send.call_args.kwargs
# `enqueue=lambda: None` — verify by calling and checking no-op.
assert kwargs["enqueue"]() is None
# `run` should call deliver_wake_nudge_from_queue when invoked.
kwargs["run"]()
assert ws.session.deliver_wake_nudge_from_queue_called == 1
assert kwargs["thread_name"].startswith("wake-nudge-")
def test_non_idle_state_ignored(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("foo", "bar", "any")
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
for state in (
WorkstreamState.RUNNING,
WorkstreamState.THINKING,
WorkstreamState.ATTENTION,
WorkstreamState.ERROR,
):
mgr.fire_state(ws.id, state)
assert mock_send.call_count == 0
def test_unknown_ws_ignored(self, fake_mgr_and_ws):
mgr, _ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state("ghost", WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_session_none_ignored(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session = None # workstream loaded but session not yet built
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_start_is_idempotent(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
watcher.start() # no-op
ws.session._nudge_queue.enqueue("foo", "bar", "any")
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Only one subscriber was registered despite the double-start.
assert mock_send.call_count == 1
def test_shutdown_unsubscribes(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("foo", "bar", "any")
watcher = IdleNudgeWatcher(mgr)
watcher.start()
watcher.shutdown()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_shutdown_is_idempotent(self, fake_mgr_and_ws):
mgr, _ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
watcher.shutdown()
watcher.shutdown() # no error
-52
View File
@@ -777,58 +777,6 @@ class TestModelAliasResolution:
assert judge._client_factory_args["api_key"] == "alias-key"
assert judge._client_factory_args["provider_name"] == "openai"
def test_unknown_alias_inherits_session_model(self):
"""``judge.model`` is alias-only. A value that doesn't resolve
through the registry inherits the session model (same path as
an empty config.model) rather than getting pinned onto the
session provider as a raw model id that legacy behavior
silently broke whenever the session provider didn't speak the
configured model id (Anthropic session, ``judge.model =
"gpt-5-mini"`` every verdict came back as ``llm_fallback``)."""
session_provider = _make_mock_provider()
session_provider.provider_name = "anthropic"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
registry = MagicMock()
registry.has_alias.return_value = False # judge.model isn't an alias
config = JudgeConfig(enabled=True, model="gpt-5-mini")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
model_registry=registry,
)
assert judge._provider is session_provider
assert judge._model == "session-default-model"
# Context window mirrors the session, not the (uncalled) caps lookup.
assert judge._judge_context_window == 100_000
def test_empty_model_inherits_session_model(self):
"""Empty ``config.model`` is the documented self-consistency path."""
session_provider = _make_mock_provider()
session_provider.provider_name = "openai"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
config = JudgeConfig(enabled=True, model="")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
)
assert judge._provider is session_provider
assert judge._model == "session-default-model"
def test_coordinator_tool_call_returns_llm_verdict_not_fallback(self):
"""Happy-path regression for coordinator tool calls: with a properly
resolved provider, the verdict tier must be ``llm`` the
+1 -122
View File
@@ -51,11 +51,7 @@ class TestIntentVerdictCRUD:
assert v["tier"] == "heuristic"
assert v["judge_model"] == ""
assert v["latency_ms"] == 2
# ``user_decision`` defaults to ``"pending"`` (not the empty
# string) so an audit reader can distinguish in-flight rows
# from pre-convention legacy rows that carry the column's
# server_default of ``""``.
assert v["user_decision"] == "pending"
assert v["user_decision"] == ""
assert "created" in v
def test_get_nonexistent(self, db):
@@ -118,123 +114,6 @@ class TestIntentVerdictCRUD:
assert ok is False
class TestIntentVerdictUpsert:
"""``upsert_intent_verdict`` — the LLM-tier-aware persistence path.
Backs the heuristic llm_fallback "upgrade in place" pattern.
The async judge's fallback verdicts deliberately reuse the
heuristic ``verdict_id``; a plain INSERT would collide on the
PK and the upgrade would be lost to a silently-swallowed
exception (Postgres logged ``intent_verdicts_pkey`` violations
for every fallback delivery on stable/1.5 smoke tests).
"""
def test_upsert_on_fresh_id_inserts(self, db):
"""No conflict — behaves like a regular INSERT."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
v = db.get_intent_verdict("v_001")
assert v is not None
assert v["tier"] == "heuristic"
assert v["user_decision"] == "pending"
def test_upsert_on_conflict_upgrades_tier_reasoning_judge_model(self, db):
"""On PK conflict: tier, reasoning, judge_model update — every
other field is preserved. Mirrors what the judge emits when
promoting heuristic llm_fallback."""
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="heuristic",
reasoning="initial heuristic reasoning",
judge_model="",
)
)
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="llm_fallback",
reasoning="initial heuristic reasoning (LLM judge did not return a verdict)",
judge_model="gpt-5-judge",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
# The three fields that should change.
assert v["tier"] == "llm_fallback"
assert "LLM judge did not return" in v["reasoning"]
assert v["judge_model"] == "gpt-5-judge"
def test_upsert_on_conflict_preserves_user_decision(self, db):
"""LOAD-BEARING: a manually-resolved approval (user_decision=
``"approved"``) or auto-approve-stamped row (user_decision=
``"policy"``/``"blanket"``/etc.) must NOT be clobbered back to
``"pending"`` when the late LLM-fallback verdict lands.
``IntentVerdict.to_dict()`` doesn't project user_decision, so
the upsert's defaulted ``"pending"`` would silently overwrite
the real value if user_decision were in the on-conflict
SET clause."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
ok = db.update_intent_verdict("v_001", user_decision="approved")
assert ok is True
# Simulate the late LLM-fallback delivery — same verdict_id,
# default user_decision (the IntentVerdict.to_dict() shape).
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="llm_fallback",
reasoning="extended (LLM judge did not return a verdict)",
judge_model="gpt-5-judge",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
assert v["user_decision"] == "approved" # NOT clobbered to "pending"
assert v["tier"] == "llm_fallback" # but the upgrade did land
def test_upsert_on_conflict_preserves_identity_and_carried_fields(self, db):
"""Identity columns (ws_id, call_id, func_name, func_args) and
carried-verbatim columns (intent_summary, risk_level,
confidence, recommendation, evidence, latency_ms) are
excluded from the on-conflict SET verify they aren't
changed even when the second upsert passes different values
(defensive against a future judge bug that ships divergent
carried fields)."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
db.upsert_intent_verdict(
**_make_verdict_kwargs(
# Same verdict_id (conflict trigger), divergent everything else.
ws_id="ws-different",
call_id="tc_different",
func_name="bash_v2",
func_args='{"command":"rm -rf /"}',
intent_summary="totally different summary",
risk_level="critical",
confidence=0.0,
recommendation="deny",
evidence='["dangerous"]',
latency_ms=99999,
# The three fields that DO update.
tier="llm_fallback",
reasoning="upgraded reasoning",
judge_model="judge-v2",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
# All preserved from the first upsert (identity + carried).
assert v["ws_id"] == "ws-abc"
assert v["call_id"] == "tc_001"
assert v["func_name"] == "bash"
assert v["func_args"] == '{"command":"echo hello"}'
assert v["intent_summary"] == "Echo a greeting to stdout"
assert v["risk_level"] == "low"
assert v["confidence"] == 0.85
assert v["recommendation"] == "approve"
assert v["evidence"] == '["The command only prints text."]'
assert v["latency_ms"] == 2
# Only the three updated.
assert v["tier"] == "llm_fallback"
assert v["reasoning"] == "upgraded reasoning"
assert v["judge_model"] == "judge-v2"
# ---------------------------------------------------------------------------
# Bulk insert
# ---------------------------------------------------------------------------
+2 -3
View File
@@ -420,9 +420,8 @@ class TestSkillCatalogDisclosure:
session.system_messages = []
session._agent_system_messages = []
session.reasoning_effort = "medium"
from turnstone.core.nudge_queue import NudgeQueue
session._nudge_queue = NudgeQueue()
session._pending_tool_advisories = []
session._pending_user_advisories = []
session._tool_search = None
session._mcp_client = None
session._notify_on_complete = "{}"
File diff suppressed because it is too large Load Diff
-211
View File
@@ -1,211 +0,0 @@
"""Integration tests for the Phase 9 admin bulk-revoke endpoint.
POST /v1/api/admin/mcp-servers/{name}/bulk-revoke clears every user's
OAuth token for a server (admin-side counterpart to the per-user
DELETE /v1/api/mcp/oauth/connections/{server_name} that shipped in
Phase 8).
Coverage:
- requires ``admin.mcp`` permission (401/403 without).
- 404 when the named server is missing.
- 400 when the server's ``auth_type`` is not ``oauth_user``.
- 200 + ``rows_deleted`` + ``consented_users_before`` on success.
- Audit row written with
``upstream_revoke_outcome="bulk_admin_no_upstream"``.
- Token rows are gone from ``mcp_user_tokens`` post-call.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.console.server import admin_mcp_bulk_revoke
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
class _InjectAdminMcp(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="admin-user",
scopes=frozenset({"approve"}),
token_source="config",
permissions=frozenset({"read", "write", "approve", "admin.mcp"}),
)
return await call_next(request)
class _InjectNoAdminMcp(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="regular-user",
scopes=frozenset({"approve"}),
token_source="jwt",
permissions=frozenset({"read", "write", "approve"}),
)
return await call_next(request)
def _build_app(storage: SQLiteBackend, *, with_admin_mcp: bool = True) -> Starlette:
mw = _InjectAdminMcp if with_admin_mcp else _InjectNoAdminMcp
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route(
"/api/admin/mcp-servers/{name}/bulk-revoke",
admin_mcp_bulk_revoke,
methods=["POST"],
),
],
),
],
middleware=[Middleware(mw)],
)
app.state.auth_storage = storage
return app
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
backend: SQLiteBackend,
*,
name: str = "srv-oauth",
server_id: str = "srv-oauth-id",
) -> None:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="oauth_user",
)
def _seed_static_server(
backend: SQLiteBackend,
*,
name: str = "srv-static",
server_id: str = "srv-static-id",
) -> None:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="static",
)
def _seed_user_tokens(backend: SQLiteBackend, server_name: str, users: int) -> None:
for i in range(users):
backend.create_mcp_user_token(
f"user-{i}",
server_name,
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=None,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
def test_requires_admin_mcp_permission(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage, with_admin_mcp=False))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 403
def test_404_on_missing_server(storage: SQLiteBackend) -> None:
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/never-existed/bulk-revoke")
assert resp.status_code == 404
assert resp.json() == {"error": "No such server"}
def test_400_on_static_server(storage: SQLiteBackend) -> None:
_seed_static_server(storage)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-static/bulk-revoke")
assert resp.status_code == 400
body = resp.json()
assert "oauth_user" in body["error"]
def test_400_on_invalid_server_name(storage: SQLiteBackend) -> None:
# double-underscore is reserved for the prefixed-tool-name encoding.
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/bad__name/bulk-revoke")
assert resp.status_code == 400
def test_200_on_success_with_no_consented_users(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
assert body["rows_deleted"] == 0
assert body["consented_users_before"] == 0
def test_200_clears_all_user_tokens(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_user_tokens(storage, "srv-oauth", users=3)
# Token for another server must survive the bulk-revoke.
_seed_oauth_server(storage, name="srv-other", server_id="srv-other-id")
_seed_user_tokens(storage, "srv-other", users=2)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
assert body["rows_deleted"] == 3
assert body["consented_users_before"] == 3
# Target server's tokens are gone; bystander's tokens survive.
assert storage.count_mcp_consented_users_by_server("srv-oauth") == 0
assert storage.count_mcp_consented_users_by_server("srv-other") == 2
def test_audits_with_bulk_admin_no_upstream(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_user_tokens(storage, "srv-oauth", users=2)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
# Pull the most-recent audit row for the bulk_revoked action and
# verify it carries the deferral marker.
events = storage.list_audit_events(limit=10)
bulk_rows = [e for e in events if e.get("action") == "mcp_server.oauth.bulk_revoked"]
assert len(bulk_rows) == 1
detail = bulk_rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("upstream_revoke_outcome") == "bulk_admin_no_upstream"
assert detail.get("rows_deleted") == 2
assert detail.get("consented_users_before") == 2
assert detail.get("name") == "srv-oauth"
+148 -931
View File
File diff suppressed because it is too large Load Diff
-117
View File
@@ -1,117 +0,0 @@
"""Structural gate against the Phase 7b sibling-bug pattern.
Phase 7b's bug-1 was a single ``f"MCP X error: {e}"`` site dropping a
structured-error JSON. Phase 8 introduces the ``consent_url`` field on
the same JSON envelope: every ``_structured_error(...)`` invocation
that emits ``mcp_consent_required`` or ``mcp_insufficient_scope`` MUST
also pass a ``consent_url=`` kwarg, otherwise the dashboard renderer
can't surface a re-consent button.
This test is purely structural it scans the source of
:mod:`turnstone.core.mcp_client` and asserts every consent-required /
insufficient-scope ``_structured_error`` call carries
``consent_url=``. It catches future regressions where a new exec path
adds a fourth call site and forgets the kwarg.
"""
from __future__ import annotations
import re
from pathlib import Path
import turnstone.core.mcp_client as _mcp_client_module
_USER_ACTIONABLE_CODES = ("mcp_consent_required", "mcp_insufficient_scope")
def _read_source() -> str:
path = Path(_mcp_client_module.__file__)
return path.read_text(encoding="utf-8")
def _find_structured_error_blocks(source: str) -> list[tuple[int, str]]:
"""Return ``(line_no, block)`` pairs for every ``_structured_error(...)``.
Each block is the call's argument list expanded across however many
lines the formatter chose. Uses a paren-counting walk so multi-line
kwargs and nested expressions are captured correctly.
"""
blocks: list[tuple[int, str]] = []
needle = "_structured_error("
idx = 0
while True:
loc = source.find(needle, idx)
if loc < 0:
break
# Skip the function definition itself.
if source[loc - 4 : loc] == "def ":
idx = loc + len(needle)
continue
line_no = source.count("\n", 0, loc) + 1
depth = 1
end = loc + len(needle)
while end < len(source) and depth > 0:
ch = source[end]
if ch == "(":
depth += 1
elif ch == ")":
depth -= 1
end += 1
blocks.append((line_no, source[loc:end]))
idx = end
return blocks
def test_every_user_actionable_structured_error_passes_consent_url() -> None:
source = _read_source()
blocks = _find_structured_error_blocks(source)
user_actionable_blocks = [
(ln, blk)
for ln, blk in blocks
if any(f'code="{code}"' in blk for code in _USER_ACTIONABLE_CODES)
]
# Sanity check: ensure we actually scanned the file the audit cares
# about (a stale path or import would otherwise silently pass with
# zero matches).
assert user_actionable_blocks, (
"No mcp_consent_required / mcp_insufficient_scope _structured_error "
"call sites found — has the audit been pointed at the wrong file?"
)
missing: list[tuple[int, str]] = []
for ln, blk in user_actionable_blocks:
if "consent_url=" not in blk:
# Strip whitespace and truncate so the failure message is
# readable in CI.
collapsed = re.sub(r"\s+", " ", blk).strip()
missing.append((ln, collapsed[:200]))
assert not missing, (
"Sibling-bug regression: the following consent-required / "
"insufficient-scope _structured_error sites are missing the "
"consent_url= kwarg.\n" + "\n".join(f" line {ln}: {snippet}" for ln, snippet in missing)
)
def test_audit_finds_all_known_user_actionable_sites() -> None:
"""Lock the count so accidental deletions are caught.
There are 13 user-actionable ``_structured_error`` call sites today
(4 each in the tool / resource / prompt token-classify branches +
3 in the post-retry-failed branches + 1 in ``_handle_auth_403``'s
insufficient-scope branch). If a new exec path is added the count
can rise; if a branch is removed the count can fall both are
fine, but require an intentional bump of this number to confirm
the change went through review.
"""
source = _read_source()
blocks = _find_structured_error_blocks(source)
user_actionable_count = sum(
1 for _, blk in blocks if any(f'code="{code}"' in blk for code in _USER_ACTIONABLE_CODES)
)
assert user_actionable_count == 13, (
f"Expected 13 user-actionable _structured_error sites, got "
f"{user_actionable_count}. If this is intentional, bump the "
f"expected count and document why in the commit message."
)
-235
View File
@@ -1,235 +0,0 @@
"""Tests for ``turnstone.core.mcp_crypto`` cipher + config loading.
Covers token-at-rest encryption for OAuth-MCP.
"""
from __future__ import annotations
import base64
import pytest
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenDecryptError,
MCPTokenKeyConfigError,
_key_fingerprint,
_validate_key,
load_mcp_token_cipher_config,
)
def _new_raw_key() -> bytes:
"""Return a fresh 32-byte Fernet key as raw bytes (post-base64-decode)."""
return base64.urlsafe_b64decode(Fernet.generate_key())
# ---------------------------------------------------------------------------
# Cipher round-trip
# ---------------------------------------------------------------------------
class TestCipherRoundTrip:
def test_round_trip_single_key(self) -> None:
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
plaintext = b"access_token_12345"
ct = cipher.encrypt(plaintext)
assert ct != plaintext
assert cipher.decrypt(ct) == plaintext
def test_round_trip_unicode_token(self) -> None:
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
# Tokens may legitimately carry UTF-8 bytes (e.g. JWT with
# non-ASCII claim values). Round-trip a multi-byte sequence.
plaintext = "tok_é中💯".encode()
ct = cipher.encrypt(plaintext)
assert cipher.decrypt(ct) == plaintext
def test_wrong_key_raises_decrypt_error(self) -> None:
cipher_a = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
cipher_b = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
ct = cipher_a.encrypt(b"secret")
with pytest.raises(MCPTokenDecryptError) as exc_info:
cipher_b.decrypt(ct)
# Audit-trail correlation: error must carry the fingerprints of
# the keys actually attempted, not a placeholder.
assert exc_info.value.key_fingerprints_attempted
assert exc_info.value.key_fingerprints_attempted == cipher_b.key_fingerprints
# ---------------------------------------------------------------------------
# Rotation (MultiFernet behavior)
# ---------------------------------------------------------------------------
class TestRotation:
def test_rotation_forward(self) -> None:
"""Encrypt with a new-only cipher, decrypt with a [v2, v1] cluster.
Mirrors the operational situation where a node already has the
rotated key list installed and a peer just wrote a row under v2.
"""
v1 = _new_raw_key()
v2 = _new_raw_key()
new_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2,)))
cluster = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
ct = new_only.encrypt(b"hello")
assert cluster.decrypt(ct) == b"hello"
def test_rotation_backward_keeps_old_decryptable(self) -> None:
"""A row written under the OLD key (v1) must still decrypt after
rotation places v2 first and keeps v1 as fallback."""
v1 = _new_raw_key()
v2 = _new_raw_key()
old_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1,)))
rotated = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
ct = old_only.encrypt(b"legacy")
assert rotated.decrypt(ct) == b"legacy"
# ---------------------------------------------------------------------------
# Config loader
# ---------------------------------------------------------------------------
def _patch_load_config(monkeypatch: pytest.MonkeyPatch, payload: dict) -> None:
"""Override ``turnstone.core.config.load_config`` to return ``payload``
when the ``"security"`` section is requested."""
def fake(section: str | None = None) -> dict:
if section == "security":
return payload
return {}
import turnstone.core.config as cfg_mod
monkeypatch.setattr(cfg_mod, "load_config", fake)
class TestLoadConfig:
def test_load_singular_key(self, monkeypatch: pytest.MonkeyPatch) -> None:
key = Fernet.generate_key().decode()
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": key})
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 1
def test_load_plural_overrides_singular(self, monkeypatch: pytest.MonkeyPatch) -> None:
plural = [Fernet.generate_key().decode(), Fernet.generate_key().decode()]
_patch_load_config(
monkeypatch,
{
"mcp_token_encryption_keys": plural,
"mcp_token_encryption_key": Fernet.generate_key().decode(),
},
)
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 2 # plural wins, singular ignored
def test_load_returns_none_when_absent(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {})
assert load_mcp_token_cipher_config() is None
def test_load_empty_plural_falls_through_to_singular(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Operator wrote ``mcp_token_encryption_keys = []`` AND set a
singular value: empty plural is treated as absent."""
key = Fernet.generate_key().decode()
_patch_load_config(
monkeypatch,
{"mcp_token_encryption_keys": [], "mcp_token_encryption_key": key},
)
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 1
def test_load_invalid_base64_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": "###not-base64###"})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
# Operator-facing hint is part of every error message.
assert "regenerate with:" in str(exc_info.value)
def test_load_wrong_length_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
# 24 raw bytes → 32 base64 chars; not 32 raw bytes after decode.
short_key = base64.urlsafe_b64encode(b"\x00" * 24).decode()
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": short_key})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
assert "32 bytes" in str(exc_info.value)
def test_load_non_list_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": "single-string-not-list"})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
assert "list" in str(exc_info.value).lower()
def test_load_non_string_in_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": [12345]})
with pytest.raises(MCPTokenKeyConfigError):
load_mcp_token_cipher_config()
# ---------------------------------------------------------------------------
# Fingerprint stability
# ---------------------------------------------------------------------------
class TestFingerprint:
def test_key_fingerprint_stable_and_short(self) -> None:
key = _new_raw_key()
fp1 = _key_fingerprint(key)
fp2 = _key_fingerprint(key)
assert fp1 == fp2
# 8 bytes -> 16 hex characters.
assert len(fp1) == 16
assert all(c in "0123456789abcdef" for c in fp1)
def test_different_keys_have_different_fingerprints(self) -> None:
fp1 = _key_fingerprint(_new_raw_key())
fp2 = _key_fingerprint(_new_raw_key())
assert fp1 != fp2
def test_cipher_fingerprints_match_keys(self) -> None:
v1 = _new_raw_key()
v2 = _new_raw_key()
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1, v2)))
assert cipher.key_fingerprints == (
_key_fingerprint(v1),
_key_fingerprint(v2),
)
# ---------------------------------------------------------------------------
# Direct ``_validate_key`` — exercises edge cases not reachable via loader
# ---------------------------------------------------------------------------
class TestValidateKey:
def test_empty_string_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
_validate_key("", label="x")
def test_whitespace_only_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
_validate_key(" ", label="x")
def test_label_propagated_in_error(self) -> None:
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
_validate_key("###", label="my_label_42")
assert "my_label_42" in str(exc_info.value)
# ---------------------------------------------------------------------------
# MCPTokenCipher constructor guard
# ---------------------------------------------------------------------------
class TestCipherConstructorGuard:
def test_empty_keys_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
MCPTokenCipher(MCPTokenCipherConfig(keys=()))
+27 -28
View File
@@ -4,7 +4,6 @@ from __future__ import annotations
from typing import Any
from tests.conftest import _seed_static_state
from turnstone.core.mcp_client import MCPClientManager
# ---------------------------------------------------------------------------
@@ -106,18 +105,14 @@ class TestRemoveServerSync:
"""remove_server_sync cleans up all per-server state dicts."""
mgr = MCPClientManager({"test": {"command": "echo"}})
# Simulate state as if the server was connected
_seed_static_state(
mgr,
"test",
tools=[_fake_openai_tool()],
resources=[_fake_resource_dict()],
prompts=[_fake_prompt_dict()],
supports_list_changed=True,
supports_resources=True,
supports_resource_list_changed=True,
supports_prompts=True,
supports_prompt_list_changed=True,
)
mgr._per_server_tools["test"] = [_fake_openai_tool()]
mgr._per_server_resources["test"] = [_fake_resource_dict()]
mgr._per_server_prompts["test"] = [_fake_prompt_dict()]
mgr._supports_list_changed["test"] = True
mgr._supports_resources["test"] = True
mgr._supports_resource_list_changed["test"] = True
mgr._supports_prompts["test"] = True
mgr._supports_prompt_list_changed["test"] = True
mgr._rebuild_tools()
mgr._rebuild_resources()
mgr._rebuild_prompts()
@@ -132,7 +127,14 @@ class TestRemoveServerSync:
assert len(mgr.get_tools()) == 0
assert mgr.resource_count == 0
assert mgr.prompt_count == 0
assert "test" not in mgr._static_servers
assert "test" not in mgr._per_server_tools
assert "test" not in mgr._per_server_resources
assert "test" not in mgr._per_server_prompts
assert "test" not in mgr._supports_list_changed
assert "test" not in mgr._supports_resources
assert "test" not in mgr._supports_resource_list_changed
assert "test" not in mgr._supports_prompts
assert "test" not in mgr._supports_prompt_list_changed
def test_removes_config_to_prevent_reconnect(self) -> None:
"""remove_server_sync removes from _server_configs to prevent reconnect."""
@@ -144,8 +146,8 @@ class TestRemoveServerSync:
def test_preserves_other_servers(self) -> None:
"""Removing one server does not affect another server's state."""
mgr = MCPClientManager({"srv_a": {}, "srv_b": {}})
_seed_static_state(mgr, "srv_a", tools=[_fake_openai_tool("mcp__srv_a__foo")])
_seed_static_state(mgr, "srv_b", tools=[_fake_openai_tool("mcp__srv_b__bar")])
mgr._per_server_tools["srv_a"] = [_fake_openai_tool("mcp__srv_a__foo")]
mgr._per_server_tools["srv_b"] = [_fake_openai_tool("mcp__srv_b__bar")]
mgr._rebuild_tools()
assert len(mgr.get_tools()) == 2
@@ -177,17 +179,13 @@ class TestGetServerStatus:
"""Status of a connected server reports correct tool/resource/prompt counts."""
mgr = MCPClientManager({"test": {}})
# Simulate connected state
_seed_static_state(
mgr,
"test",
session=object(), # any truthy value
tools=[
_fake_openai_tool("mcp__test__a"),
_fake_openai_tool("mcp__test__b"),
],
resources=[_fake_resource_dict()],
prompts=[_fake_prompt_dict()],
)
mgr._sessions["test"] = object() # any truthy value
mgr._per_server_tools["test"] = [
_fake_openai_tool("mcp__test__a"),
_fake_openai_tool("mcp__test__b"),
]
mgr._per_server_resources["test"] = [_fake_resource_dict()]
mgr._per_server_prompts["test"] = [_fake_prompt_dict()]
status = mgr.get_server_status("test")
assert status["connected"] is True
@@ -227,7 +225,8 @@ class TestGetAllServerStatus:
def test_mixed_connected_and_disconnected(self) -> None:
"""Status correctly reflects a mix of connected and disconnected servers."""
mgr = MCPClientManager({"up": {}, "down": {}})
_seed_static_state(mgr, "up", session=object(), tools=[_fake_openai_tool("mcp__up__x")])
mgr._sessions["up"] = object()
mgr._per_server_tools["up"] = [_fake_openai_tool("mcp__up__x")]
statuses = mgr.get_all_server_status()
assert statuses["up"]["connected"] is True
-242
View File
@@ -1,242 +0,0 @@
"""Unit tests for ``turnstone.core.mcp_http_parsers``.
The parser replaces the prior hand-rolled scanners that used
``header.lower().find("scope")`` to locate parameter names that approach
misparsed ``scope`` embedded inside other tokens (``xscope``) or inside
quoted-string values of preceding params. Each adversarial case below
asserts the new tokenizer respects RFC 7235 ``challenge auth-param``
boundaries; the docstrings document the equivalent input that broke the
naive parser. Negative-test verification: temporarily reverting
``parse_www_authenticate_scope`` to delegate to ``header.lower().find("scope")``
makes ``test_scope_inside_realm_value`` and ``test_scope_inside_xscope`` fail.
"""
from __future__ import annotations
import time
import pytest
from turnstone.core.mcp_http_parsers import (
parse_www_authenticate_bearer,
parse_www_authenticate_error,
parse_www_authenticate_scope,
)
class TestParseScope:
def test_basic_scope(self) -> None:
header = 'Bearer error="insufficient_scope", scope="files:read mail:send"'
assert parse_www_authenticate_scope(header) == ("files:read", "mail:send")
def test_no_scope_param(self) -> None:
assert parse_www_authenticate_scope('Bearer error="invalid_token"') == ()
def test_unterminated_quoted_string_returns_empty(self) -> None:
assert parse_www_authenticate_scope('Bearer scope="files:read') == ()
def test_escaped_chars_in_value_drops_invalid_scope_token(self) -> None:
# RFC 7230 §3.2.6 backslash escapes decode the literal scope to
# ``files:read "weird"``. RFC 6749 §3.3 ``scope-token`` forbids
# ``"``, so ``"weird"`` is dropped and only ``files:read``
# survives the post-split validation.
header = r'Bearer scope="files:read \"weird\""'
assert parse_www_authenticate_scope(header) == ("files:read",)
def test_empty_string(self) -> None:
assert parse_www_authenticate_scope("") == ()
def test_unquoted_scope_value(self) -> None:
# Unquoted single token.
assert parse_www_authenticate_scope("Bearer scope=files:read") == ("files:read",)
# --- the four headline misparse cases ---
def test_scope_inside_xscope(self) -> None:
"""``Bearer xscope="value"`` must NOT be read as ``scope``.
The naive ``find("scope")`` matched at position 7 inside
``xscope`` and returned ``("value",)``.
"""
assert parse_www_authenticate_scope('Bearer xscope="value"') == ()
def test_scope_inside_realm_value(self) -> None:
"""``Bearer realm="my scope=fake", scope="real"`` must return ``("real",)``.
The naive parser found ``scope=`` inside the quoted ``realm``
value first and returned ``("fake",)``.
"""
header = 'Bearer realm="my scope=fake", scope="real"'
assert parse_www_authenticate_scope(header) == ("real",)
def test_scope_inside_quoted_realm_with_escaped_quotes(self) -> None:
"""``Bearer realm="foo scope=\\"admin:write\\" bar"`` returns ``()``.
The inner ``scope=`` is wholly inside the quoted-string value of
``realm`` there is no top-level ``scope`` auth-param, so the
result is empty.
"""
header = r'Bearer realm="foo scope=\"admin:write\" bar"'
assert parse_www_authenticate_scope(header) == ()
def test_scope_token_validation_drops_control_bytes(self) -> None:
"""Tokens containing CR / LF / tab / DEL / quote are dropped.
RFC 6749 §3.3 restricts ``scope-token`` to visible ASCII
excluding ``"`` and ``\\``. The splitter applies that
validation so a malicious AS cannot smuggle CRLF (or the like)
through a future log / notification path that prints the scope
list verbatim. ``"a\\rb"`` and ``"\\nc"`` fail validation;
``"d"`` survives. The legitimate space separator splits ``d``
into its own token.
"""
# Build via concatenation so the assertion stays intelligible.
header = 'Bearer scope="a\rb \nc d"'
assert parse_www_authenticate_scope(header) == ("d",)
class TestParseError:
def test_basic_quoted_error(self) -> None:
assert (
parse_www_authenticate_error('Bearer error="insufficient_scope"')
== "insufficient_scope"
)
def test_other_quoted_error_tokens(self) -> None:
assert parse_www_authenticate_error('Bearer error="invalid_token"') == "invalid_token"
assert parse_www_authenticate_error('Bearer error="invalid_request"') == "invalid_request"
def test_no_error_param(self) -> None:
assert parse_www_authenticate_error("Bearer realm=foo") is None
def test_error_description_does_not_match_error(self) -> None:
"""``error_description`` is its own auth-param key, not ``error``.
The tokenizer reads ``_`` as part of the token (RFC 7230 ``tchar``),
so ``error_description`` becomes one key, ``error`` another.
"""
assert parse_www_authenticate_error('Bearer error_description="bad"') is None
def test_unquoted_error(self) -> None:
# Some ASes don't quote the error token.
assert (
parse_www_authenticate_error("Bearer error=insufficient_scope") == "insufficient_scope"
)
def test_empty_string(self) -> None:
assert parse_www_authenticate_error("") is None
def test_error_inside_realm_value(self) -> None:
"""``Bearer realm="my error=fake", error="real"`` must return ``"real"``.
Naive parser grabbed ``fake`` from inside the ``realm`` quoted
value.
"""
header = 'Bearer realm="my error=fake", error="real"'
assert parse_www_authenticate_error(header) == "real"
class TestBearerDict:
def test_returns_lowercased_keys(self) -> None:
header = 'Bearer Realm="x", Error="y", Scope="a b"'
params = parse_www_authenticate_bearer(header)
assert params == {"realm": "x", "error": "y", "scope": "a b"}
def test_non_bearer_scheme_returns_empty(self) -> None:
assert parse_www_authenticate_bearer('Basic realm="x"') == {}
def test_no_scheme(self) -> None:
assert parse_www_authenticate_bearer('realm="x"') == {}
def test_bearer_only_no_params(self) -> None:
assert parse_www_authenticate_bearer("Bearer ") == {}
def test_bearer_with_no_space_returns_empty(self) -> None:
# ``BearerToken`` is not a Bearer challenge (no separator).
assert parse_www_authenticate_bearer("BearerToken") == {}
def test_first_value_wins_on_duplicate(self) -> None:
# If a malformed AS sends two ``scope=`` params we keep the first.
# The earlier ``find()``-based scanner would have returned the
# last; either choice is legal for malformed input but we need
# to be consistent.
header = 'Bearer scope="first", scope="second"'
assert parse_www_authenticate_bearer(header) == {"scope": "first"}
def test_trailing_comma(self) -> None:
header = 'Bearer error="x",'
assert parse_www_authenticate_bearer(header) == {"error": "x"}
def test_multiple_commas(self) -> None:
header = 'Bearer ,, error="x",,, scope="y"'
assert parse_www_authenticate_bearer(header) == {"error": "x", "scope": "y"}
def test_embedded_escaped_quote(self) -> None:
header = r'Bearer realm="he said \"hi\""'
assert parse_www_authenticate_bearer(header) == {"realm": 'he said "hi"'}
def test_param_without_value_skipped(self) -> None:
header = 'Bearer realm, error="x"'
# ``realm`` without ``=`` is dropped; ``error`` survives.
assert parse_www_authenticate_bearer(header) == {"error": "x"}
@pytest.mark.parametrize(
"header,expected",
[
("", {}),
("Bearer", {}),
('Bearer realm=""', {"realm": ""}),
('Bearer realm="", scope=""', {"realm": "", "scope": ""}),
],
)
def test_edge_cases(self, header: str, expected: dict[str, str]) -> None:
assert parse_www_authenticate_bearer(header) == expected
class TestPathologicalInput:
def test_oversized_pathological_input_rejected_under_50ms(self) -> None:
"""Headers longer than the defensive cap return ``{}`` immediately.
The cap is set to 4096 bytes real ASes emit a few hundred bytes
at most. This guards both ``parse_www_authenticate_bearer``
callers against pathological input from a misbehaving server.
The previous ``header.lower().find("scope", i)`` loop was
O(N**2) a 100 KB header with no ``=`` took ~330 ms because
each ``find`` rescanned the entire suffix. The single-pass
tokenizer (capped at 4 KB) reduces this to a one-shot length
check that returns ``{}`` in microseconds, so the budget is
generous regardless of which side of the cap was hit.
"""
big = "Bearer scope=" + "a" * 10_000
start = time.perf_counter()
result = parse_www_authenticate_scope(big)
elapsed = time.perf_counter() - start
assert result == ()
assert elapsed < 0.05, f"oversized-header reject took {elapsed * 1000:.1f}ms"
def test_within_cap_long_header_under_50ms(self) -> None:
"""A 4 KB header with thousands of ``find`` candidates still parses fast.
Stays under the cap so the tokenizer actually runs end to end
the goal is to prove the inner loop is O(N), not just that the
cap rejects oversized input.
"""
# Pack the header right up to the cap with non-matching
# auth-params, then put the real ``scope`` at the end.
filler_parts = []
size = len("Bearer ")
i = 0
while size < 3900:
part = f'xscope{i}="ignore", '
if size + len(part) > 3900:
break
filler_parts.append(part)
size += len(part)
i += 1
header = "Bearer " + "".join(filler_parts) + 'scope="real"'
assert len(header) <= 4096
start = time.perf_counter()
result = parse_www_authenticate_scope(header)
elapsed = time.perf_counter() - start
assert result == ("real",)
assert elapsed < 0.05, f"4kb tokenize took {elapsed * 1000:.1f}ms"
+66 -81
View File
@@ -14,7 +14,6 @@ from unittest.mock import AsyncMock, MagicMock
import pytest
from tests.conftest import _seed_static_state
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.storage._sqlite import SQLiteBackend
@@ -110,15 +109,13 @@ class TestFullLifecycleResourcesPrompts:
def test_rebuild_resources_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_resources merges per-server resources into a unified list."""
_seed_static_state(
mgr,
"alpha",
resources=[
_make_resource("file:///a.txt", "a", "alpha"),
_make_resource("file:///b.txt", "b", "alpha"),
],
)
_seed_static_state(mgr, "beta", resources=[_make_resource("file:///c.txt", "c", "beta")])
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
_make_resource("file:///b.txt", "b", "alpha"),
]
mgr._per_server_resources["beta"] = [
_make_resource("file:///c.txt", "c", "beta"),
]
mgr._rebuild_resources()
@@ -133,19 +130,13 @@ class TestFullLifecycleResourcesPrompts:
def test_rebuild_prompts_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_prompts merges per-server prompts into a unified list."""
_seed_static_state(
mgr,
"alpha",
prompts=[_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello")],
)
_seed_static_state(
mgr,
"beta",
prompts=[
_make_prompt("mcp__beta__summarize", "summarize", "beta", "Summarize text"),
_make_prompt("mcp__beta__translate", "translate", "beta", "Translate text"),
],
)
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._per_server_prompts["beta"] = [
_make_prompt("mcp__beta__summarize", "summarize", "beta", "Summarize text"),
_make_prompt("mcp__beta__translate", "translate", "beta", "Translate text"),
]
mgr._rebuild_prompts()
@@ -173,12 +164,10 @@ class TestFullLifecycleResourcesPrompts:
try:
# Populate session and resource map
session = _make_mock_session()
_seed_static_state(
mgr,
"alpha",
session=session,
resources=[_make_resource("file:///readme.md", "readme", "alpha")],
)
mgr._sessions["alpha"] = session
mgr._per_server_resources["alpha"] = [
_make_resource("file:///readme.md", "readme", "alpha"),
]
mgr._rebuild_resources()
result = mgr.read_resource_sync("file:///readme.md", timeout=5)
@@ -205,22 +194,18 @@ class TestFullLifecycleResourcesPrompts:
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
# Register a template resource (no concrete resources)
_seed_static_state(
mgr,
"alpha",
session=session,
resources=[
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "Fetch a row",
"mimeType": "application/json",
"server": "alpha",
"template": True,
},
],
)
mgr._per_server_resources["alpha"] = [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "Fetch a row",
"mimeType": "application/json",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
# Template should not be in _resource_map
@@ -245,12 +230,10 @@ class TestFullLifecycleResourcesPrompts:
try:
session = _make_mock_session()
_seed_static_state(
mgr,
"alpha",
session=session,
prompts=[_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello")],
)
mgr._sessions["alpha"] = session
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._rebuild_prompts()
messages = mgr.get_prompt_sync(
@@ -331,42 +314,40 @@ class TestFullLifecycleResourcesPrompts:
def test_shutdown_clears_all_state(self, mgr: MCPClientManager) -> None:
"""shutdown() clears sessions, tools, resources, prompts, and listeners."""
# Populate state
_seed_static_state(
mgr,
"alpha",
session=MagicMock(),
tools=[
{
"type": "function",
"function": {
"name": "mcp__alpha__search",
"description": "Search",
"parameters": {},
},
}
],
resources=[
_make_resource("file:///a.txt", "a", "alpha"),
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "alpha",
"template": True,
mgr._sessions["alpha"] = MagicMock()
mgr._per_server_tools["alpha"] = [
{
"type": "function",
"function": {
"name": "mcp__alpha__search",
"description": "Search",
"parameters": {},
},
],
prompts=[_make_prompt("mcp__alpha__greet", "greet", "alpha")],
)
}
]
mgr._rebuild_tools()
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha"),
]
mgr._rebuild_prompts()
mgr._listeners.append(lambda: None)
mgr._resource_listeners.append(lambda: None)
mgr._prompt_listeners.append(lambda: None)
# Verify populated
assert len(mgr._static_servers) == 1
assert len(mgr._sessions) == 1
assert len(mgr._tools) == 1
assert len(mgr._resources) == 2 # 1 concrete + 1 template
assert len(mgr._template_prefixes) == 1
@@ -374,7 +355,7 @@ class TestFullLifecycleResourcesPrompts:
mgr.shutdown()
assert len(mgr._static_servers) == 0
assert len(mgr._sessions) == 0
assert len(mgr._tools) == 0
assert len(mgr._tool_map) == 0
assert len(mgr._resources) == 0
@@ -395,15 +376,19 @@ class TestFullLifecycleResourcesPrompts:
mgr.add_resource_listener(lambda: resource_fired.append(1))
mgr.add_prompt_listener(lambda: prompt_fired.append(1))
_seed_static_state(mgr, "alpha", tools=[])
mgr._per_server_tools["alpha"] = []
mgr._rebuild_tools()
assert len(tool_fired) == 1
_seed_static_state(mgr, "alpha", resources=[_make_resource("file:///x.txt", "x", "alpha")])
mgr._per_server_resources["alpha"] = [
_make_resource("file:///x.txt", "x", "alpha"),
]
mgr._rebuild_resources()
assert len(resource_fired) == 1
_seed_static_state(mgr, "alpha", prompts=[_make_prompt("mcp__alpha__p1", "p1", "alpha")])
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__p1", "p1", "alpha"),
]
mgr._rebuild_prompts()
assert len(prompt_fired) == 1
-780
View File
@@ -1,780 +0,0 @@
"""Integration tests for the MCP OAuth ``/connections`` endpoints.
Covers the list and revoke handlers that surface user-owned MCP server
consents to the settings UI:
* ``GET /v1/api/mcp/oauth/connections`` non-secret projection only.
* ``DELETE /v1/api/mcp/oauth/connections/{server_name}`` best-effort
upstream revoke (RFC 7009) followed by the authoritative local
delete; cross-user attempts return 404 with the exact same body
shape as a never-existed row to avoid leaking tenant existence.
"""
from __future__ import annotations
import asyncio
from typing import TYPE_CHECKING, Any
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from tests.conftest import make_mcp_token_cipher
from turnstone.core.auth import AuthResult
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import (
handle_mcp_oauth_list_connections,
handle_mcp_oauth_revoke_connection,
)
from turnstone.core.oidc import OIDCConfig
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
# ---------------------------------------------------------------------------
# Fixtures + helpers (mirror tests/test_mcp_oauth_handlers.py)
# ---------------------------------------------------------------------------
class _InjectAuthMiddleware(BaseHTTPMiddleware):
"""Stamp a fixed authenticated user on every request."""
def __init__(self, app: Any, user_id: str = "user-1") -> None:
super().__init__(app)
self._user_id = user_id
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id=self._user_id,
scopes=frozenset({"write"}),
token_source="config",
permissions=frozenset({"read", "write"}),
)
return await call_next(request)
class _NoAuthMiddleware(BaseHTTPMiddleware):
"""Leave ``request.state.auth_result`` unset so handlers see anon."""
async def dispatch(self, request: Request, call_next: Any) -> Response:
return await call_next(request)
async def _list_handler(request: Request) -> Response:
return await handle_mcp_oauth_list_connections(request)
async def _revoke_handler(request: Request) -> Response:
return await handle_mcp_oauth_revoke_connection(request)
def _build_app(
*,
storage: SQLiteBackend,
http_client: httpx.AsyncClient | MagicMock,
token_store: MCPTokenStore | None,
user_id: str = "user-1",
mcp_client: Any = None,
authenticated: bool = True,
) -> Starlette:
middleware: list[Middleware]
if authenticated:
middleware = [Middleware(_InjectAuthMiddleware, user_id=user_id)]
else:
middleware = [Middleware(_NoAuthMiddleware)]
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/mcp/oauth/connections", _list_handler),
Route(
"/api/mcp/oauth/connections/{server_name}",
_revoke_handler,
methods=["DELETE"],
),
],
),
],
middleware=middleware,
)
app.state.auth_storage = storage
app.state.mcp_token_store = token_store
app.state.mcp_oauth_http_client = http_client
app.state.mcp_oauth_refresh_locks = {}
app.state.mcp_oauth_dcr_locks = {}
app.state.mcp_oauth_metadata_cache = {}
app.state.mcp_oauth_last_cleanup_monotonic = 0.0
app.state.oidc_config = OIDCConfig(enabled=False, redirect_base="https://testserver")
if mcp_client is not None:
app.state.mcp_client = mcp_client
return app
def _make_token_store(backend: SQLiteBackend) -> MCPTokenStore:
return MCPTokenStore(backend, make_mcp_token_cipher(), node_id="test")
def _seed_oauth_user_server(
backend: SQLiteBackend,
*,
name: str = "srv-oauth",
server_id: str = "srv-id-1",
cached_issuer: str | None = "https://as.example.com",
) -> str:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid profile",
oauth_audience="https://mcp.example.com",
oauth_authorization_server_url=None,
)
if cached_issuer is not None:
backend.update_mcp_server(server_id, oauth_as_issuer_cached=cached_issuer)
return server_id
def _seed_user_token(
token_store: MCPTokenStore,
*,
user_id: str = "user-1",
server_name: str = "srv-oauth",
refresh_token: str | None = "refresh-secret",
) -> None:
token_store.create_user_token(
user_id,
server_name,
access_token="access-secret",
refresh_token=refresh_token,
expires_at="2099-12-31T00:00:00",
scopes="openid profile",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _good_as_metadata_doc(
*, revocation_endpoint: str | None = "https://as.example.com/revoke"
) -> dict[str, Any]:
doc: dict[str, Any] = {
"issuer": "https://as.example.com",
"authorization_endpoint": "https://as.example.com/authorize",
"token_endpoint": "https://as.example.com/token",
"registration_endpoint": "https://as.example.com/register",
"jwks_uri": "https://as.example.com/jwks",
"code_challenge_methods_supported": ["S256"],
"token_endpoint_auth_methods_supported": ["none", "client_secret_basic"],
}
if revocation_endpoint is not None:
doc["revocation_endpoint"] = revocation_endpoint
return doc
def _mk_response(
status_code: int = 200,
json_body: Any = None,
headers: dict[str, str] | None = None,
) -> MagicMock:
import json as _json
resp = MagicMock(spec=httpx.Response)
resp.status_code = status_code
resp.headers = headers or {}
body_str = _json.dumps(json_body) if json_body is not None else ""
resp.content = body_str.encode("utf-8")
if json_body is not None:
resp.json.return_value = json_body
else:
resp.json.side_effect = ValueError("no body")
resp.text = body_str
return resp
def _public_addr_patch():
return patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("93.184.216.34", 0))])
def _drain_revoke_upstream_tasks(client: TestClient, timeout: float = 2.0) -> None:
"""Block until all in-flight upstream-revoke tasks complete.
Phase 8 perf-1 made the RFC 7009 AS round-trip a fire-and-forget
task so the user-visible 204 isn't gated on the AS. The tasks were
scheduled on the TestClient's portal loop; we re-enter that loop
via :attr:`TestClient.portal` to await them. Tests that assert
against the upstream POST must call this helper before the
assertion.
"""
from turnstone.core.mcp_oauth import _revoke_upstream_tasks
portal = getattr(client, "portal", None)
if portal is None:
return
async def _drain() -> None:
pending = list(_revoke_upstream_tasks)
if pending:
async with asyncio.timeout(timeout):
await asyncio.gather(*pending, return_exceptions=True)
portal.call(_drain)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
backend = SQLiteBackend(str(tmp_path / "test.db"))
backend.create_user("user-1", "user1", "User One", "hash")
backend.create_user("user-2", "user2", "User Two", "hash")
return backend
@pytest.fixture
def http_client_mock() -> MagicMock:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock()
client.post = AsyncMock()
return client
# ---------------------------------------------------------------------------
# GET /connections
# ---------------------------------------------------------------------------
class TestListConnections:
def test_list_connections_unauthenticated_401(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
token_store = _make_token_store(storage)
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
authenticated=False,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 401
assert resp.json() == {"error": "Authentication required"}
def test_list_connections_no_token_store_503(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
app = _build_app(storage=storage, http_client=http_client_mock, token_store=None)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 503
def test_list_connections_empty_user_returns_empty_list(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
token_store = _make_token_store(storage)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 200
assert resp.json() == {"connections": []}
def test_list_connections_returns_users_consents(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage, name="srv-a", server_id="srv-id-a")
_seed_oauth_user_server(storage, name="srv-b", server_id="srv-id-b")
token_store = _make_token_store(storage)
_seed_user_token(token_store, server_name="srv-a")
_seed_user_token(token_store, server_name="srv-b")
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 200
body = resp.json()
assert "connections" in body
servers = sorted(row["server_name"] for row in body["connections"])
assert servers == ["srv-a", "srv-b"]
def test_list_connections_isolates_by_user(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, user_id="user-1", server_name="srv-oauth")
_seed_user_token(token_store, user_id="user-2", server_name="srv-oauth")
# User-1 sees only user-1's row.
app = _build_app(
storage=storage, http_client=http_client_mock, token_store=token_store, user_id="user-1"
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
rows = resp.json()["connections"]
assert all(row["user_id"] == "user-1" for row in rows)
assert len(rows) == 1
# User-2 sees only user-2's row.
app2 = _build_app(
storage=storage, http_client=http_client_mock, token_store=token_store, user_id="user-2"
)
client2 = TestClient(app2, raise_server_exceptions=False)
resp2 = client2.get("/v1/api/mcp/oauth/connections")
rows2 = resp2.json()["connections"]
assert all(row["user_id"] == "user-2" for row in rows2)
assert len(rows2) == 1
def test_list_connections_does_not_leak_secret_fields(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
rows = resp.json()["connections"]
assert rows
for row in rows:
for forbidden in (
"access_token",
"refresh_token",
"access_token_ct",
"refresh_token_ct",
):
assert forbidden not in row, f"secret field {forbidden!r} leaked in {row!r}"
# ---------------------------------------------------------------------------
# DELETE /connections/{server_name}
# ---------------------------------------------------------------------------
class TestRevokeConnection:
def test_revoke_connection_unauthenticated_401(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store)
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
authenticated=False,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 401
def test_revoke_connection_missing_row_404(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
token_store = _make_token_store(storage)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-nonexistent")
assert resp.status_code == 404
assert resp.json() == {"error": "No such connection"}
def test_revoke_connection_local_delete_succeeds_204(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
# No refresh token → upstream revoke is skipped entirely.
_seed_user_token(token_store, refresh_token=None)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row is gone.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# Upstream not contacted.
http_client_mock.post.assert_not_called()
def test_revoke_connection_with_revocation_endpoint_calls_upstream(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
http_client_mock.post.return_value = _mk_response(200)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
# ``with TestClient(...)`` keeps a persistent portal so the
# fire-and-forget upstream-revoke task isn't cancelled when
# the request handler returns. See ``_drain_revoke_upstream_tasks``.
# The SSRF-validator's ``socket.getaddrinfo`` patch must wrap
# the drain too — the discovery call now runs on the background
# task and resolves the AS hostname after the request returns.
with (
TestClient(app, raise_server_exceptions=False) as client,
_public_addr_patch(),
):
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row is gone.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# The upstream RFC 7009 POST is fire-and-forget post-Phase-8 perf-1
# so the test must drain the in-flight task set before asserting.
_drain_revoke_upstream_tasks(client)
# Upstream POSTed to revocation_endpoint with refresh-token grant.
assert http_client_mock.post.await_count == 1
call = http_client_mock.post.await_args
assert call.args[0] == "https://as.example.com/revoke"
data = call.kwargs.get("data") or {}
assert data.get("token") == "refresh-secret"
assert data.get("token_type_hint") == "refresh_token"
assert data.get("client_id") == "client-abc"
def test_revoke_connection_without_revocation_endpoint_skips_upstream(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(
200, _good_as_metadata_doc(revocation_endpoint=None)
)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
# ``with TestClient(...)`` keeps the portal alive for the
# background task drain.
with (
TestClient(app, raise_server_exceptions=False) as client,
_public_addr_patch(),
):
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row gone, upstream POST never made.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# Drain the fire-and-forget discovery task before asserting on
# the AS POST — the task runs ``discover_authorization_server``
# but does NOT proceed to POST because revocation_endpoint is
# absent.
_drain_revoke_upstream_tasks(client)
http_client_mock.post.assert_not_called()
def test_revoke_connection_upstream_failure_still_204(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
# AS returns 500 — local delete must still succeed.
http_client_mock.post.return_value = _mk_response(500)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
with _public_addr_patch():
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
assert token_store.get_user_token("user-1", "srv-oauth") is None
def test_revoke_connection_audit_event_emitted_with_user_revoked_reason(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token=None)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Audit row was written via the storage API (tests don't poke at
# the SQLite schema directly — the table name is an internal
# detail).
events = storage.list_audit_events(action="mcp_server.oauth.token_revoked")
assert len(events) == 1
ev = events[0]
assert ev["user_id"] == "user-1"
# resource_id is the immutable server_id PK, not the name.
assert ev["resource_id"] == "srv-id-1"
import json as _json
detail = _json.loads(ev["detail"]) if isinstance(ev["detail"], str) else ev["detail"]
assert detail["reason"] == "user_revoked"
assert detail["upstream_revoke_outcome"] == "no_refresh_token"
assert detail["server_name"] == "srv-oauth"
def test_revoke_connection_cross_user_attempt_404(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
# Owned by user-2, not user-1.
_seed_user_token(token_store, user_id="user-2", server_name="srv-oauth")
app = _build_app(
storage=storage, http_client=http_client_mock, token_store=token_store, user_id="user-1"
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
# Cross-user attempt MUST surface as a generic 404, byte-identical
# body to the never-existed case (no tenant existence leak).
assert resp.status_code == 404
assert resp.json() == {"error": "No such connection"}
# Drain pending tasks defensively, then confirm the upstream
# endpoint was NEVER contacted on the 404-cross-user path. A
# bug that scheduled the AS round-trip before the cross-user
# check would leak existence via the AS-side 200/4xx response.
_drain_revoke_upstream_tasks(client)
http_client_mock.post.assert_not_called()
# User-2's row is untouched.
assert token_store.get_user_token("user-2", "srv-oauth") is not None
def test_revoke_connection_evicts_pool_session(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token=None)
mcp_client_mock = MagicMock()
# ``evict_user_session`` is the public sync surface on
# MCPClientManager; mirror its signature here so the handler's
# ``hasattr`` gate triggers.
mcp_client_mock.evict_user_session = MagicMock(return_value=None)
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
mcp_client=mcp_client_mock,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
mcp_client_mock.evict_user_session.assert_called_once_with("user-1", "srv-oauth")
def test_revoke_connection_pool_eviction_failure_does_not_block_204(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token=None)
mcp_client_mock = MagicMock()
mcp_client_mock.evict_user_session = MagicMock(side_effect=RuntimeError("loop closed"))
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
mcp_client=mcp_client_mock,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local delete still happened.
assert token_store.get_user_token("user-1", "srv-oauth") is None
def test_revoke_connection_204_not_gated_on_slow_upstream(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
"""The user-visible 204 must return promptly even when the
upstream AS round-trip is slow / hanging. Pre-perf-1 the
handler awaited ``revoke_token_at_as`` synchronously, so a
stuck AS could block the user's revoke confirmation. The
fire-and-forget refactor moves the call onto a background task
so the 204 returns in well under 1s regardless of AS latency.
Bound is conservative for CI runner jitter.
"""
import time
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
async def _slow_post(*_args: Any, **_kwargs: Any) -> Any:
# Simulate a slow / unreachable AS — must NOT gate the
# user-visible 204 on this round-trip.
await asyncio.sleep(5.0)
return _mk_response(200)
http_client_mock.post = AsyncMock(side_effect=_slow_post)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
with _public_addr_patch():
start = time.monotonic()
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
elapsed = time.monotonic() - start
assert resp.status_code == 204
# 1s ceiling — the 204 must return on the local-delete path
# without waiting on the AS POST (which sleeps 5s above). Bound
# is intentionally generous for CI runner jitter; the actual
# path is on the order of milliseconds.
assert elapsed < 1.0, (
f"204 returned in {elapsed:.3f}s — should be <1s; the "
"fire-and-forget upstream revoke isn't decoupled from the "
"response."
)
# The local row IS gone — the authoritative delete ran before
# the 204 returned, even though the AS round-trip is still
# in flight.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# Cancel any in-flight tasks so the test client can exit cleanly.
from turnstone.core.mcp_oauth import _revoke_upstream_tasks
portal = getattr(client, "portal", None)
if portal is not None:
for task in list(_revoke_upstream_tasks):
portal.call(task.cancel)
def test_revoke_connection_sheds_upstream_when_task_set_full(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
"""Round-2 q-2 regression: the soft cap on ``_revoke_upstream_tasks``
is the only protection against unbounded background-task pile-up
under a coordinated mass-revoke. When the set is full, the local
delete still runs but no upstream task is scheduled; the audit
detail records ``upstream_revoke_outcome="shed_by_cap"`` and
the AS endpoint is never contacted.
"""
from turnstone.core.mcp_oauth import (
_REVOKE_UPSTREAM_TASKS_MAX,
_revoke_upstream_tasks,
)
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
sentinel_event_holder: dict[str, asyncio.Event] = {}
# Use ``with TestClient(...)`` so the portal stays alive — we
# need to schedule sentinel tasks on the portal's loop and the
# tasks must outlive the request to actually fill the set.
with (
TestClient(app, raise_server_exceptions=False) as client,
_public_addr_patch(),
):
portal = client.portal
assert portal is not None
async def _create_sentinel_event() -> asyncio.Event:
event = asyncio.Event()
sentinel_event_holder["event"] = event
return event
sentinel_event = portal.call(_create_sentinel_event)
async def _wait_on_event() -> None:
await sentinel_event.wait()
async def _fill_task_set() -> list[asyncio.Task[None]]:
tasks: list[asyncio.Task[None]] = []
for _ in range(_REVOKE_UPSTREAM_TASKS_MAX):
t = asyncio.create_task(_wait_on_event())
_revoke_upstream_tasks.add(t)
tasks.append(t)
return tasks
sentinels = portal.call(_fill_task_set)
assert len(_revoke_upstream_tasks) >= _REVOKE_UPSTREAM_TASKS_MAX
try:
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row is still gone — authoritative delete ran.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# AS endpoint MUST NOT have been contacted.
http_client_mock.post.assert_not_called()
# Audit detail records the categorical shed outcome.
events = storage.list_audit_events(action="mcp_server.oauth.token_revoked")
assert len(events) == 1
detail = events[0]["detail"]
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail["upstream_revoke_outcome"] == "shed_by_cap"
finally:
# Release sentinels so the portal can shut down cleanly.
async def _release() -> None:
sentinel_event.set()
for t in sentinels:
t.cancel()
await asyncio.gather(*sentinels, return_exceptions=True)
portal.call(_release)
# ---------------------------------------------------------------------------
# evict_user_session helper sanity checks
# ---------------------------------------------------------------------------
class TestEvictUserSession:
def test_evict_user_session_no_loop_is_silent_noop(self) -> None:
from turnstone.core.mcp_client import MCPClientManager
mgr = MCPClientManager.__new__(MCPClientManager)
mgr._loop = None # type: ignore[attr-defined]
# Must not raise.
mgr.evict_user_session("user-1", "srv-oauth")
def test_evict_user_session_dispatches_to_loop(self) -> None:
from turnstone.core.mcp_client import MCPClientManager
mgr = MCPClientManager.__new__(MCPClientManager)
loop = asyncio.new_event_loop()
try:
mgr._loop = loop # type: ignore[attr-defined]
mgr._user_pool_entries = {} # type: ignore[attr-defined]
mgr._last_pool_notification_refresh = {} # type: ignore[attr-defined]
evicted: list[tuple[str, str]] = []
def _fake_evict(key: tuple[str, str]) -> None:
evicted.append(key)
mgr._evict_session = _fake_evict # type: ignore[method-assign]
# Run the dispatch on a separate thread so the loop can drain.
import threading
done = threading.Event()
def _run_loop() -> None:
loop.call_later(0.05, loop.stop)
loop.run_forever()
done.set()
t = threading.Thread(target=_run_loop, daemon=True)
t.start()
mgr.evict_user_session("user-1", "srv-oauth")
done.wait(timeout=1.0)
assert evicted == [("user-1", "srv-oauth")]
finally:
if not loop.is_closed():
loop.close()
-626
View File
@@ -1,626 +0,0 @@
"""Discovery tests for the per-(user, server) MCP OAuth flow.
Covers PRM (RFC 9728) and AS metadata (RFC 8414) discovery, including:
- override URL takes precedence
- PRM happy path: server URL -> .well-known/oauth-protected-resource
-> ``authorization_servers[0]``
- PRM 401 + ``WWW-Authenticate: Bearer resource_metadata="..."`` follows
the URL.
- AS metadata without S256 -> :class:`MCPOAuthDiscoveryError`.
- SSRF rejection on AS issuer URL.
- In-memory cache hit/miss + persistent cache write to
``mcp_servers.oauth_as_issuer_cached``.
"""
from __future__ import annotations
import asyncio
import time
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
from turnstone.core.mcp_oauth import (
ASMetadata,
MCPOAuthDiscoveryError,
_parse_prm_url_from_www_authenticate,
discover_authorization_server,
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _mk_response(
status_code: int = 200,
json_body: Any = None,
headers: dict[str, str] | None = None,
) -> MagicMock:
"""Build a MagicMock that quacks like ``httpx.Response``."""
resp = MagicMock(spec=httpx.Response)
resp.status_code = status_code
resp.headers = headers or {}
resp.content = (str(json_body) if json_body is not None else "").encode("utf-8")
if json_body is not None:
resp.json.return_value = json_body
else:
resp.json.side_effect = ValueError("no body")
resp.text = str(json_body) if json_body is not None else ""
return resp
def _good_as_metadata_doc() -> dict[str, Any]:
return {
"issuer": "https://as.example.com",
"authorization_endpoint": "https://as.example.com/authorize",
"token_endpoint": "https://as.example.com/token",
"jwks_uri": "https://as.example.com/jwks",
"code_challenge_methods_supported": ["S256"],
"token_endpoint_auth_methods_supported": ["none", "client_secret_basic"],
"registration_endpoint": "https://as.example.com/register",
}
def _public_addr_patch():
return patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("93.184.216.34", 0))])
def _mk_storage_mock(server_id: str = "srv-id") -> MagicMock:
storage = MagicMock()
storage.update_mcp_server.return_value = True
return storage
# ---------------------------------------------------------------------------
# PRM parser
# ---------------------------------------------------------------------------
class TestParsePRMUrl:
def test_extracts_resource_metadata_url(self) -> None:
header = (
'Bearer error="invalid_token", '
'resource_metadata="https://srv.example.com/.well-known/oauth-protected-resource"'
)
url = _parse_prm_url_from_www_authenticate(header)
assert url == "https://srv.example.com/.well-known/oauth-protected-resource"
def test_returns_none_when_absent(self) -> None:
assert _parse_prm_url_from_www_authenticate('Bearer realm="x"') is None
def test_handles_empty_header(self) -> None:
assert _parse_prm_url_from_www_authenticate("") is None
def test_handles_escaped_quote_in_value(self) -> None:
"""RFC 7230 quoted-string allows ``\\"`` — naive ``[^"]+`` truncates.
A malicious or buggy resource server could send an embedded
escaped quote; the parser must yield the unescaped value, not
the prefix up to the escaped quote.
"""
header = 'Bearer resource_metadata="https://srv.example.com/with\\"quote"'
url = _parse_prm_url_from_www_authenticate(header)
assert url == 'https://srv.example.com/with"quote'
def test_handles_escaped_backslash(self) -> None:
header = 'Bearer resource_metadata="https://srv.example.com/back\\\\slash"'
url = _parse_prm_url_from_www_authenticate(header)
assert url == "https://srv.example.com/back\\slash"
def test_unterminated_quoted_string_returns_none(self) -> None:
# Closing quote missing — naive regex would still match, but
# the proper parser should reject malformed input.
header = 'Bearer resource_metadata="https://srv.example.com/no-close'
assert _parse_prm_url_from_www_authenticate(header) is None
# ---------------------------------------------------------------------------
# discover_authorization_server happy paths
# ---------------------------------------------------------------------------
class TestDiscoveryOverride:
def test_override_url_skips_prm(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert isinstance(meta, ASMetadata)
assert meta.token_endpoint == "https://as.example.com/token"
# Only the AS metadata URL was hit, not PRM.
called_urls = [c.args[0] for c in client.get.call_args_list]
assert all("oauth-authorization-server" in u for u in called_urls)
class TestDiscoveryPRM:
def test_prm_happy_path(self) -> None:
async def _get(url, *args, **kwargs):
if url.endswith("/oauth-protected-resource"):
return _mk_response(
200,
{
"resource": "https://mcp.example.com",
"authorization_servers": ["https://as.example.com"],
},
)
if url.endswith("/oauth-authorization-server"):
return _mk_response(200, _good_as_metadata_doc())
raise AssertionError(f"unexpected URL: {url}")
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.issuer == "https://as.example.com"
def test_prm_401_follows_www_authenticate(self) -> None:
async def _get(url, *args, **kwargs):
if url == "https://mcp.example.com/.well-known/oauth-protected-resource":
return _mk_response(
401,
headers={
"www-authenticate": (
'Bearer error="invalid_token", '
"resource_metadata="
'"https://meta.example.com/prm"'
)
},
json_body=None,
)
if url == "https://meta.example.com/prm":
return _mk_response(
200,
{
"authorization_servers": ["https://as.example.com"],
},
)
if url.endswith("/oauth-authorization-server"):
return _mk_response(200, _good_as_metadata_doc())
raise AssertionError(f"unexpected URL: {url}")
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.token_endpoint == "https://as.example.com/token"
def test_prm_401_without_resource_metadata_raises(self) -> None:
async def _get(url, *args, **kwargs):
return _mk_response(401, headers={"www-authenticate": "Basic realm=x"})
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="resource_metadata"):
asyncio.run(_run())
# ---------------------------------------------------------------------------
# AS metadata validation
# ---------------------------------------------------------------------------
class TestASMetadataValidation:
def test_no_s256_raises(self) -> None:
doc = _good_as_metadata_doc()
doc["code_challenge_methods_supported"] = ["plain"]
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="S256"):
asyncio.run(_run())
def test_missing_endpoints_raises(self) -> None:
doc = _good_as_metadata_doc()
del doc["token_endpoint"]
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="missing required"):
asyncio.run(_run())
def test_third_party_endpoint_rejected(self) -> None:
doc = _good_as_metadata_doc()
doc["token_endpoint"] = "https://attacker.example.com/token"
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="token_endpoint"):
asyncio.run(_run())
def test_ssrf_on_override_rejected(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock()
storage = _mk_storage_mock()
async def _run():
# Resolve to private 10.x — SSRF guard fires before any HTTP call.
with patch(
"socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("10.0.0.1", 0))],
):
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://internal.corp.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError):
asyncio.run(_run())
client.get.assert_not_called()
# ---------------------------------------------------------------------------
# Caching
# ---------------------------------------------------------------------------
class TestMetadataCache:
def test_cache_miss_then_hit(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
cache: dict[str, tuple[ASMetadata, float]] = {}
async def _run():
with _public_addr_patch():
first = await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
metadata_cache=cache,
)
second = await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer="https://as.example.com",
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
metadata_cache=cache,
)
return first, second
first, second = asyncio.run(_run())
assert first.token_endpoint == second.token_endpoint
# First call hit AS metadata; second call hit the cache.
assert client.get.call_count == 1
def test_cache_expiry_refetches(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
# Pre-populate cache with a very stale entry.
stale_meta = ASMetadata(
issuer="https://as.example.com",
authorization_endpoint="https://as.example.com/authorize",
token_endpoint="https://as.example.com/token",
registration_endpoint=None,
revocation_endpoint=None,
jwks_uri=None,
code_challenge_methods_supported=("S256",),
token_endpoint_auth_methods_supported=(),
)
cache = {"https://as.example.com": (stale_meta, time.monotonic() - 10**6)}
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
metadata_cache=cache,
)
meta = asyncio.run(_run())
# Stale entry was bypassed -> we hit the network.
assert client.get.call_count == 1
assert meta.token_endpoint == "https://as.example.com/token"
def test_persistent_cache_write_on_first_resolution(self) -> None:
async def _get(url, *args, **kwargs):
if url.endswith("/oauth-protected-resource"):
return _mk_response(200, {"authorization_servers": ["https://as.example.com"]})
return _mk_response(200, _good_as_metadata_doc())
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
asyncio.run(_run())
# update_mcp_server was called once with the cached issuer.
storage.update_mcp_server.assert_called_once_with(
"srv-id", oauth_as_issuer_cached="https://as.example.com"
)
def test_persistent_cache_skip_when_already_cached(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer="https://as.example.com",
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
asyncio.run(_run())
storage.update_mcp_server.assert_not_called()
# ---------------------------------------------------------------------------
# sec-3 — cached_issuer re-validated on read
# ---------------------------------------------------------------------------
class TestCachedIssuerSSRFRevalidation:
"""A cached issuer URL must still pass SSRF validation on every read.
Defense-in-depth: an admin who points ``oauth_as_issuer_cached`` at a
private address (or a hostname that has rebound to one) should not
bypass the guard just because the value was already in the row.
"""
def test_cached_issuer_rejected_clears_row_and_falls_through_to_prm(self) -> None:
async def _get(url: str, *args: Any, **kwargs: Any) -> MagicMock:
if url.endswith("/oauth-protected-resource"):
return _mk_response(200, {"authorization_servers": ["https://as.example.com"]})
if url.endswith("/oauth-authorization-server"):
return _mk_response(200, _good_as_metadata_doc())
raise AssertionError(f"unexpected URL: {url}")
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
# cached_issuer points at a private host. SSRF guard fires on
# the cached value first, the row is cleared, and PRM
# discovery runs as a fallback.
async def _run() -> Any:
with patch(
"socket.getaddrinfo",
# Private resolution for "internal.corp", public for everything else.
side_effect=lambda host, *a, **kw: [
(2, 1, 6, "", ("10.0.0.1" if "internal" in host else "93.184.216.34", 0))
],
):
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer="https://internal.corp.example.com",
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.token_endpoint == "https://as.example.com/token"
# The bad cached_issuer was cleared from the row.
clear_calls = [
c
for c in storage.update_mcp_server.call_args_list
if c.kwargs.get("oauth_as_issuer_cached") is None
]
assert clear_calls, "cached_issuer should have been cleared"
# ---------------------------------------------------------------------------
# revocation_endpoint parsing (RFC 8414)
# ---------------------------------------------------------------------------
class TestASMetadataRevocationEndpoint:
def test_as_metadata_parses_revocation_endpoint(self) -> None:
doc = _good_as_metadata_doc()
doc["revocation_endpoint"] = "https://as.example.com/revoke"
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run() -> ASMetadata:
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.revocation_endpoint == "https://as.example.com/revoke"
def test_as_metadata_revocation_endpoint_absent(self) -> None:
doc = _good_as_metadata_doc()
doc.pop("revocation_endpoint", None)
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run() -> ASMetadata:
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.revocation_endpoint is None
def test_as_metadata_revocation_endpoint_rejected_when_cross_origin(self) -> None:
doc = _good_as_metadata_doc()
doc["revocation_endpoint"] = "https://attacker.example.com/revoke"
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run() -> ASMetadata:
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="revocation_endpoint"):
asyncio.run(_run())
File diff suppressed because it is too large Load Diff
-133
View File
@@ -1,133 +0,0 @@
"""Smoke tests for the new OAuth-MCP storage tables.
Phase 2 only adds the schema token CRUD lands in Phase 3 and pending-
state CRUD in Phase 4. These tests verify the tables exist after
``init_storage`` and accept the documented row shape via raw SQL.
"""
from __future__ import annotations
import sqlalchemy as sa
from turnstone.core.storage._schema import mcp_oauth_pending, mcp_user_tokens
class TestMcpUserTokensTable:
def test_table_exists_and_accepts_row(self, backend) -> None:
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_user_tokens),
{
"user_id": "u1",
"server_name": "srv-a",
"access_token_ct": b"\x00ciphertext-a",
"refresh_token_ct": b"\x00ciphertext-r",
"expires_at": "2026-05-04T12:00:00",
"scopes": "openid profile",
"as_issuer": "https://auth.example.com",
"audience": "https://mcp.example.com",
"created": "2026-05-04T11:00:00",
"last_refreshed": None,
},
)
conn.commit()
row = conn.execute(
sa.select(mcp_user_tokens).where(
(mcp_user_tokens.c.user_id == "u1") & (mcp_user_tokens.c.server_name == "srv-a")
)
).one()
assert row.access_token_ct == b"\x00ciphertext-a"
assert row.refresh_token_ct == b"\x00ciphertext-r"
assert row.scopes == "openid profile"
assert row.audience == "https://mcp.example.com"
def test_composite_pk_distinguishes_user_server(self, backend) -> None:
"""Same user, different server => two rows; same (user, server) => conflict."""
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_user_tokens),
[
{
"user_id": "u1",
"server_name": "srv-a",
"access_token_ct": b"a",
"refresh_token_ct": None,
"expires_at": None,
"scopes": None,
"as_issuer": "https://auth.example.com",
"audience": "https://a.example.com",
"created": "2026-05-04T11:00:00",
"last_refreshed": None,
},
{
"user_id": "u1",
"server_name": "srv-b",
"access_token_ct": b"b",
"refresh_token_ct": None,
"expires_at": None,
"scopes": None,
"as_issuer": "https://auth.example.com",
"audience": "https://b.example.com",
"created": "2026-05-04T11:00:00",
"last_refreshed": None,
},
],
)
conn.commit()
count = conn.execute(sa.select(sa.func.count()).select_from(mcp_user_tokens)).scalar()
assert count == 2
class TestMcpOauthPendingTable:
def test_table_exists_and_accepts_row(self, backend) -> None:
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_oauth_pending),
{
"state": "rand-state-xyz",
"user_id": "u1",
"server_name": "srv-a",
"code_verifier": "verifier-blob",
"return_url": "/admin/mcp-servers",
"created_at": "2026-05-04T11:00:00",
},
)
conn.commit()
row = conn.execute(
sa.select(mcp_oauth_pending).where(mcp_oauth_pending.c.state == "rand-state-xyz")
).one()
assert row.user_id == "u1"
assert row.server_name == "srv-a"
assert row.return_url == "/admin/mcp-servers"
def test_state_pk_unique(self, backend) -> None:
"""A second insert with the same state value raises IntegrityError."""
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_oauth_pending),
{
"state": "dup-state",
"user_id": "u1",
"server_name": "srv-a",
"code_verifier": "v",
"return_url": "/x",
"created_at": "2026-05-04T11:00:00",
},
)
conn.commit()
import pytest
from sqlalchemy.exc import IntegrityError
with pytest.raises(IntegrityError), backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_oauth_pending),
{
"state": "dup-state",
"user_id": "u2",
"server_name": "srv-b",
"code_verifier": "v",
"return_url": "/y",
"created_at": "2026-05-04T11:01:00",
},
)
conn.commit()
-53
View File
@@ -1,53 +0,0 @@
"""PKCE pair-generation tests for the MCP OAuth flow.
Verifies the contract documented in RFC 7636 §4.1 and §4.2:
- ``code_verifier`` is a high-entropy 43..128 character urlsafe-base64 string.
- ``code_challenge`` is the BASE64URL-NO-PADDING encoding of
``SHA256(verifier)``.
"""
from __future__ import annotations
import base64
import hashlib
import string
from turnstone.core.mcp_oauth import generate_pkce_pair
_URLSAFE_CHARS = set(string.ascii_letters + string.digits + "-_")
class TestGeneratePkcePair:
def test_returns_tuple_of_strings(self) -> None:
verifier, challenge = generate_pkce_pair()
assert isinstance(verifier, str)
assert isinstance(challenge, str)
def test_verifier_length_in_rfc_range(self) -> None:
for _ in range(20):
verifier, _ = generate_pkce_pair()
assert 43 <= len(verifier) <= 128
def test_verifier_is_urlsafe(self) -> None:
for _ in range(20):
verifier, _ = generate_pkce_pair()
assert all(ch in _URLSAFE_CHARS for ch in verifier)
def test_challenge_matches_sha256_of_verifier(self) -> None:
for _ in range(20):
verifier, challenge = generate_pkce_pair()
digest = hashlib.sha256(verifier.encode("ascii")).digest()
expected = base64.urlsafe_b64encode(digest).rstrip(b"=").decode("ascii")
assert challenge == expected
def test_challenge_has_no_padding(self) -> None:
for _ in range(20):
_, challenge = generate_pkce_pair()
assert "=" not in challenge
def test_pairs_are_unique(self) -> None:
pairs = {generate_pkce_pair() for _ in range(50)}
# 50 random draws shouldn't collide; if they do we have a much
# bigger problem than this assertion.
assert len(pairs) == 50
File diff suppressed because it is too large Load Diff
-399
View File
@@ -1,399 +0,0 @@
"""Tests for :func:`turnstone.core.mcp_oauth.revoke_token_at_as`.
The helper is best-effort RFC 7009 token revocation. It must:
- skip cleanly when the AS metadata doesn't advertise a revocation endpoint
- POST the form body when one is present (with optional client_secret)
- never raise on non-2xx, network errors, or timeouts caller doesn't
want try/except in cleanup paths
- never use ``exc_info=True`` chained ``__context__`` may carry an
``httpx.Request`` whose ``Authorization`` header holds a bearer; the
bearer-leak invariant requires structured fields with type names only
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
from turnstone.core.mcp_oauth import (
ASMetadata,
MCPOAuthDiscoveryError,
_attempt_upstream_revoke,
revoke_token_at_as,
)
def _make_as_metadata(
*,
revocation_endpoint: str | None = "https://as.example.com/revoke",
) -> ASMetadata:
return ASMetadata(
issuer="https://as.example.com",
authorization_endpoint="https://as.example.com/authorize",
token_endpoint="https://as.example.com/token",
registration_endpoint=None,
revocation_endpoint=revocation_endpoint,
jwks_uri=None,
code_challenge_methods_supported=("S256",),
token_endpoint_auth_methods_supported=("client_secret_basic",),
)
def _mk_response(status_code: int) -> MagicMock:
resp = MagicMock(spec=httpx.Response)
resp.status_code = status_code
return resp
class TestRevocationUnsupported:
def test_revoke_token_skipped_when_revocation_endpoint_none(self) -> None:
as_meta = _make_as_metadata(revocation_endpoint=None)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
client.post.assert_not_called()
info_events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_unsupported" in info_events
class TestRevocationSuccess:
def test_revoke_token_succeeds_on_200(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(200))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret="s-secret",
)
)
# POST shape — URL + form body keys.
client.post.assert_awaited_once()
call_args = client.post.call_args
assert call_args.args[0] == "https://as.example.com/revoke"
body = call_args.kwargs["data"]
assert body == {
"token": "r-secret",
"token_type_hint": "refresh_token",
"client_id": "client-1",
"client_secret": "s-secret",
}
info_events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_succeeded" in info_events
def test_revoke_token_omits_client_secret_when_none(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(200))
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
body = client.post.call_args.kwargs["data"]
assert "client_secret" not in body
assert body["token"] == "r-secret"
assert body["token_type_hint"] == "refresh_token"
assert body["client_id"] == "client-1"
def test_revoke_token_succeeds_on_204(self) -> None:
# RFC 7009 says the AS MAY return any 2xx; treat the whole range
# as success.
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(204))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
info_events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_succeeded" in info_events
class TestRevocationFailureLogged:
def _run_and_capture(self, status: int) -> list[Any]:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(status))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
return mock_log.info.call_args_list
def test_revoke_token_logs_on_400_does_not_raise(self) -> None:
calls = self._run_and_capture(400)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
# Must include status field.
failed_call = next(c for c in calls if c.args[0] == "mcp_server.oauth.revocation_failed")
assert failed_call.kwargs.get("status") == 400
def test_revoke_token_logs_on_401_does_not_raise(self) -> None:
calls = self._run_and_capture(401)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(c for c in calls if c.args[0] == "mcp_server.oauth.revocation_failed")
assert failed_call.kwargs.get("status") == 401
def test_revoke_token_logs_on_403_does_not_raise(self) -> None:
calls = self._run_and_capture(403)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
def test_revoke_token_logs_on_5xx_does_not_raise(self) -> None:
calls = self._run_and_capture(500)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(c for c in calls if c.args[0] == "mcp_server.oauth.revocation_failed")
assert failed_call.kwargs.get("status") == 500
class TestRevocationExceptionPaths:
def test_revoke_token_handles_network_error(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(side_effect=httpx.ConnectError("boom"))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(
c
for c in mock_log.info.call_args_list
if c.args[0] == "mcp_server.oauth.revocation_failed"
)
assert failed_call.kwargs.get("error") == "ConnectError"
def test_revoke_token_handles_httpx_timeout(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(side_effect=httpx.TimeoutException("slow"))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(
c
for c in mock_log.info.call_args_list
if c.args[0] == "mcp_server.oauth.revocation_failed"
)
assert failed_call.kwargs.get("error") == "TimeoutException"
def test_revoke_token_handles_asyncio_timeout(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
async def _slow(*_args: Any, **_kwargs: Any) -> Any:
await asyncio.sleep(10.0)
raise AssertionError("should have timed out")
client.post = AsyncMock(side_effect=_slow)
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
timeout_seconds=0.05,
)
)
events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(
c
for c in mock_log.info.call_args_list
if c.args[0] == "mcp_server.oauth.revocation_failed"
)
# ``asyncio.timeout`` raises ``TimeoutError`` (Python's builtin)
# on cancellation.
assert failed_call.kwargs.get("error") == "TimeoutError"
def test_revoke_token_no_exc_info_in_logs(self) -> None:
"""Bearer-leak invariant: the revoke path must NEVER set
``exc_info=True``. Chained ``__context__`` may include an
``httpx.Request`` whose ``Authorization`` header holds a
bearer; the traceback formatter would render it.
"""
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(side_effect=httpx.ConnectError("boom"))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
# No info call may carry exc_info.
for call in mock_log.info.call_args_list:
assert "exc_info" not in call.kwargs, (
f"mcp_server.oauth log info({call.args[0]!r}) used exc_info — "
"this violates the bearer-leak invariant"
)
# Defensively: also check warning + exception levels for the
# same call site.
for call in mock_log.warning.call_args_list:
assert "exc_info" not in call.kwargs
mock_log.exception.assert_not_called()
class TestAttemptUpstreamRevokeNeverRaises:
"""Round-2 q-3 regression: ``_attempt_upstream_revoke``'s docstring
claims ``Never raises``. Background-task semantics make this load-
bearing a propagated exception logs ``Task exception was never
retrieved`` because the ``set.discard`` done-callback doesn't read
``task.exception()``.
The wrapper's narrow inner ``except`` clauses (``MCPOAuthDiscoveryError``,
``MCPTokenDecryptError``) leave room for any other exception type
raised by ``discover_authorization_server`` /
``storage.get_mcp_oauth_client_secret_ct`` / ``token_store.cipher.decrypt``
to escape. The outer ``try/except Exception`` is what keeps the
contract honest. These tests pin that gate.
"""
def _build_args(self) -> dict[str, Any]:
token_store = MagicMock()
token_store.cipher = MagicMock()
token_store.cipher.decrypt.return_value = b"shh"
storage = MagicMock()
storage.get_mcp_oauth_client_secret_ct.return_value = None
return {
"http_client": MagicMock(spec=httpx.AsyncClient),
"metadata_cache": None,
"storage": storage,
"token_store": token_store,
"server_name": "srv-oauth",
"server_row": {
"url": "https://mcp.example.com",
"oauth_client_id": "client-1",
"oauth_authorization_server_url": None,
"oauth_as_issuer_cached": None,
},
"server_id_for_audit": "srv-id-1",
"refresh_token": "r-secret",
}
def test_attempt_upstream_revoke_swallows_unexpected_exception(self) -> None:
"""A generic exception from a path the inner handlers don't
cover MUST be caught at the outer boundary and logged with type
name only (no exc_info=True per the bearer-leak invariant).
"""
args = self._build_args()
async def _boom(*_a: Any, **_kw: Any) -> Any:
raise RuntimeError("network blew up")
with (
patch("turnstone.core.mcp_oauth.discover_authorization_server", side_effect=_boom),
patch("turnstone.core.mcp_oauth.log") as mock_log,
):
# MUST NOT raise.
asyncio.run(_attempt_upstream_revoke(**args))
events = [call.args[0] for call in mock_log.info.call_args_list]
assert "mcp_server.oauth.upstream_revoke_failed" in events, (
"outer try/except must log mcp_server.oauth.upstream_revoke_failed "
"with the exception type name when an unexpected exception escapes "
"the narrow inner handlers"
)
for call in mock_log.info.call_args_list:
assert "exc_info" not in call.kwargs, (
"outer-block log must not use exc_info=True — chained "
"__context__ may carry an httpx.Request bearer"
)
def test_attempt_upstream_revoke_logs_discovery_failure(self) -> None:
"""Round-2 bug-1: ``MCPOAuthDiscoveryError`` MUST emit
``upstream_revoke_discovery_failed`` so operators have visibility
into a silent-discovery-failure path that previously logged
nothing while the audit row recorded ``upstream_revoke_outcome=scheduled``.
"""
args = self._build_args()
async def _disc_fail(*_a: Any, **_kw: Any) -> Any:
raise MCPOAuthDiscoveryError("PRM fetch 503")
with (
patch(
"turnstone.core.mcp_oauth.discover_authorization_server",
side_effect=_disc_fail,
),
patch("turnstone.core.mcp_oauth.log") as mock_log,
):
asyncio.run(_attempt_upstream_revoke(**args))
events = [call.args[0] for call in mock_log.info.call_args_list]
assert "mcp_server.oauth.upstream_revoke_discovery_failed" in events
assert "mcp_server.oauth.upstream_revoke_failed" not in events
-212
View File
@@ -1,212 +0,0 @@
"""Storage CRUD tests for the per-(user, server) MCP OAuth pending-state table.
Validates the storage-protocol additions for the per-(user, server)
OAuth flow:
- ``create_mcp_oauth_pending_state``
- ``pop_mcp_oauth_pending_state`` (atomic, with TTL)
- ``cleanup_expired_mcp_oauth_pending_states``
- ``get_mcp_oauth_client_secret_ct``
"""
from __future__ import annotations
import sqlalchemy as sa
class TestCreateAndPop:
def test_round_trip(self, backend) -> None:
backend.create_mcp_oauth_pending_state(
"state-1",
"user-a",
"srv-x",
"verifier-blob",
"/admin/mcp-servers",
)
row = backend.pop_mcp_oauth_pending_state("state-1", max_age_seconds=600)
assert row is not None
assert row["state"] == "state-1"
assert row["user_id"] == "user-a"
assert row["server_name"] == "srv-x"
assert row["code_verifier"] == "verifier-blob"
assert row["return_url"] == "/admin/mcp-servers"
def test_pop_consumes_row(self, backend) -> None:
backend.create_mcp_oauth_pending_state("s2", "u", "s", "v", "/r")
first = backend.pop_mcp_oauth_pending_state("s2")
assert first is not None
# Second pop must miss — row was consumed.
second = backend.pop_mcp_oauth_pending_state("s2")
assert second is None
def test_pop_missing_returns_none(self, backend) -> None:
assert backend.pop_mcp_oauth_pending_state("never-existed") is None
class TestTTL:
def test_pop_rejects_expired_row(self, backend) -> None:
backend.create_mcp_oauth_pending_state("old-state", "u", "s", "v", "/r")
# Backdate it so it's older than the TTL window.
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_oauth_pending SET created_at = '2020-01-01T00:00:00' "
"WHERE state = 'old-state'"
)
)
conn.commit()
# Default TTL is 600s — the row is decades old.
row = backend.pop_mcp_oauth_pending_state("old-state")
assert row is None
# Even though pop returned None, the row must have been wiped — a
# second pop with a giant TTL must still see nothing.
again = backend.pop_mcp_oauth_pending_state("old-state", max_age_seconds=10**9)
assert again is None
def test_pop_accepts_fresh_row(self, backend) -> None:
backend.create_mcp_oauth_pending_state("fresh", "u", "s", "v", "/r")
row = backend.pop_mcp_oauth_pending_state("fresh", max_age_seconds=600)
assert row is not None
assert row["state"] == "fresh"
class TestCleanup:
def test_cleanup_deletes_only_expired(self, backend) -> None:
backend.create_mcp_oauth_pending_state("old", "u", "s", "v", "/r")
backend.create_mcp_oauth_pending_state("new", "u", "s", "v", "/r")
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_oauth_pending SET created_at = '2020-01-01T00:00:00' "
"WHERE state = 'old'"
)
)
conn.commit()
deleted = backend.cleanup_expired_mcp_oauth_pending_states(max_age_seconds=600)
assert deleted == 1
# Old gone, new still around.
assert backend.pop_mcp_oauth_pending_state("old") is None
survivor = backend.pop_mcp_oauth_pending_state("new")
assert survivor is not None
def test_cleanup_no_rows(self, backend) -> None:
assert backend.cleanup_expired_mcp_oauth_pending_states() == 0
class TestGetOAuthClientSecretCt:
def test_returns_none_when_unset(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-id",
name="srv-x",
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
)
assert backend.get_mcp_oauth_client_secret_ct("srv-id") is None
def test_returns_ciphertext_after_set(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-id",
name="srv-x",
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
)
ct = b"\x00\xff\x42encrypted-blob"
ok = backend.set_mcp_oauth_client_secret_ct("srv-id", ct)
assert ok is True
out = backend.get_mcp_oauth_client_secret_ct("srv-id")
assert out == ct
def test_returns_none_for_missing_server(self, backend) -> None:
assert backend.get_mcp_oauth_client_secret_ct("does-not-exist") is None
def _create_user_token_row(
backend,
*,
user_id: str,
server_name: str,
created: str,
) -> None:
"""Insert a token row + backdate ``created`` so ordering is deterministic.
The storage helper stamps ``created`` from ``datetime.now(UTC)``; for
multi-row ordering tests we backdate via raw SQL so the inserts stay
independent of clock resolution.
"""
backend.create_mcp_user_token(
user_id,
server_name,
access_token_ct=b"ct-access",
refresh_token_ct=b"ct-refresh",
expires_at="2026-05-04T12:00:00",
scopes="openid",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_user_tokens SET created = :created "
"WHERE user_id = :uid AND server_name = :sn"
),
{"created": created, "uid": user_id, "sn": server_name},
)
conn.commit()
class TestListMCPUserTokenMetadataByUser:
def test_list_mcp_user_token_metadata_by_user_empty(self, backend) -> None:
assert backend.list_mcp_user_token_metadata_by_user("nobody") == []
def test_list_mcp_user_token_metadata_by_user_single_server(self, backend) -> None:
_create_user_token_row(
backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
rows = backend.list_mcp_user_token_metadata_by_user("u1")
assert len(rows) == 1
assert rows[0]["user_id"] == "u1"
assert rows[0]["server_name"] == "srv-a"
assert rows[0]["as_issuer"] == "https://auth.example.com"
assert rows[0]["audience"] == "https://mcp.example.com"
assert rows[0]["scopes"] == "openid"
# Projection MUST omit ciphertext columns — the SQL no longer
# selects them, so the TypedDict has no key.
assert "access_token_ct" not in rows[0]
assert "refresh_token_ct" not in rows[0]
def test_list_mcp_user_token_metadata_by_user_multiple_servers(self, backend) -> None:
_create_user_token_row(
backend, user_id="u1", server_name="srv-c", created="2026-05-03T00:00:00"
)
_create_user_token_row(
backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
_create_user_token_row(
backend, user_id="u1", server_name="srv-b", created="2026-05-02T00:00:00"
)
rows = backend.list_mcp_user_token_metadata_by_user("u1")
assert [r["server_name"] for r in rows] == ["srv-a", "srv-b", "srv-c"]
def test_list_mcp_user_token_metadata_by_user_isolates_by_user(self, backend) -> None:
_create_user_token_row(
backend, user_id="user-a", server_name="srv-a", created="2026-05-01T00:00:00"
)
_create_user_token_row(
backend, user_id="user-a", server_name="srv-b", created="2026-05-02T00:00:00"
)
_create_user_token_row(
backend, user_id="user-b", server_name="srv-a", created="2026-05-03T00:00:00"
)
rows_a = backend.list_mcp_user_token_metadata_by_user("user-a")
assert {r["server_name"] for r in rows_a} == {"srv-a", "srv-b"}
assert all(r["user_id"] == "user-a" for r in rows_a)
rows_b = backend.list_mcp_user_token_metadata_by_user("user-b")
assert len(rows_b) == 1
assert rows_b[0]["user_id"] == "user-b"
assert rows_b[0]["server_name"] == "srv-a"
-304
View File
@@ -1,304 +0,0 @@
"""Boundary tests for the Phase 9 pending-consent write path.
Drives ``MCPClientManager._dispatch_pool_sync`` (and the helper it
calls, ``_record_pending_consent_best_effort``) and asserts that
deferred-consent records reach storage only on non-interactive callers.
Per ``feedback_tests_through_boundaries.md``, at least one test must
drive the real sync dispatcher real ``_is_structured_error``
real ``_record_pending_consent_best_effort`` plumb-through; the
``_helpers`` unit tests below cover the classifier in isolation, but
the end-to-end test is the structural gate that catches
plumb-through regressions.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import threading
from typing import Any
from unittest.mock import patch
import pytest
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import (
_PENDING_CONSENT_PERSIST_CODES,
MCPClientManager,
_parse_pending_consent_envelope,
)
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
# ---------------------------------------------------------------------------
# Helper-level unit tests (cheap, no event loop)
# ---------------------------------------------------------------------------
class TestParseEnvelope:
def test_consent_required_no_scopes(self) -> None:
env = json.dumps({"error": {"code": "mcp_consent_required", "server": "x", "detail": "d"}})
assert _parse_pending_consent_envelope(env) == ("mcp_consent_required", None)
def test_insufficient_scope_with_scopes(self) -> None:
env = json.dumps(
{
"error": {
"code": "mcp_insufficient_scope",
"server": "x",
"detail": "d",
"scopes_required": ["read", "write"],
}
}
)
assert _parse_pending_consent_envelope(env) == (
"mcp_insufficient_scope",
["read", "write"],
)
def test_operator_codes_filtered(self) -> None:
# Key-unknown / url-insecure / *_forbidden are operator-actionable,
# NOT user-consent-shaped. They must not produce pending-consent
# rows, regardless of whether the caller is interactive.
for code in (
"mcp_token_undecryptable_key_unknown",
"mcp_oauth_url_insecure",
"mcp_tool_call_forbidden",
"mcp_resource_read_forbidden",
"mcp_prompt_get_forbidden",
):
env = json.dumps({"error": {"code": code, "server": "x", "detail": "d"}})
assert _parse_pending_consent_envelope(env) is None, code
def test_malformed_json_returns_none(self) -> None:
assert _parse_pending_consent_envelope("not json") is None
assert _parse_pending_consent_envelope("") is None
def test_persist_codes_set_is_expected(self) -> None:
# Pin the contract — adding a new persistable code here is a
# deliberate design decision and should require a test update.
assert {
"mcp_consent_required",
"mcp_insufficient_scope",
} == _PENDING_CONSENT_PERSIST_CODES
# ---------------------------------------------------------------------------
# End-to-end plumb-through (drives _dispatch_pool_sync)
# ---------------------------------------------------------------------------
def _seed_oauth_server(backend: Any, *, name: str = "pool-srv") -> None:
backend.create_mcp_server(
server_id="srv-" + name,
name=name,
transport="streamable-http",
command="",
args="[]",
url="https://example.com/mcp",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-" + name, auth_type="oauth_user")
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="phase9-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _wire_mgr(mgr: MCPClientManager, backend: Any) -> None:
cipher = make_mcp_token_cipher()
from types import SimpleNamespace
from unittest.mock import MagicMock
app_state = SimpleNamespace(
auth_storage=backend,
mcp_token_store=MCPTokenStore(backend, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
mgr.set_storage(backend)
mgr.set_app_state(app_state)
def test_dispatch_persists_pending_for_non_interactive_caller(
running_loop_mgr: Any, backend: Any
) -> None:
"""Non-interactive caller hits ``mcp_consent_required`` → a
``mcp_pending_consent`` row appears for ``(user_id, server_name)``."""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
# Structured error envelope surfaces as RuntimeError to the caller.
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_consent_required"
# Persistent row written for the dashboard badge.
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
assert r["user_id"] == "user-a"
assert r["server_name"] == "pool-srv"
assert r["error_code"] == "mcp_consent_required"
assert r["occurrence_count"] == 1
def test_dispatch_does_not_persist_for_interactive_caller(
running_loop_mgr: Any, backend: Any
) -> None:
"""Interactive caller hits the same error path → NO row written.
Interactive (WEB / CLI) sessions surface the consent prompt in-flight
via the Phase 8 SSE renderer; persisting would just produce
immediately-stale dashboard badges.
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError),
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=True,
)
assert backend.list_mcp_pending_consent_by_user("user-a") == []
def test_dispatch_returns_envelope_unchanged_on_storage_failure(
running_loop_mgr: Any, backend: Any
) -> None:
"""When ``upsert_mcp_pending_consent`` raises, the agent-observable
contract is unchanged: the structured-error ``RuntimeError`` still
surfaces with the original ``mcp_consent_required`` code. The doc-
string promises best-effort persistence; this test pins that
promise so a regression that propagates the storage exception would
fail visibly.
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
original_upsert = backend.upsert_mcp_pending_consent
def _raise(*_a: Any, **_kw: Any) -> None:
raise RuntimeError("storage offline")
backend.upsert_mcp_pending_consent = _raise # type: ignore[method-assign]
try:
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
finally:
backend.upsert_mcp_pending_consent = original_upsert # type: ignore[method-assign]
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_consent_required"
def test_dispatch_does_not_persist_for_operator_actionable_code(
running_loop_mgr: Any, backend: Any
) -> None:
"""Decrypt-failure → operator-actionable; even non-interactive callers
must NOT produce a user-facing pending-consent record (the user can't
resolve this by re-consenting).
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _decrypt_failure(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_decrypt_failure,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_token_undecryptable_key_unknown"
# The operator-actionable code does NOT produce a pending-consent row.
assert backend.list_mcp_pending_consent_by_user("user-a") == []
-259
View File
@@ -1,259 +0,0 @@
"""HTTP tests for the Phase 9 pending-consent endpoints.
Covers:
- ``GET /v1/api/mcp/oauth/pending`` (install gate + read path)
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` (single clear)
- ``DELETE /v1/api/mcp/oauth/pending`` (bulk clear)
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.core.auth import AuthResult
from turnstone.core.mcp_oauth import (
handle_mcp_oauth_clear_all_pending,
handle_mcp_oauth_clear_pending,
handle_mcp_oauth_list_pending,
)
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
class _InjectAuthMiddleware(BaseHTTPMiddleware):
"""Stamp a fixed authenticated user on every request."""
def __init__(self, app: Any, user_id: str = "user-1") -> None:
super().__init__(app)
self._user_id = user_id
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id=self._user_id,
scopes=frozenset({"write"}),
token_source="config",
permissions=frozenset({"read", "write"}),
)
return await call_next(request)
def _build_app(storage: SQLiteBackend, *, user_id: str = "user-1") -> Starlette:
class _Mw(_InjectAuthMiddleware):
def __init__(self, app: Any) -> None:
super().__init__(app, user_id=user_id)
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/mcp/oauth/pending", handle_mcp_oauth_list_pending),
Route(
"/api/mcp/oauth/pending",
handle_mcp_oauth_clear_all_pending,
methods=["DELETE"],
),
Route(
"/api/mcp/oauth/pending/{server_name}",
handle_mcp_oauth_clear_pending,
methods=["DELETE"],
),
],
),
],
middleware=[Middleware(_Mw)],
)
app.state.auth_storage = storage
return app
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
backend = SQLiteBackend(str(tmp_path / "test.db"))
backend.create_user("user-1", "user1", "User One", "hash")
backend.create_user("user-2", "user2", "User Two", "hash")
return backend
def _seed_oauth_server(backend: SQLiteBackend, *, name: str = "srv-x") -> None:
backend.create_mcp_server(
server_id="srv-id-" + name,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="oauth_user",
)
def _seed_pending(
backend: SQLiteBackend,
*,
user_id: str = "user-1",
server_name: str = "srv-x",
error_code: str = "mcp_consent_required",
now_iso: str = "2026-05-11T12:00:00",
) -> None:
backend.upsert_mcp_pending_consent(
user_id=user_id,
server_name=server_name,
error_code=error_code,
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=now_iso,
)
class TestListPending:
def test_install_gate_short_circuits_on_no_oauth_servers(self, storage: SQLiteBackend) -> None:
# Seed a pending row but NO oauth_user MCP server — the gate
# must short-circuit to {pending: 0} regardless.
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"pending": 0, "servers": []}
def test_lists_pending_records_for_authenticated_user(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
body = resp.json()
assert body["pending"] == 1
assert len(body["servers"]) == 1
assert body["servers"][0]["server_name"] == "srv-x"
assert body["servers"][0]["error_code"] == "mcp_consent_required"
def test_does_not_leak_cross_user_records(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage, user_id="user-2")
client = TestClient(_build_app(storage, user_id="user-1"))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"pending": 0, "servers": []}
class TestClearPending:
def test_delete_single(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
assert storage.list_mcp_pending_consent_by_user("user-1") == []
def test_delete_missing_still_returns_204(self, storage: SQLiteBackend) -> None:
# Idempotent — must not leak cross-user existence info via 404.
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
assert resp.status_code == 204
def test_delete_does_not_touch_cross_user_rows(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage, user_id="user-1")
_seed_pending(storage, user_id="user-2")
client = TestClient(_build_app(storage, user_id="user-1"))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
# User-2's row survives.
assert len(storage.list_mcp_pending_consent_by_user("user-2")) == 1
class TestAuditTrail:
def test_single_dismiss_audits(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "single"
assert detail.get("cleared") == 1
def test_single_dismiss_audits_even_when_no_row_existed(self, storage: SQLiteBackend) -> None:
# Cross-tenant non-observability requires a 204 in the never-existed
# case — the audit row distinguishes a real dismiss from a stuffed
# attempt by recording ``cleared=0``.
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
assert resp.status_code == 204
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "single"
assert detail.get("cleared") == 0
def test_bulk_dismiss_audits(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_oauth_server(storage, name="srv-y")
_seed_pending(storage, server_name="srv-x")
_seed_pending(storage, server_name="srv-y")
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 2}
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "bulk"
assert detail.get("cleared") == 2
class TestClearAllPending:
def test_bulk_clear(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_oauth_server(storage, name="srv-y")
_seed_pending(storage, server_name="srv-x")
_seed_pending(storage, server_name="srv-y")
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 2}
assert storage.list_mcp_pending_consent_by_user("user-1") == []
def test_bulk_clear_zero_when_empty(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 0}
-263
View File
@@ -1,263 +0,0 @@
"""Storage CRUD tests for the Phase 9 ``mcp_pending_consent`` table.
Validates protocol additions backing the dashboard pending-consent badge:
- ``upsert_mcp_pending_consent`` insert + on-conflict refresh
- ``list_mcp_pending_consent_by_user`` read path
- ``delete_mcp_pending_consent`` single-row clear
- ``delete_all_mcp_pending_consent_by_user`` bulk clear
- ``count_mcp_consented_users_by_server`` admin status pill
- ``any_oauth_user_mcp_servers`` install-level gate
"""
from __future__ import annotations
def _iso(ts: str = "2026-05-11T12:00:00") -> str:
return ts
class TestUpsertAndList:
def test_insert_round_trip(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required="read write",
last_ws_id="ws-1",
last_tool_call_id="tool-1",
now_iso=_iso(),
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
assert r["user_id"] == "user-a"
assert r["server_name"] == "srv-x"
assert r["error_code"] == "mcp_consent_required"
assert r["scopes_required"] == "read write"
assert r["last_ws_id"] == "ws-1"
assert r["last_tool_call_id"] == "tool-1"
assert r["occurrence_count"] == 1
assert r["first_seen_at"] == r["last_seen_at"]
def test_upsert_bumps_count_and_refreshes_recency(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T12:00:00",
)
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_insufficient_scope",
scopes_required="read",
last_ws_id="ws-2",
last_tool_call_id="tool-2",
now_iso="2026-05-11T13:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
# Recency fields refreshed to the second call's values; count bumped.
assert r["occurrence_count"] == 2
assert r["error_code"] == "mcp_insufficient_scope"
assert r["scopes_required"] == "read"
assert r["last_ws_id"] == "ws-2"
assert r["last_tool_call_id"] == "tool-2"
assert r["last_seen_at"] == "2026-05-11T13:00:00"
# first_seen_at preserved — that's the load-bearing audit value.
assert r["first_seen_at"] == "2026-05-11T12:00:00"
def test_list_orders_by_last_seen_desc(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-old",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T10:00:00",
)
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-new",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T11:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert [r["server_name"] for r in rows] == ["srv-new", "srv-old"]
def test_per_user_isolation(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.list_mcp_pending_consent_by_user("user-b") == []
class TestDelete:
def test_delete_single(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is True
assert backend.list_mcp_pending_consent_by_user("user-a") == []
# Second delete returns False (no row).
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is False
def test_delete_missing_returns_false(self, backend) -> None:
assert backend.delete_mcp_pending_consent("never", "missing") is False
def test_delete_all_by_user(self, backend) -> None:
for name in ("srv-a", "srv-b", "srv-c"):
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name=name,
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
# Cross-user row that must NOT be touched.
backend.upsert_mcp_pending_consent(
user_id="user-b",
server_name="srv-z",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_all_mcp_pending_consent_by_user("user-a") == 3
assert backend.list_mcp_pending_consent_by_user("user-a") == []
assert len(backend.list_mcp_pending_consent_by_user("user-b")) == 1
class TestCountConsentedUsersByServer:
def _seed_server(self, backend, name: str = "srv-x") -> None:
backend.create_mcp_server(
server_id="srv-id-" + name,
name=name,
transport="streamable-http",
command="",
args="[]",
url="https://example.com/mcp",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-id-" + name, auth_type="oauth_user")
def test_counts_distinct_non_expired_users(self, backend) -> None:
self._seed_server(backend)
future = "2099-01-01T00:00:00"
backend.create_mcp_user_token(
"alice",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=future,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
backend.create_mcp_user_token(
"bob",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=None, # null treated as non-expired
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
# Different server — must not count.
self._seed_server(backend, name="srv-y")
backend.create_mcp_user_token(
"carol",
"srv-y",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=future,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
assert backend.count_mcp_consented_users_by_server("srv-x") == 2
assert backend.count_mcp_consented_users_by_server("srv-y") == 1
def test_excludes_expired(self, backend) -> None:
self._seed_server(backend)
backend.create_mcp_user_token(
"alice",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at="2020-01-01T00:00:00", # well in the past
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
assert backend.count_mcp_consented_users_by_server("srv-x") == 0
def test_zero_when_no_rows(self, backend) -> None:
assert backend.count_mcp_consented_users_by_server("missing") == 0
class TestInstallGate:
def test_any_oauth_user_returns_false_on_empty(self, backend) -> None:
assert backend.any_oauth_user_mcp_servers() is False
def test_any_oauth_user_ignores_static_rows(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-1",
name="static-only",
transport="streamable-http",
command="",
args="[]",
url="https://example.com",
headers='{"Authorization": "Bearer x"}',
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
assert backend.any_oauth_user_mcp_servers() is False
def test_any_oauth_user_returns_true_when_one_exists(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-2",
name="oauth-srv",
transport="streamable-http",
command="",
args="[]",
url="https://example.com",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-2", auth_type="oauth_user")
assert backend.any_oauth_user_mcp_servers() is True
-912
View File
@@ -1,912 +0,0 @@
"""Phase 6 integration tests — real-transport drives 401/403 through the SDK.
These are the structural exit criterion for Phase 6. They MUST drive
through the real ``streamablehttp_client``, the real httpx response-hook
path, and a REAL upstream MCP server (a ``FastMCP`` in-process subprocess
with a starlette middleware that programmatically returns 401/403 with
crafted ``WWW-Authenticate`` headers).
Direct ``httpx.HTTPStatusError`` injection is FORBIDDEN here Phase 5
bug-1 was masked precisely by that pattern (the production code path
was structurally unreachable, but the unit-test injection bypassed the
SDK's swallow). The integration tests gate that the production path
actually receives the carrier signal end-to-end.
The fixture upstream is built in-thread (uvicorn on its own asyncio
loop in a background thread) same pattern as
``tests/spike_sdk_concurrency.py``. Per the orchestrator's startup-cost
note, measured at ~0.05s per fixture spin-up locally; well under the
2s threshold for default-collection inclusion.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import socket
import threading
import time
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
import pytest
import uvicorn
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
# Quiet noisy logs during tests.
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
# ---------------------------------------------------------------------------
# Fixture upstream — programmable BehaviorMiddleware
# ---------------------------------------------------------------------------
class BehaviorMiddleware(BaseHTTPMiddleware):
"""Inspects per-request behaviour state and returns 401/403 on demand.
The behaviour is steered by a mutable ``behaviour`` dict on the
middleware instance; tests mutate it via the fixture handle.
Records every request's Authorization header for assertion.
Behaviour semantics:
* ``"once_401"``: return 401 once, then 200 thereafter.
* ``"always_401"``: always return 401.
* ``"once_403_insufficient"``: return 403 with insufficient_scope once.
* ``"once_403_generic"``: return 403 without error param once.
* ``"once_multi_www_auth_403"``: return 403 with TWO
``WWW-Authenticate`` headers first ``Bearer`` challenge
carries the SAFE scopes, second carries INJECTED scopes. The
dispatcher must report only the first.
* ``"never"`` (default): pass through to the real handler.
``www_authenticate`` overrides the default header crafted per shape.
"""
def __init__(self, app: Any, behaviour: dict[str, Any]) -> None:
super().__init__(app)
self._behaviour = behaviour
async def dispatch(self, request: Request, call_next: Callable[..., Any]) -> Response:
from starlette.responses import Response as StarletteResponse
# Record the Authorization header for assertion. POST is the
# tools/call request the dispatcher sends.
if request.method == "POST" and "/mcp" in str(request.url):
self._behaviour.setdefault("post_auth_headers", []).append(
request.headers.get("authorization")
)
mode = self._behaviour.get("mode", "never")
if mode == "once_401":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "always_401":
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "once_403_insufficient":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate",
'Bearer error="insufficient_scope", scope="files:write mail:send"',
)
},
)
elif mode == "once_403_generic" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get("www_authenticate", "Bearer realm=mcp")
},
)
elif mode == "once_multi_www_auth_403" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
# Two ``WWW-Authenticate: Bearer ...`` challenges. The
# first carries ``error=insufficient_scope`` but NO
# ``scope=`` parameter; the second carries the INJECTED
# scopes the dispatcher must NOT report. The first
# challenge intentionally lacks ``scope`` because
# ``parse_www_authenticate_bearer`` uses ``setdefault`` —
# if the first challenge HAD a scope, ``setdefault`` would
# already win on first-occurrence. The vector this test
# guards is the case where a defended absence becomes a
# silent presence: a hook regression to ``get(...)`` joins
# repeated headers with ``, `` and the parser then folds
# the second challenge's scope into the first challenge's
# params dict because there is no first-occurrence to
# protect.
response = StarletteResponse("forbidden", status_code=403)
response.headers.append(
"www-authenticate",
'Bearer realm="legit", error="insufficient_scope"',
)
response.headers.append(
"www-authenticate",
'Bearer error="insufficient_scope", scope="org:admin db:write"',
)
return response
return await call_next(request)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _build_server(port: int, behaviour: dict[str, Any]) -> uvicorn.Server:
mcp = FastMCP(name="phase6-target", streamable_http_path="/mcp")
@mcp.tool()
async def echo(payload: str = "default") -> str:
return f"echoed:{payload}"
app = mcp.streamable_http_app()
app.add_middleware(BehaviorMiddleware, behaviour=behaviour)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"upstream at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def upstream():
"""Boot a FastMCP fixture upstream in a background thread.
Yields ``(url, behaviour)`` where ``behaviour`` is a mutable dict
the test mutates to steer the middleware (set ``mode`` to one of
the BehaviorMiddleware shapes).
"""
port = _find_free_port()
behaviour: dict[str, Any] = {}
server = _build_server(port, behaviour)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="phase6-upstream")
t.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp", behaviour
finally:
server.should_exit = True
t.join(timeout=5)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
refresh_token: str | None = "refresh-rrr",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token=refresh_token,
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
# ---------------------------------------------------------------------------
# Test 21: 401 → refresh-and-retry → success
# ---------------------------------------------------------------------------
def test_integration_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Real upstream returns 401 once with ``WWW-Authenticate: Bearer
error="invalid_token"``, then 200. Dispatcher carrier captures the
401, ``force_refresh=True`` mints a new bearer (stubbed), retry
succeeds. Hard invariant 3: breaker counter remains 0.
Drives through the REAL ``streamablehttp_client`` and a REAL
upstream subprocess (no ``httpx.HTTPStatusError`` injection). This
is the structural exit gate for Phase 6 the equivalent unit
tests CANNOT prove the production wiring works because the SDK
swallows the underlying exception.
"""
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
# Override URL to point at the local upstream (loopback http:// is
# exempt from the URL-validator).
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
result = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "hi"}, user_id="user-1", timeout=15
)
assert "echoed:hi" in result
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# Server saw at least 2 POSTs to /mcp (initial + retry).
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, f"expected >=2 POSTs; got {len(post_headers)}"
# Retry carries a different bearer than the initial.
initial = post_headers[0]
retry = post_headers[1]
assert initial != retry, (
"retry attached the same bearer as the initial; the dispatcher "
"did not pick up the refreshed token."
)
# Pool entry has a session after the successful retry.
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
# ---------------------------------------------------------------------------
# Test 22: 401 + refresh failure → mcp_consent_required
# ---------------------------------------------------------------------------
def test_integration_401_with_refresh_failure_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
# Structured-error envelopes flow back via ``RuntimeError(json_str)``
# so the session-layer ``except Exception`` handler routes the
# consent card uniformly across tool / resource / prompt dispatchers.
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
# Phase 8 — consent_url surfaces a /start URL the dashboard can open
# in a popup. URL-encoded server name; no scopes baked in (the AS
# picks up the configured scopes server-side at /start).
assert payload["error"]["consent_url"] == "/v1/api/mcp/oauth/start?server=pool-srv"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Test 23: 403 + insufficient_scope → mcp_insufficient_scope with parsed scopes
# ---------------------------------------------------------------------------
def test_integration_403_insufficient_scope_emits_structured_error(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_insufficient"
behaviour["www_authenticate"] = (
'Bearer error="insufficient_scope", scope="files:write mail:send"'
)
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_insufficient_scope"
assert payload["error"]["scopes_required"] == ["files:write", "mail:send"]
# Phase 8 — consent_url carries the step-up scopes URL-encoded so the
# dashboard can union them with the configured set at /start.
assert payload["error"]["consent_url"] == (
"/v1/api/mcp/oauth/start?server=pool-srv&scopes=files%3Awrite%20mail%3Asend"
)
# No retry — exactly ONE POST attempted before the structured error.
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Test 24: 403 without insufficient_scope → generic forbidden
# ---------------------------------------------------------------------------
def test_integration_403_no_insufficient_scope_emits_generic_forbidden(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_generic"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_tool_call_forbidden"
assert "scopes_required" not in payload["error"]
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
# ---------------------------------------------------------------------------
# sec-1: multi-WWW-Authenticate header injection — only the FIRST
# Bearer challenge feeds the structured-error / audit emission.
# ---------------------------------------------------------------------------
def test_integration_403_multi_www_authenticate_drops_injected_scopes(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Upstream returns a 403 with TWO ``WWW-Authenticate: Bearer ...``
challenges. The first carries ``error=insufficient_scope`` but NO
``scope=`` parameter; the second carries INJECTED scopes
(``["org:admin", "db:write"]``). The dispatcher must report
``scopes_required == []`` derived from the first challenge alone
never the second challenge's injected scopes.
Two layers of defence cooperate (either alone neutralises the
vector; both run together so a regression in one cannot silently
re-open it):
1. ``_make_capturing_http_factory._hook`` reads
``response.headers.get_list("www-authenticate")[0]`` rather than
``response.headers.get(...)`` the latter joins repeated
headers with ``", "`` which the RFC 7235 tokenizer would
otherwise consume as a continuation of the first challenge.
2. ``parse_www_authenticate_bearer`` stops at the first ``Bearer``
challenge boundary even if the input was already joined, so a
hook regression to ``get(...)`` would NOT re-open the vector.
The first challenge intentionally lacks ``scope=`` the parser
uses ``setdefault`` so a first-occurrence ``scope`` would already
win and mask a single-layer regression. The undefended-absence
case is what proves both layers actually do their job.
Negative-test (CRITICAL Phase 5 lesson): verified by reverting
the hook to ``response.headers.get("www-authenticate")`` AND
removing the ``_looks_like_bearer_challenge_start`` guard in
``parse_www_authenticate_bearer``. The test then fails because
``scopes_required`` becomes ``["org:admin", "db:write"]`` the
injected scopes from the second challenge silently fold into the
first challenge's params dict via httpx's comma-joined header
value (the absence of a first-occurrence scope means nothing
blocks the fold).
"""
url, behaviour = upstream
behaviour["mode"] = "once_multi_www_auth_403"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_insufficient_scope", (
f"expected mcp_insufficient_scope; got {payload!r}"
)
# ``scopes_required`` derives from the FIRST challenge alone, which
# carries no ``scope=`` parameter. The injected second challenge
# MUST NOT appear here.
assert payload["error"]["scopes_required"] == [], (
"Multi-header injection slipped through: dispatcher reported "
"scopes from the SECOND Bearer challenge. Got "
f"{payload['error']['scopes_required']!r}; expected []."
)
# ---------------------------------------------------------------------------
# Test 25: 401 retry ceiling — never recurse
# ---------------------------------------------------------------------------
def test_integration_401_retry_ceiling(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Upstream always returns 401; refresh stub keeps minting tokens.
After exactly ONE retry, dispatcher emits ``mcp_consent_required``.
"""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
refresh_count = 0
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
nonlocal refresh_count
if kwargs.get("force_refresh"):
refresh_count += 1
return TokenLookupResult(kind="token", token=f"refreshed-{refresh_count}")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
# Exactly ONE refresh round-trip.
assert refresh_count == 1, f"expected exactly 1 refresh round-trip; got {refresh_count}"
# Server saw EXACTLY 2 POSTs (initial + 1 retry).
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 2, (
f"expected exactly 2 POSTs (initial + 1 retry); got {len(post_headers)}"
)
# ---------------------------------------------------------------------------
# Test 26: breaker unaffected by repeated auth failures (slow — 50 cycles)
# ---------------------------------------------------------------------------
def test_integration_breaker_unaffected_by_auth_failures(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""50 sequential dispatches all hit 401 with refresh-failed → 50
cycles of ``mcp_consent_required``. ``_consecutive_failures`` MUST
stay at 0 throughout (hard invariant 3 verified end-to-end).
"""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
for _ in range(50):
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Test 27: static path unaffected by Phase 6 changes
# ---------------------------------------------------------------------------
def test_integration_static_path_unaffected(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Static-path connect against an unauthed upstream succeeds without
going through the capturing factory. This is the integration-level
mirror of ``test_reconnect_preserves_static_state_identity``.
Drives the static path against the same fixture upstream (with
``behaviour={}`` so middleware passes through) confirms the
static path's session lifecycle is byte-identical even when the
pool path's auth introspection is wired up.
"""
url, _behaviour = upstream
# No mode → middleware passes through to FastMCP.
mgr, loop, _ = running_loop_mgr
# Manually configure mgr with a static-path server pointing at the
# fixture upstream. Use _connect_one (not the pool path).
cfg = {"type": "streamable-http", "url": url}
async def _connect_static() -> None:
await mgr._connect_one("static-srv", cfg)
fut = asyncio.run_coroutine_threadsafe(_connect_static(), loop)
fut.result(timeout=15)
state_before = mgr._static_servers.get("static-srv")
assert state_before is not None
assert state_before.session is not None
# Snapshot identity.
state_id_before = id(state_before)
session_before = state_before.session
# Reconnect — the canonical regression check is that the
# StaticServerState object identity is preserved.
fut = asyncio.run_coroutine_threadsafe(_connect_static(), loop)
fut.result(timeout=15)
state_after = mgr._static_servers.get("static-srv")
assert state_after is not None
assert id(state_after) == state_id_before, (
"Static path StaticServerState identity changed across reconnect; "
"hard invariant 1 violated."
)
assert state_after.session is not None
assert state_after.session is not session_before, (
"Reconnect did not actually replace the session"
)
# ---------------------------------------------------------------------------
# Test 27b: static dispatch unaffected by Phase 8 consent_url kwarg
# ---------------------------------------------------------------------------
def test_static_dispatch_unaffected_by_consent_url_kwarg(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Static-auth tool dispatch must be byte-identical post-Phase 8.
The Phase 8 changes only ADD a ``consent_url`` kwarg to
``_structured_error`` invocations on the pool path. Static dispatch
must not pick up the field there's no consent flow for
``auth_type='none'`` / ``'static'`` servers, and exposing one would
confuse the dashboard renderer. Asserts a successful tool result is
a plain string with no JSON envelope and no ``consent_url`` substring.
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — succeeds
mgr, loop, _ = running_loop_mgr
cfg = {"type": "streamable-http", "url": url}
async def _connect_static() -> None:
await mgr._connect_one("static-srv", cfg)
fut = asyncio.run_coroutine_threadsafe(_connect_static(), loop)
fut.result(timeout=15)
# Drive call_tool_sync without a user_id — the static path is taken.
result = mgr.call_tool_sync("mcp__static-srv__echo", {"payload": "static-x"}, timeout=15)
# Static path returns the FastMCP fixture's echo string.
assert "echoed:static-x" in result
# No JSON envelope leaked through; specifically no consent_url field.
assert "consent_url" not in result, (
f"Static-auth tool dispatch surfaced a consent_url; result: {result!r}"
)
# Defensive: result is not a JSON-encoded structured error.
try:
parsed = json.loads(result)
except (json.JSONDecodeError, ValueError):
parsed = None
if isinstance(parsed, dict):
assert "error" not in parsed, (
f"Static-auth dispatch returned a structured-error envelope; got {parsed!r}"
)
# ---------------------------------------------------------------------------
# Test 28: pool reuse — 401 on a SECOND dispatch (carrier owned by entry)
# ---------------------------------------------------------------------------
def test_integration_pool_reuse_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Reused pool sessions still capture 401 correctly.
Dispatch 1 hits a passthrough upstream (200) and populates
``entry.session``. Dispatch 2 reuses that session no fresh
connect, so a per-dispatch ``_AuthCapture`` would never reach
the httpx response hook (the hook closes over the carrier passed
at first connect, which lives on the entry). A correctly-wired
entry-owned carrier is the only shape that lets dispatch 2's 401
surface to the dispatcher.
Two independent production bugs gate this test passing; both must
hold for reused-session 401 recovery to work end-to-end:
1. The carrier must live on the pool entry (not per-dispatch) so
the response hook bound at first connect writes to the same
object the dispatcher reads across reuse. Verified by reverting
``PoolEntryState.auth_capture`` to a per-dispatch
``_AuthCapture()`` allocation: the carrier-fired event never
reaches the dispatcher and the test times out.
2. The dispatcher must race ``call_tool`` against the carrier's
fired event. The SDK's ``_receive_loop`` runs in BaseSession's
TaskGroup nested inside ``streamablehttp_client``'s TaskGroup;
when an upstream 4xx fires, the outer TaskGroup cancels
``_receive_loop`` mid-finally before it can deliver
``CONNECTION_CLOSED`` to the response stream's waiting
receiver. anyio's ``send_nowait`` skips waiters with pending
cancellation but our dispatch task (created via
``run_coroutine_threadsafe`` for the reused-session case) has
NO pending cancellation, so the send delivers but the receiver
never wakes (the waiter's Event is set on stale state). Result:
a forever-hung ``response_stream_reader.receive()``. Verified
by reverting the ``asyncio.wait({call_task, fired_task})``
race in ``_dispatch_pool_with_entry`` to a bare ``await
session.call_tool(...)``: the test times out.
This test is the structural gate against the per-dispatch carrier
pattern: it looks right in code review and passes single-dispatch
integration tests, but breaks silently on session reuse and the
SDK-level hang the carrier fix exposes silently strands the
dispatcher even when the carrier is correct.
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — dispatch 1 succeeds
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
# Dispatch 1: passthrough success. Establishes the pooled session.
result1 = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "first"}, user_id="user-1", timeout=15
)
assert "echoed:first" in result1
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
session_after_first = entry.session
assert session_after_first is not None, (
"test setup: dispatch 1 did not populate entry.session; "
"subsequent dispatch will not exercise the reuse path"
)
# Reconfigure upstream to 401 once on the next call. Reset the
# auth-headers log so we can count dispatch-2's POSTs cleanly.
behaviour["post_auth_headers"] = []
behaviour["mode"] = "once_401"
behaviour["_fired"] = False
# Dispatch 2: same (user, server). The hook from dispatch 1's
# connect is still bound to entry.auth_capture. The 401 fires;
# the dispatcher's auth_401 path triggers refresh-and-retry.
result2 = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "second"}, user_id="user-1", timeout=15
)
assert "echoed:second" in result2, (
f"reused-session 401 retry did not succeed. result: {result2!r}. "
"If this is JSON with mcp_consent_required, the dispatcher "
"fell through to consent_required emission; if a generic "
"tool error, the carrier was empty (auth branch unreachable)."
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0, (
"auth failures must not trip the per-server breaker"
)
# Dispatch 2 produces multiple POSTs: the original 401 with the
# rejected bearer, then the retry's full connect handshake
# (initialize + notifications/initialized + tools/list) followed by
# the actual tools/call — all under the refreshed bearer. The retry
# reconnects because the auth_401 handler evicted the broken session.
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, (
f"expected >=2 POSTs after dispatch 2 (401 + retry); "
f"got {len(post_headers)}: {post_headers}"
)
# First POST is the original bearer that got 401'd.
assert post_headers[0] == "Bearer access-aaa", (
f"first POST was {post_headers[0]!r}; expected the original bearer"
)
# Every subsequent POST carries the refreshed bearer (the retry
# ran with force_refresh=True and reconnected with the new token).
refreshed = post_headers[1:]
assert all(h == "Bearer refreshed-bearer" for h in refreshed), (
f"retry POSTs carried unexpected bearer(s); observed: {post_headers}"
)
File diff suppressed because it is too large Load Diff
@@ -1,720 +0,0 @@
"""Phase 7b integration tests — real-transport prompt get 401/403/etc.
Mirror of :mod:`tests.test_mcp_pool_auth_resource_integration` for the
prompt path (RFC §3.3). Drives through the real ``streamablehttp_client``,
real httpx response-hook plumbing, and a real upstream subprocess
(``FastMCP`` with a programmable ``BehaviorMiddleware``). Direct
``httpx.HTTPStatusError`` injection is forbidden (invariant 14).
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import socket
import threading
import time
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
import pytest
import uvicorn
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
class BehaviorMiddleware(BaseHTTPMiddleware):
"""Programmable upstream behaviour — see
:mod:`tests.test_mcp_pool_auth_integration` for the semantics. This
copy serves the prompt integration tests.
"""
def __init__(self, app: Any, behaviour: dict[str, Any]) -> None:
super().__init__(app)
self._behaviour = behaviour
async def dispatch(self, request: Request, call_next: Callable[..., Any]) -> Response:
from starlette.responses import Response as StarletteResponse
if request.method == "POST" and "/mcp" in str(request.url):
self._behaviour.setdefault("post_auth_headers", []).append(
request.headers.get("authorization")
)
mode = self._behaviour.get("mode", "never")
if mode == "once_401":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "always_401":
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "once_403_insufficient":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate",
'Bearer error="insufficient_scope", scope="prompts:read"',
)
},
)
elif mode == "once_403_generic" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get("www_authenticate", "Bearer realm=mcp")
},
)
return await call_next(request)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _build_server(port: int, behaviour: dict[str, Any]) -> uvicorn.Server:
mcp = FastMCP(name="phase7b-prompt-target", streamable_http_path="/mcp")
@mcp.prompt()
def greet(who: str = "world") -> str:
return f"Hello, {who}!"
@mcp.prompt()
def summarize(topic: str = "today") -> str:
return f"Please summarize {topic}."
app = mcp.streamable_http_app()
app.add_middleware(BehaviorMiddleware, behaviour=behaviour)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"upstream at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def upstream():
port = _find_free_port()
behaviour: dict[str, Any] = {}
server = _build_server(port, behaviour)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="phase7b-prompt-upstream")
t.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp", behaviour
finally:
server.should_exit = True
t.join(timeout=5)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
refresh_token: str | None = "refresh-rrr",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token=refresh_token,
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _seed_pool_prompt_map(
mgr: MCPClientManager,
user_id: str,
server_name: str,
prefixed_name: str,
original_name: str,
) -> None:
"""Pre-seed ``_user_prompt_map`` so ``_resolve_pool_target_prompt``
finds the prefixed name. Production wires this through
``_connect_one_pool``; the integration tests seed it directly so the
test focuses on the dispatch behaviour after resolution succeeds.
"""
async def _seed() -> None:
entry = await mgr._ensure_pool_entry((user_id, server_name))
entry.prompts = [
{
"name": prefixed_name,
"original_name": original_name,
"server": server_name,
"description": "",
"arguments": [],
}
]
mgr._rebuild_user_prompt_map(user_id)
assert mgr._loop is not None
asyncio.run_coroutine_threadsafe(_seed(), mgr._loop).result(timeout=5)
# ---------------------------------------------------------------------------
# I-PR-1: 401 → refresh → retry → success (prompt path)
# ---------------------------------------------------------------------------
def test_prompt_get_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Real upstream returns 401 once, then 200. Carrier captures 401,
force_refresh=True mints a new bearer, retry returns the prompt
messages. Hard invariant 3: breaker counter remains 0.
"""
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
messages = mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "everyone"},
user_id="user-1",
timeout=15,
)
assert isinstance(messages, list)
assert len(messages) == 1
assert messages[0]["role"] == "user"
assert "everyone" in messages[0]["content"]
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, f"expected >=2 POSTs; got {len(post_headers)}"
assert post_headers[0] != post_headers[1], (
"retry attached the same bearer as the initial; the dispatcher "
"did not pick up the refreshed token."
)
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
# ---------------------------------------------------------------------------
# I-PR-2: persistent 401 → mcp_consent_required (prompt path) → RuntimeError
# ---------------------------------------------------------------------------
def test_prompt_get_persistent_401_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-PR-3: 403 + insufficient_scope → mcp_insufficient_scope (prompt path)
# ---------------------------------------------------------------------------
def test_prompt_get_403_insufficient_scope_emits_structured_error(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_insufficient"
behaviour["www_authenticate"] = 'Bearer error="insufficient_scope", scope="prompts:read"'
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_insufficient_scope"
assert payload["error"]["scopes_required"] == ["prompts:read"]
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-PR-3b: 403 generic → mcp_prompt_get_forbidden
# ---------------------------------------------------------------------------
def test_prompt_get_403_generic_forbidden(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_generic"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
# Per the kind="prompt" wiring of `_handle_auth_403`, the
# operation-specific code surfaces here rather than the tool path's
# generic mcp_tool_call_forbidden.
assert payload["error"]["code"] == "mcp_prompt_get_forbidden"
assert "scopes_required" not in payload["error"]
# ---------------------------------------------------------------------------
# I-PR-6: breaker isolation — auth failures NEVER trip the breaker
# ---------------------------------------------------------------------------
def test_prompt_get_breaker_unaffected_by_auth_failures(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Repeated 401 + refresh-failed cycles leave breaker at 0
(hard invariant 3 verified end-to-end for the prompt path)."""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
# Re-seed each iteration: symmetric eviction (Phase 7b) clears
# ``_user_prompt_map`` on auth failure so the next dispatch's
# resolver would miss without a fresh seed. Production reconnect
# repopulates this; the test simulates that out-of-band.
for _ in range(10):
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Negative tests — token lookup edge cases (prompt path)
# ---------------------------------------------------------------------------
def test_prompt_get_missing_token_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=10,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_consent_required"
def test_prompt_get_decrypt_failure_emits_token_undecryptable(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=10,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_token_undecryptable_key_unknown"
def test_prompt_get_http_url_emits_url_insecure(
running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""An ``http://`` (non-loopback) oauth_user URL must surface
``mcp_oauth_url_insecure`` BEFORE the bearer is attached.
"""
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://example.com/mcp")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_oauth_url_insecure"
def test_prompt_get_unknown_name_raises_value_error(
running_loop_mgr: Any,
) -> None:
"""When the prefixed name doesn't resolve to either pool or static,
the static-path code raises ``ValueError``. Per-user-first
resolution (scope decision 0.1) means user_id-bearing callers still
hit this path when their pool catalog doesn't carry the name."""
mgr, _loop, _ = running_loop_mgr
with pytest.raises(ValueError, match="Unknown MCP prompt"):
mgr.get_prompt_sync(
"mcp__nonexistent__missing",
None,
user_id="user-1",
timeout=5,
)
# ---------------------------------------------------------------------------
# I-PR-E2E: real discovery + dispatch in same connect (no _seed_pool_prompt_map)
# ---------------------------------------------------------------------------
def test_prompt_get_e2e_discovery_then_dispatch_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Drive REAL discovery + dispatch end-to-end through the pool path.
Mirror of the tool path's
``test_integration_pool_reuse_401_refresh_and_retry_succeeds``: skips
the ``_seed_pool_prompt_map`` shortcut and lets ``_connect_one_pool``
populate ``_user_prompt_map`` from the real ``prompts/list``
upstream response. Verifies that the entry's discovered prompts
match what the FastMCP fixture advertises AND that
``_user_prompt_map[user_id]`` is populated with the prefixed name
after dispatch proving the discovery path actually fired.
This is the structural gate against a regression where prompt
dispatch silently bypasses discovery (e.g., a mis-wired resolver
that finds the (server, original) via prefix-parsing alone never
populates the per-user catalog).
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — discovery + dispatch both succeed
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
# NB: no `_seed_pool_prompt_map` — the resolver finds (server, original)
# via the `mcp__{server}__{prompt}` prefix and hands off to
# ``_dispatch_pool_prompt_sync``, which lazy-connects via
# ``_connect_one_pool``. The connect runs the real ``prompts/list``
# against the FastMCP fixture and populates the per-user catalog.
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
messages = mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
assert isinstance(messages, list)
assert len(messages) == 1
assert messages[0]["role"] == "user"
assert "world" in messages[0]["content"]
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# Discovery populated the entry's prompts with both fixtures
# (``greet`` and ``summarize``) — proves real ``prompts/list``
# ran during the connect, not just the targeted ``prompts/get``.
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
assert entry.prompts is not None
discovered_names = {p["name"] for p in entry.prompts}
assert "mcp__pool-srv__greet" in discovered_names
assert "mcp__pool-srv__summarize" in discovered_names
# ``_rebuild_user_prompt_map`` ran during the connect, populating the
# per-user catalog. This is the signal that discovery wired into the
# routing tables — without it, a follow-up ``get_prompt_sync`` would
# need to re-resolve via prefix parsing every time.
user_prompt_map = mgr._user_prompt_map.get("user-1") or {}
assert "mcp__pool-srv__greet" in user_prompt_map
assert "mcp__pool-srv__summarize" in user_prompt_map
@@ -1,690 +0,0 @@
"""Phase 7b integration tests — real-transport resource read 401/403/etc.
Mirror of :mod:`tests.test_mcp_pool_auth_integration` for the resource
path (RFC §3.2). Drives through the real ``streamablehttp_client``,
real httpx response-hook plumbing, and a real upstream subprocess
(``FastMCP`` with a programmable ``BehaviorMiddleware``). Direct
``httpx.HTTPStatusError`` injection is forbidden (invariant 14).
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import socket
import threading
import time
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
import pytest
import uvicorn
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
class BehaviorMiddleware(BaseHTTPMiddleware):
"""Programmable upstream behaviour — see
:mod:`tests.test_mcp_pool_auth_integration` for the semantics. This
copy serves the resource integration tests.
"""
def __init__(self, app: Any, behaviour: dict[str, Any]) -> None:
super().__init__(app)
self._behaviour = behaviour
async def dispatch(self, request: Request, call_next: Callable[..., Any]) -> Response:
from starlette.responses import Response as StarletteResponse
if request.method == "POST" and "/mcp" in str(request.url):
self._behaviour.setdefault("post_auth_headers", []).append(
request.headers.get("authorization")
)
mode = self._behaviour.get("mode", "never")
if mode == "once_401":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "always_401":
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "once_403_insufficient":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate",
'Bearer error="insufficient_scope", scope="files:read"',
)
},
)
elif mode == "once_403_generic" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get("www_authenticate", "Bearer realm=mcp")
},
)
return await call_next(request)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _build_server(port: int, behaviour: dict[str, Any]) -> uvicorn.Server:
mcp = FastMCP(name="phase7b-resource-target", streamable_http_path="/mcp")
@mcp.resource("res://hello")
def hello() -> str:
return "world"
@mcp.resource("res://json/data")
def jdata() -> str:
return '{"k": 1}'
# Echo tool exists so the e2e test can trigger ``_connect_one_pool``
# (and the full tool + resource + prompt discovery) via prefix-parsed
# ``call_tool_sync`` BEFORE the resource read. The other tests in this
# module use ``_seed_pool_resource_map`` and never invoke tools, so
# adding the tool is invisible to them.
@mcp.tool()
async def echo(payload: str = "default") -> str:
return f"echoed:{payload}"
app = mcp.streamable_http_app()
app.add_middleware(BehaviorMiddleware, behaviour=behaviour)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"upstream at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def upstream():
port = _find_free_port()
behaviour: dict[str, Any] = {}
server = _build_server(port, behaviour)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="phase7b-resource-upstream")
t.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp", behaviour
finally:
server.should_exit = True
t.join(timeout=5)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
refresh_token: str | None = "refresh-rrr",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token=refresh_token,
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _seed_pool_resource_map(
mgr: MCPClientManager, user_id: str, server_name: str, uri: str
) -> None:
"""Pre-seed ``_user_resource_map`` so ``_resolve_pool_target_resource``
finds the URI. Production wires this through ``_connect_one_pool``;
the integration tests seed it directly so the test focuses on the
dispatch behaviour after resolution succeeds.
"""
async def _seed() -> None:
entry = await mgr._ensure_pool_entry((user_id, server_name))
entry.resources = [
{
"uri": uri,
"name": "",
"description": "",
"mimeType": "",
"server": server_name,
}
]
mgr._rebuild_user_resource_map(user_id)
assert mgr._loop is not None
asyncio.run_coroutine_threadsafe(_seed(), mgr._loop).result(timeout=5)
# ---------------------------------------------------------------------------
# I-RP-1: 401 → refresh → retry → success (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Real upstream returns 401 once, then 200. Carrier captures 401,
force_refresh=True mints a new bearer, retry returns the resource.
Hard invariant 3: breaker counter remains 0.
"""
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
result = mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
assert result == "world"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, f"expected >=2 POSTs; got {len(post_headers)}"
assert post_headers[0] != post_headers[1], (
"retry attached the same bearer as the initial; the dispatcher "
"did not pick up the refreshed token."
)
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
# ---------------------------------------------------------------------------
# I-RP-2: persistent 401 → mcp_consent_required (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_persistent_401_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-RP-3: 403 + insufficient_scope → mcp_insufficient_scope (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_403_insufficient_scope_emits_structured_error(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_insufficient"
behaviour["www_authenticate"] = 'Bearer error="insufficient_scope", scope="files:read"'
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_insufficient_scope"
assert payload["error"]["scopes_required"] == ["files:read"]
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-RP-3b: 403 generic → mcp_resource_read_forbidden
# ---------------------------------------------------------------------------
def test_resource_read_403_generic_forbidden(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_generic"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
# Per the kind="resource" wiring of `_handle_auth_403`, the
# operation-specific code surfaces here rather than the tool path's
# generic mcp_tool_call_forbidden.
assert payload["error"]["code"] == "mcp_resource_read_forbidden"
assert "scopes_required" not in payload["error"]
# ---------------------------------------------------------------------------
# I-RP-6: breaker isolation — auth failures NEVER trip the breaker
# ---------------------------------------------------------------------------
def test_resource_read_breaker_unaffected_by_auth_failures(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Repeated 401 + refresh-failed cycles leave breaker at 0
(hard invariant 3 verified end-to-end for the resource path)."""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
# Re-seed each iteration: symmetric eviction (Phase 7b) clears
# ``_user_resource_map`` on auth failure so the next dispatch's
# resolver would miss without a fresh seed. Production reconnect
# repopulates this; the test simulates that out-of-band.
for _ in range(10):
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Negative tests — token lookup edge cases (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_missing_token_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=10)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
def test_resource_read_decrypt_failure_emits_token_undecryptable(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=10)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_token_undecryptable_key_unknown"
def test_resource_read_http_url_emits_url_insecure(
running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""An ``http://`` (non-loopback) oauth_user URL must surface
``mcp_oauth_url_insecure`` BEFORE the bearer is attached.
"""
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://example.com/mcp")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=5)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_oauth_url_insecure"
def test_resource_read_unknown_uri_raises_value_error(
running_loop_mgr: Any,
) -> None:
"""When the URI doesn't resolve to either pool or static, the
static-path code raises ``ValueError``. Per-user-first resolution
(scope decision 0.1) means user_id-bearing callers still hit this
path when their pool catalog doesn't carry the URI."""
mgr, _loop, _ = running_loop_mgr
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("res://nonexistent", user_id="user-1", timeout=5)
# ---------------------------------------------------------------------------
# I-RP-E2E: real discovery + dispatch in same connect (no _seed_pool_resource_map)
# ---------------------------------------------------------------------------
def test_resource_read_e2e_discovery_then_dispatch_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Drive REAL discovery + dispatch end-to-end through the pool path.
Mirror of the tool path's
``test_integration_pool_reuse_401_refresh_and_retry_succeeds``: skips
the ``_seed_pool_resource_map`` shortcut and lets ``_connect_one_pool``
populate ``_user_resource_map`` from the real ``resources/list``
upstream response. Verifies that the entry's discovered resources
match what the FastMCP fixture advertises AND that
``_user_resource_map[user_id]`` is populated with the URI(s) after
discovery proving the discovery path actually fired.
Resource URIs do NOT carry a server-name prefix (unlike tools and
prompts), so the resource resolver cannot derive (server, uri) by
parsing alone. The test triggers the connect via a prefix-parsed
``call_tool_sync`` first (which runs the full
tools+resources+prompts discovery against the FastMCP fixture),
then drives ``read_resource_sync`` against a URI that the
upstream advertised proving that real discovery wired the URI
into the per-user catalog.
Structural gate against a regression where resource discovery is
silently skipped (e.g., a capability-gating bug that drops the
``resources/list`` call but keeps the connect succeeding).
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — discovery + dispatch both succeed
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
# NB: no `_seed_pool_resource_map` — the connect runs the real
# ``resources/list`` against the FastMCP fixture and populates the
# per-user catalog. The tool call below triggers that connect because
# ``_resolve_pool_target`` derives (server, original) from the
# ``mcp__pool-srv__echo`` prefix and lazy-connects via
# ``_connect_one_pool``.
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
# Step 1: trigger the connect via prefix-parsed tool dispatch.
# Discovery (tools + resources + prompts) populates the per-user
# catalogs.
tool_result = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "ignite"}, user_id="user-1", timeout=15
)
assert "echoed:ignite" in tool_result
# Step 2: now that discovery has populated ``_user_resource_map``,
# the resource resolver finds ``res://hello`` and dispatches the
# read on the SAME pool entry / session.
result = mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
assert result == "world"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# Discovery populated the entry's resources with both fixtures
# (``res://hello`` and ``res://json/data``) — proves real
# ``resources/list`` ran during the connect, not just the targeted
# ``resources/read``.
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
assert entry.resources is not None
discovered_uris = {r["uri"] for r in entry.resources if not r.get("template")}
assert "res://hello" in discovered_uris
assert "res://json/data" in discovered_uris
# ``_rebuild_user_resource_map`` ran during the connect, populating
# the per-user catalog. This is the signal that discovery wired into
# the routing tables — without it, ``read_resource_sync`` would have
# raised ValueError because the resolver had no entry for the URI.
user_resource_map = mgr._user_resource_map.get("user-1") or {}
assert "res://hello" in user_resource_map
assert "res://json/data" in user_resource_map
-286
View File
@@ -1,286 +0,0 @@
"""Tests for ``MCPTokenStore`` ciphertext-aware CRUD.
Phase 3 of the OAuth-MCP RFC: validates the encrypt/decrypt boundary
between :class:`MCPTokenStore` and the storage protocol's ciphertext-only
columns. Exercises the row-not-deleted-on-decrypt-failure invariant.
"""
from __future__ import annotations
import base64
import pytest
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenDecryptError,
MCPTokenStore,
)
def _make_cipher() -> MCPTokenCipher:
raw = base64.urlsafe_b64decode(Fernet.generate_key())
return MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,)))
def _make_store(backend, *, audit: bool = False) -> tuple[MCPTokenStore, MCPTokenCipher]:
cipher = _make_cipher()
store = MCPTokenStore(
backend,
cipher,
node_id="test-node",
audit_storage=backend if audit else None,
)
return store, cipher
def _seed_server(backend, *, server_id: str = "srv-id-1", name: str = "srv-a") -> str:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
)
return server_id
# ---------------------------------------------------------------------------
# User-token CRUD
# ---------------------------------------------------------------------------
class TestUserTokenCRUD:
def test_create_and_get_round_trip(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="access-aaa",
refresh_token="refresh-bbb",
expires_at="2026-05-04T12:00:00",
scopes="openid profile",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
plain = store.get_user_token("u1", "srv-a")
assert plain is not None
assert plain["user_id"] == "u1"
assert plain["server_name"] == "srv-a"
assert plain["access_token"] == "access-aaa"
assert plain["refresh_token"] == "refresh-bbb"
assert plain["scopes"] == "openid profile"
assert plain["audience"] == "https://mcp.example.com"
def test_create_with_no_refresh_token(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="access-only",
refresh_token=None,
expires_at=None,
scopes=None,
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
plain = store.get_user_token("u1", "srv-a")
assert plain is not None
assert plain["access_token"] == "access-only"
assert plain["refresh_token"] is None
def test_get_missing_returns_none(self, backend) -> None:
store, _ = _make_store(backend)
assert store.get_user_token("nobody", "srv-a") is None
def test_update_after_refresh(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="old-access",
refresh_token="old-refresh",
expires_at="2026-05-04T12:00:00",
scopes="openid",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
ok = store.update_user_token_after_refresh(
"u1",
"srv-a",
access_token="new-access",
refresh_token="new-refresh",
expires_at="2026-05-04T13:00:00",
)
assert ok is True
plain = store.get_user_token("u1", "srv-a")
assert plain is not None
assert plain["access_token"] == "new-access"
assert plain["refresh_token"] == "new-refresh"
assert plain["expires_at"] == "2026-05-04T13:00:00"
# Preserved columns:
assert plain["scopes"] == "openid"
assert plain["as_issuer"] == "https://auth.example.com"
# last_refreshed got stamped:
assert plain["last_refreshed"] is not None
def test_update_after_refresh_missing_row_returns_false(self, backend) -> None:
store, _ = _make_store(backend)
ok = store.update_user_token_after_refresh(
"u1",
"srv-a",
access_token="x",
refresh_token=None,
expires_at=None,
)
assert ok is False
def test_delete(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="a",
refresh_token=None,
expires_at=None,
scopes=None,
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
assert store.delete_user_token("u1", "srv-a") is True
assert store.get_user_token("u1", "srv-a") is None
# Idempotent: deleting again returns False.
assert store.delete_user_token("u1", "srv-a") is False
# ---------------------------------------------------------------------------
# Client-secret writer
# ---------------------------------------------------------------------------
class TestClientSecretWriter:
def test_set_oauth_client_secret_round_trip(self, backend) -> None:
store, cipher = _make_store(backend)
server_id = _seed_server(backend)
ok = store.set_oauth_client_secret(server_id, "plaintext-secret")
assert ok is True
# Read raw via get_mcp_server: ciphertext != plaintext, decrypts back.
raw = backend.get_mcp_server(server_id)
assert raw is not None
ct = raw["oauth_client_secret_ct"]
assert isinstance(ct, (bytes, bytearray, memoryview))
ct_bytes = bytes(ct)
assert ct_bytes != b"plaintext-secret"
assert cipher.decrypt(ct_bytes) == b"plaintext-secret"
def test_set_oauth_client_secret_clear_with_none(self, backend) -> None:
store, _ = _make_store(backend)
server_id = _seed_server(backend)
store.set_oauth_client_secret(server_id, "x")
assert store.set_oauth_client_secret(server_id, None) is True
raw = backend.get_mcp_server(server_id)
assert raw is not None
assert raw["oauth_client_secret_ct"] is None
def test_set_oauth_client_secret_missing_server_returns_false(self, backend) -> None:
store, _ = _make_store(backend)
ok = store.set_oauth_client_secret("does-not-exist", "x")
assert ok is False
# ---------------------------------------------------------------------------
# Decrypt failure: row preservation invariant
# ---------------------------------------------------------------------------
class TestDecryptFailureInvariant:
def test_get_user_token_with_wrong_key_raises_decrypt_error(self, backend) -> None:
"""CRITICAL: when no installed key can decrypt a stored row,
``get_user_token`` MUST NOT auto-delete the row. The row is
still valid; this node just doesn't have the right key.
"""
# Write under cipher A.
store_a, _cipher_a = _make_store(backend)
store_a.create_user_token(
"u1",
"srv-a",
access_token="secret-access",
refresh_token="secret-refresh",
expires_at="2026-05-04T12:00:00",
scopes="openid",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
raw_before = backend.get_mcp_user_token("u1", "srv-a")
assert raw_before is not None
ct_before = bytes(raw_before["access_token_ct"])
# Read under cipher B (different key).
store_b, cipher_b = _make_store(backend)
with pytest.raises(MCPTokenDecryptError) as exc_info:
store_b.get_user_token("u1", "srv-a")
# The exception carries the keys we tried — useful for audit.
assert exc_info.value.key_fingerprints_attempted == cipher_b.key_fingerprints
# Row MUST still exist with ciphertext intact.
raw_after = backend.get_mcp_user_token("u1", "srv-a")
assert raw_after is not None
assert bytes(raw_after["access_token_ct"]) == ct_before
def test_decrypt_failure_emits_audit_when_configured(self, backend) -> None:
"""When ``audit_storage`` is set, decrypt failures emit a
``mcp_server.oauth.token_decrypt_failure`` audit event."""
store_a, _ = _make_store(backend)
store_a.create_user_token(
"u1",
"srv-a",
access_token="x",
refresh_token=None,
expires_at=None,
scopes=None,
as_issuer="https://a",
audience="https://m",
)
store_b, cipher_b = _make_store(backend, audit=True)
with pytest.raises(MCPTokenDecryptError):
store_b.get_user_token("u1", "srv-a")
events = backend.list_audit_events(limit=10)
actions = {ev.get("action") for ev in events}
assert "mcp_server.oauth.token_decrypt_failure" in actions
# ---------------------------------------------------------------------------
# Client-secret reader — q-9
# ---------------------------------------------------------------------------
class TestClientSecretReader:
def test_get_oauth_client_secret_returns_none_when_row_absent(self, backend) -> None:
store, _ = _make_store(backend)
assert store.get_oauth_client_secret("does-not-exist") is None
def test_get_oauth_client_secret_returns_none_when_column_null(self, backend) -> None:
store, _ = _make_store(backend)
server_id = _seed_server(backend)
# No set_oauth_client_secret call — column stays NULL.
assert store.get_oauth_client_secret(server_id) is None
def test_get_oauth_client_secret_round_trip(self, backend) -> None:
store, _ = _make_store(backend)
server_id = _seed_server(backend)
store.set_oauth_client_secret(server_id, "shhh-its-secret")
assert store.get_oauth_client_secret(server_id) == "shhh-its-secret"
def test_get_oauth_client_secret_raises_on_key_mismatch(self, backend) -> None:
store_a, _ = _make_store(backend)
server_id = _seed_server(backend)
store_a.set_oauth_client_secret(server_id, "secret-under-key-a")
# Cipher B has a different key — decrypt fails loudly.
store_b, _ = _make_store(backend)
with pytest.raises(MCPTokenDecryptError):
store_b.get_oauth_client_secret(server_id)
-107
View File
@@ -1,107 +0,0 @@
"""Tests for ``MCPTokenStore.list_user_token_metadata``.
Validates the non-secret projection used by the settings UI: ciphertext
columns are stripped, ordering is preserved, and the empty case returns
``[]``. Decrypt is intentionally skipped the list view must never need
the access/refresh secrets.
"""
from __future__ import annotations
import base64
import sqlalchemy as sa
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenStore,
)
def _make_cipher() -> MCPTokenCipher:
raw = base64.urlsafe_b64decode(Fernet.generate_key())
return MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,)))
def _make_store(backend) -> MCPTokenStore:
return MCPTokenStore(backend, _make_cipher(), node_id="test-node")
def _seed_token(
store: MCPTokenStore,
backend,
*,
user_id: str,
server_name: str,
created: str,
) -> None:
"""Create a token via the store and backdate ``created`` for ordering."""
store.create_user_token(
user_id,
server_name,
access_token="access-secret",
refresh_token="refresh-secret",
expires_at="2026-05-04T12:00:00",
scopes="openid profile",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_user_tokens SET created = :created "
"WHERE user_id = :uid AND server_name = :sn"
),
{"created": created, "uid": user_id, "sn": server_name},
)
conn.commit()
class TestListUserTokenMetadata:
def test_list_user_token_metadata_returns_non_secret_fields_only(self, backend) -> None:
store = _make_store(backend)
_seed_token(
store, backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
rows = store.list_user_token_metadata("u1")
assert len(rows) == 1
meta = rows[0]
# Secrets MUST be absent.
assert "access_token" not in meta
assert "refresh_token" not in meta
assert "access_token_ct" not in meta
assert "refresh_token_ct" not in meta
# Non-secret columns surface verbatim.
assert meta["user_id"] == "u1"
assert meta["server_name"] == "srv-a"
assert meta["scopes"] == "openid profile"
assert meta["as_issuer"] == "https://auth.example.com"
assert meta["audience"] == "https://mcp.example.com"
assert meta["expires_at"] == "2026-05-04T12:00:00"
assert meta["created"] == "2026-05-01T00:00:00"
assert meta["last_refreshed"] is None
def test_list_user_token_metadata_empty(self, backend) -> None:
store = _make_store(backend)
assert store.list_user_token_metadata("nobody") == []
def test_list_user_token_metadata_preserves_creation_order(self, backend) -> None:
store = _make_store(backend)
_seed_token(
store, backend, user_id="u1", server_name="srv-c", created="2026-05-03T00:00:00"
)
_seed_token(
store, backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
_seed_token(
store, backend, user_id="u1", server_name="srv-b", created="2026-05-02T00:00:00"
)
rows = store.list_user_token_metadata("u1")
assert [r["server_name"] for r in rows] == ["srv-a", "srv-b", "srv-c"]
assert [r["created"] for r in rows] == [
"2026-05-01T00:00:00",
"2026-05-02T00:00:00",
"2026-05-03T00:00:00",
]
File diff suppressed because it is too large Load Diff
-972
View File
@@ -1,972 +0,0 @@
"""Tests for the per-(user, server) MCP session pool.
Covers Phase 5 of the OAuth-MCP rollout: pool data structures,
``_ensure_pool_entry`` lazy allocation, ``_connect_one_pool`` plumbing,
the dispatch state machine in ``_dispatch_pool``, idle / LRU eviction,
failure classification, and ``user_id`` thread-through.
The static path (``auth_type {none, static}``) MUST stay
byte-identical see ``test_mcp_client.py``'s
``test_reconnect_preserves_static_state_identity``.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import threading
import time
from contextlib import AsyncExitStack
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager, PoolEntryState
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Fixtures and helpers
# ---------------------------------------------------------------------------
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
"""A fresh SQLite backend per test (not the shared singleton)."""
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token="refresh-rrr",
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
"""Background-loop fixture matching the static-path test convention.
Tests that need a wired-up app_state assign it via ``mgr.set_app_state``.
"""
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
# Drain the eviction task before stopping the loop so its log/stream
# handlers don't fire after pytest has torn its handlers down. Mirrors
# the production ``shutdown()`` shape.
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _run_on_loop(loop: asyncio.AbstractEventLoop, coro: Any) -> Any:
"""Submit *coro* to *loop*, wait for the result with a 5s timeout."""
fut = asyncio.run_coroutine_threadsafe(coro, loop)
return fut.result(timeout=5)
# ---------------------------------------------------------------------------
# Pool data structures
# ---------------------------------------------------------------------------
class TestPoolDataStructures:
"""``_user_pool_entries``, ``_user_pool_locks``, eviction-task state."""
def test_pool_state_starts_empty(self) -> None:
mgr = MCPClientManager({})
assert mgr._user_pool_entries == {}
assert mgr._user_pool_last_used == {}
assert mgr._user_pool_locks == {}
assert mgr._user_pool_eviction_task is None
def test_set_app_state_persists(self) -> None:
mgr = MCPClientManager({})
sentinel = SimpleNamespace(token_store=object())
mgr.set_app_state(sentinel)
assert mgr._app_state is sentinel
def test_ensure_pool_entry_allocates_lock_on_loop(self, running_loop_mgr) -> None:
"""``asyncio.Lock`` MUST be created on the mcp-loop (RFC §2.0 #2)."""
mgr, loop, _thread = running_loop_mgr
key = ("user-A", "pool-srv")
entry = _run_on_loop(loop, mgr._ensure_pool_entry(key))
assert isinstance(entry, PoolEntryState)
assert entry.key == key
assert isinstance(entry.open_lock, asyncio.Lock)
# Calling again returns the same entry / lock object.
entry2 = _run_on_loop(loop, mgr._ensure_pool_entry(key))
assert entry2 is entry
assert entry2.open_lock is entry.open_lock
# ---------------------------------------------------------------------------
# Lazy connect (`_connect_one_pool`)
# ---------------------------------------------------------------------------
class _AsyncCM:
"""Awaitable async context manager that returns ``value`` from __aenter__."""
def __init__(self, value: Any) -> None:
self._value = value
async def __aenter__(self) -> Any:
return self._value
async def __aexit__(self, *exc: Any) -> bool:
return False
class TestLazyConnect:
def test_connect_pool_injects_authorization_header(self, running_loop_mgr) -> None:
from unittest.mock import patch
mgr, loop, _ = running_loop_mgr
observed_kwargs: dict[str, Any] = {}
async def _probe(*_args: Any, **_kwargs: Any) -> None:
return None
fake_session = MagicMock()
fake_session.initialize = AsyncMock(return_value=None)
# Phase 7b: ``_connect_one_pool`` discovers tools, resources,
# and prompts after ``initialize()`` returns (resources/prompts
# capability-gated). The capability stub returns a tools-only
# advertisement so the test can keep its narrow focus on the
# bearer-injection contract; resources/prompts paths are
# exercised by the real-transport tests in
# ``tests/test_mcp_user_catalog.py``.
fake_caps = MagicMock()
fake_caps.resources = None
fake_caps.prompts = None
fake_session.get_server_capabilities = MagicMock(return_value=fake_caps)
fake_session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
def _stream_factory(*, url: str, headers: dict[str, str]) -> _AsyncCM:
observed_kwargs["url"] = url
observed_kwargs["headers"] = dict(headers)
return _AsyncCM((AsyncMock(), AsyncMock(), lambda: None))
with (
patch("turnstone.core.mcp_client.streamablehttp_client", side_effect=_stream_factory),
patch.object(mgr, "_tcp_probe", side_effect=_probe),
patch("turnstone.core.mcp_client.ClientSession", return_value=_AsyncCM(fake_session)),
):
cfg = {
"type": "streamable-http",
"url": "https://mcp.example.com/sse",
"headers": {},
}
entry = _run_on_loop(
loop,
mgr._connect_one_pool(("user-1", "pool-srv"), cfg, "access-aaa"),
)
assert entry.session is fake_session
assert observed_kwargs["headers"]["Authorization"] == "Bearer access-aaa"
def test_connect_pool_rejects_non_http_transport(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
cfg = {"type": "stdio", "command": "echo"}
with pytest.raises(RuntimeError, match="streamable-http"):
_run_on_loop(
loop,
mgr._connect_one_pool(("user-1", "pool-srv"), cfg, "access-aaa"),
)
def test_pool_path_does_not_touch_static_servers(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
# Pre-seed a static-path entry so accidental writes are observable.
from turnstone.core.mcp_client import StaticServerState
sentinel = StaticServerState(name="static-srv", session=MagicMock())
mgr._static_servers["static-srv"] = sentinel
async def _seed_pool() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = MagicMock()
entry.last_used = time.monotonic()
_run_on_loop(loop, _seed_pool())
# Pool side has its own state; the static dict is untouched.
assert mgr._static_servers["static-srv"] is sentinel
assert mgr._user_pool_entries[("user-1", "pool-srv")].session is not None
# ---------------------------------------------------------------------------
# Eviction
# ---------------------------------------------------------------------------
class TestEviction:
def test_idle_eviction_closes_stale_entries(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0 # everything is stale
async def _seed() -> list[PoolEntryState]:
entries = []
for i in range(3):
entry = await mgr._ensure_pool_entry((f"u{i}", "pool-srv"))
entry.session = MagicMock()
entries.append(entry)
return entries
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
assert mgr._user_pool_entries == {}
def test_eviction_skips_locked_entries(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0
async def _seed_and_lock() -> tuple[asyncio.Lock, asyncio.Event]:
entry = await mgr._ensure_pool_entry(("u-busy", "pool-srv"))
entry.session = MagicMock()
held = asyncio.Event()
async def _hold() -> None:
async with entry.open_lock:
held.set()
await asyncio.sleep(0.5)
asyncio.create_task(_hold())
await held.wait()
return entry.open_lock, held
_run_on_loop(loop, _seed_and_lock())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
# Entry survives because eviction skipped the locked key.
assert ("u-busy", "pool-srv") in mgr._user_pool_entries
def test_lru_cap_evicts_oldest(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 999_999.0 # TTL effectively disabled
mgr._user_pool_lru_max = 2
async def _seed() -> None:
base = time.monotonic()
for i in range(5):
key = (f"u{i}", "pool-srv")
entry = await mgr._ensure_pool_entry(key)
entry.session = MagicMock()
# Recent timestamps so TTL doesn't fire — only LRU should.
entry.last_used = base + i
mgr._user_pool_last_used[key] = base + i
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
assert len(mgr._user_pool_entries) <= 2
# The two newest survive (u3, u4).
assert ("u4", "pool-srv") in mgr._user_pool_entries
assert ("u3", "pool-srv") in mgr._user_pool_entries
def test_eviction_resilient_to_close_errors(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0
broken_stack = MagicMock(spec=AsyncExitStack)
broken_stack.aclose = AsyncMock(side_effect=RuntimeError("close failed"))
async def _seed() -> None:
for i in range(2):
entry = await mgr._ensure_pool_entry((f"u{i}", "pool-srv"))
entry.session = MagicMock()
entry.stack = broken_stack
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
# Eviction must not raise even if close fails.
_run_on_loop(loop, _evict())
# All entries removed from the dict regardless.
assert mgr._user_pool_entries == {}
# ---------------------------------------------------------------------------
# Dispatch state machine
# ---------------------------------------------------------------------------
class TestDispatchStateMachine:
"""One row per state in the §1.5 / RFC §6 state machine."""
def _wire_pool(
self, mgr: MCPClientManager, storage: SQLiteBackend, cipher: Any
) -> SimpleNamespace:
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
return state
def test_no_token_emits_consent_required(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
self._wire_pool(mgr, storage, cipher)
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
def test_decrypt_failure_does_not_emit_consent(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
from turnstone.core.mcp_crypto import MCPTokenDecryptError
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher)
state = self._wire_pool(mgr, storage, cipher)
def _raise(*args, **kwargs):
raise MCPTokenDecryptError(
"no installed key can decrypt",
key_fingerprints_attempted=("aabbccdd",),
)
state.mcp_token_store.get_user_token = _raise
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_token_undecryptable_key_unknown"
# Operator fingerprints stay server-side (audit log + structured log);
# the agent-facing payload must NOT carry them onward to the LLM
# provider.
assert "key_fingerprints_attempted" not in payload["error"]
def test_refresh_failure_emits_consent(self, running_loop_mgr, storage: SQLiteBackend) -> None:
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
# Seed an expired token with no refresh — the classified getter
# treats this as "refresh_failed" (deletes the row, returns the
# tagged result).
_seed_user_token(storage, cipher, expires_in_seconds=-1000)
state = self._wire_pool(mgr, storage, cipher)
# Drop the refresh token to force the no-refresh-token branch.
state.mcp_token_store.delete_user_token("user-1", "pool-srv")
state.mcp_token_store.create_user_token(
"user-1",
"pool-srv",
access_token="access-aaa",
refresh_token=None,
expires_at=(datetime.now(UTC) - timedelta(seconds=1000)).strftime("%Y-%m-%dT%H:%M:%S"),
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
def test_token_present_dispatches_to_session(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher, expires_in_seconds=3600)
self._wire_pool(mgr, storage, cipher)
# Pre-seed a connected pool entry so dispatch never touches the
# SDK or the network.
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "tool-result"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
async def _seed_entry() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = fake_session
_run_on_loop(loop, _seed_entry())
result = mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{"q": "hi"},
user_id="user-1",
timeout=5,
)
assert result == "tool-result"
# ---------------------------------------------------------------------------
# Failure classification
# ---------------------------------------------------------------------------
class TestClassifyFailure:
def test_transport_failure_classified_as_transport(self) -> None:
mgr = MCPClientManager({})
for exc in (
BrokenPipeError(),
ConnectionResetError(),
EOFError(),
TimeoutError("net"),
):
assert mgr._classify_failure(exc) == "transport"
def test_protocol_error_classified_as_protocol(self) -> None:
from mcp import McpError
from mcp.types import ErrorData
mgr = MCPClientManager({})
err = McpError(ErrorData(code=-32600, message="bad request"))
assert mgr._classify_failure(err) == "protocol"
def test_other_classified_as_other(self) -> None:
mgr = MCPClientManager({})
assert mgr._classify_failure(ValueError("nope")) == "other"
def test_http_401_classified_as_auth_401(self) -> None:
"""Defense-in-depth: ``HTTPStatusError`` classification still works
even though Phase 6 normally consults the carrier instead.
Phase 6 split ``"auth"`` into ``"auth_401"`` / ``"auth_403"``
so the dispatcher can refresh-and-retry only on 401.
"""
import httpx
mgr = MCPClientManager({})
req = httpx.Request("POST", "https://mcp.example.com/sse")
resp = httpx.Response(401, request=req)
exc = httpx.HTTPStatusError("unauthorized", request=req, response=resp)
assert mgr._classify_failure(exc) == "auth_401"
def test_http_403_classified_as_auth_403(self) -> None:
import httpx
mgr = MCPClientManager({})
req = httpx.Request("POST", "https://mcp.example.com/sse")
resp = httpx.Response(403, request=req)
exc = httpx.HTTPStatusError("forbidden", request=req, response=resp)
assert mgr._classify_failure(exc) == "auth_403"
def test_http_500_not_classified_as_auth(self) -> None:
import httpx
mgr = MCPClientManager({})
req = httpx.Request("POST", "https://mcp.example.com/sse")
resp = httpx.Response(500, request=req)
exc = httpx.HTTPStatusError("server", request=req, response=resp)
# 5xx is not auth — falls through to "other".
assert mgr._classify_failure(exc) == "other"
# ---------------------------------------------------------------------------
# Wired-failure paths in _dispatch_pool
# ---------------------------------------------------------------------------
class TestDispatchFailureWiring:
"""``_classify_failure`` is consulted in production, not just tests."""
def _wire_pool(
self, mgr: MCPClientManager, storage: SQLiteBackend, cipher: Any
) -> SimpleNamespace:
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
return state
def _seed_connected_session(
self, mgr: MCPClientManager, loop: asyncio.AbstractEventLoop, exc: BaseException
) -> None:
async def _seed() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
sess = MagicMock()
async def _raise(*_args: Any, **_kwargs: Any) -> Any:
raise exc
sess.call_tool = _raise
entry.session = sess
_run_on_loop(loop, _seed())
def test_dispatch_pool_transport_failure_trips_breaker(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher)
self._wire_pool(mgr, storage, cipher)
self._seed_connected_session(mgr, loop, BrokenPipeError("dead"))
with pytest.raises(BrokenPipeError):
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
# Transport failure ticks the breaker.
assert mgr._consecutive_failures.get("pool-srv", 0) == 1
# ---------------------------------------------------------------------------
# HTTPS enforcement (sec-1)
# ---------------------------------------------------------------------------
class TestHttpsEnforcement:
def test_pool_rejects_http_url_for_oauth_user(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://insecure.example.com/sse")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_oauth_url_insecure"
assert payload["error"]["server"] == "pool-srv"
def test_pool_accepts_loopback_http(self, running_loop_mgr, storage: SQLiteBackend) -> None:
"""``http://127.0.0.1`` and ``http://localhost`` should not be blocked."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://127.0.0.1:8000/sse")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
# Pre-seed a connected pool entry so dispatch succeeds without
# touching the network.
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "ok"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
async def _seed_entry() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = fake_session
_run_on_loop(loop, _seed_entry())
result = mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
# Loopback URL not rejected — dispatch reaches the (fake) session.
assert result == "ok"
def test_validate_oauth_user_url_helper(self) -> None:
from turnstone.core.mcp_client import _validate_oauth_user_url
# Acceptable: https + the exact loopback hostnames.
_validate_oauth_user_url("https://mcp.example.com/sse")
_validate_oauth_user_url("http://localhost/sse")
_validate_oauth_user_url("http://127.0.0.1:9000/sse")
_validate_oauth_user_url("http://[::1]/sse")
# Rejected: non-https + non-loopback. The ``*.localhost`` suffix
# bypass is intentionally NOT honored (RFC 6761 localhost-zone
# resolution is configuration-dependent — custom resolvers,
# /etc/hosts, Docker overlays may map ``foo.localhost`` to
# non-loopback IPs).
for bad in (
"http://mcp.example.com/sse",
"http://app.localhost/sse",
"ws://mcp.example.com/sse",
"ftp://mcp.example.com/sse",
"//mcp.example.com/sse",
):
with pytest.raises(ValueError, match="https://"):
_validate_oauth_user_url(bad)
# ---------------------------------------------------------------------------
# _resolve_pool_target parser (q-9)
# ---------------------------------------------------------------------------
class TestResolvePoolTarget:
def _make_mgr_with_oauth_server(
self, storage: SQLiteBackend, *, name: str = "pool-srv"
) -> MCPClientManager:
_seed_oauth_server(storage, name=name)
mgr = MCPClientManager({})
mgr.set_storage(storage)
return mgr
def test_malformed_prefix(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
# Wrong prefix.
assert mgr._resolve_pool_target("xyz__pool-srv__t", None, None) is None
def test_too_few_separators(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
# mcp__server with no original_name segment.
assert mgr._resolve_pool_target("mcp__pool-srv", None, None) is None
def test_empty_server_segment(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
# mcp____tool — server segment is empty.
assert mgr._resolve_pool_target("mcp____tool", None, None) is None
def test_original_with_double_underscore_round_trips(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
target = mgr._resolve_pool_target("mcp__pool-srv__do__thing", None, None)
assert target is not None
assert target[0] == "pool-srv"
# Original-name keeps its embedded ``__``.
assert target[1] == "do__thing"
# ---------------------------------------------------------------------------
# LRU + lock interlock (q-7)
# ---------------------------------------------------------------------------
class TestLruInterlock:
def test_lru_cap_skips_locked_oldest(self, running_loop_mgr) -> None:
"""LRU eviction must skip a locked entry the same way TTL does."""
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 999_999.0 # disable TTL
mgr._user_pool_lru_max = 2
async def _seed_and_lock_oldest() -> tuple[asyncio.Lock, asyncio.Event]:
base = time.monotonic()
for i in range(3):
key = (f"u{i}", "pool-srv")
entry = await mgr._ensure_pool_entry(key)
entry.session = MagicMock()
# Older index ⇒ older timestamp.
entry.last_used = base + i
mgr._user_pool_last_used[key] = base + i
# Lock the oldest (u0) so eviction must skip it and pick a younger one.
oldest = mgr._user_pool_entries[("u0", "pool-srv")]
held = asyncio.Event()
async def _hold() -> None:
async with oldest.open_lock:
held.set()
await asyncio.sleep(0.5)
asyncio.create_task(_hold())
await held.wait()
return oldest.open_lock, held
_run_on_loop(loop, _seed_and_lock_oldest())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
# Locked u0 must survive.
assert ("u0", "pool-srv") in mgr._user_pool_entries
# The oldest unlocked entry (u1) was evicted to bring count down to cap.
assert ("u1", "pool-srv") not in mgr._user_pool_entries
# ---------------------------------------------------------------------------
# Concurrent dispatch on shared session (M4 / perf-1)
# ---------------------------------------------------------------------------
class TestConcurrentDispatch:
def test_pool_concurrent_dispatch_to_same_user_server_is_serialized(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
"""Phase 6: two tool calls on the SAME (user, server) MUST serialize
on ``open_lock`` so the auth-introspection carrier never crosses
between concurrent dispatches.
Phase 5 perf-1 released ``open_lock`` before ``call_tool`` so two
concurrent same-key calls multiplexed on a shared
``ClientSession``. Phase 6 reverts that for the auth-aware path
because the per-dispatch ``_AuthCapture`` is keyed off the
``httpx.AsyncClient`` event hook releasing the lock would let
a concurrent dispatch overwrite the carrier mid-flight,
attributing one caller's 401 to another (a security bug).
Verified by reverting ``_dispatch_pool_with_entry`` to the
Phase 5 shape (release ``open_lock`` before ``call_tool``
i.e. move the ``in_flight += 1`` / ``call_tool`` / decrement
block out of the ``async with`` body) and confirming this test
observes ``max_concurrency == 2``.
"""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
observed_max_concurrency = 0
in_flight = 0
in_flight_lock = threading.Lock()
async def _call_tool(name, args):
nonlocal observed_max_concurrency, in_flight
with in_flight_lock:
in_flight += 1
observed_max_concurrency = max(observed_max_concurrency, in_flight)
try:
# Hold a moment so concurrent calls would overlap if
# they weren't serialized on ``open_lock``.
await asyncio.sleep(0.1)
content = MagicMock()
content.text = "ok"
res = MagicMock()
res.content = [content]
res.isError = False
return res
finally:
with in_flight_lock:
in_flight -= 1
fake_session = MagicMock()
fake_session.call_tool = _call_tool
async def _seed_entry() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = fake_session
_run_on_loop(loop, _seed_entry())
results: list[str] = []
errors: list[Exception] = []
def _dispatch() -> None:
try:
results.append(
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
)
except Exception as exc: # pragma: no cover — diagnostic only
errors.append(exc)
t1 = threading.Thread(target=_dispatch)
t2 = threading.Thread(target=_dispatch)
t1.start()
t2.start()
t1.join(timeout=5)
t2.join(timeout=5)
assert errors == []
assert results == ["ok", "ok"]
# ``open_lock`` held across ``call_tool`` — the second dispatch
# waits for the first to release before entering call_tool.
assert observed_max_concurrency == 1
# ---------------------------------------------------------------------------
# user_id thread-through (signature)
# ---------------------------------------------------------------------------
class TestUserIdThreadThrough:
def test_default_user_id_takes_static_path(self, running_loop_mgr) -> None:
"""``user_id=None`` must leave the static-path call byte-identical."""
mgr, _loop, _ = running_loop_mgr
# Static-path tool registered the standard way.
mgr._tool_map["mcp__static__t"] = ("static-srv", "t")
from turnstone.core.mcp_client import StaticServerState
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "static-output"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
mgr._static_servers["static-srv"] = StaticServerState(
name="static-srv", session=fake_session
)
# No user_id, no app_state — pool branch is skipped entirely.
result = mgr.call_tool_sync("mcp__static__t", {"q": "hi"}, user_id=None, timeout=5)
assert result == "static-output"
def test_user_id_with_static_path_does_not_use_pool(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
"""Caller passes user_id but the resolved server is static — pool
branch must not run because ``_lookup_server_row`` reports
``auth_type != 'oauth_user'``."""
mgr, _loop, _ = running_loop_mgr
storage.create_mcp_server(
server_id="srv-static",
name="static-srv",
transport="stdio",
url="",
command="echo",
auth_type="static",
)
mgr.set_storage(storage)
mgr.set_app_state(SimpleNamespace())
mgr._tool_map["mcp__static-srv__t"] = ("static-srv", "t")
from turnstone.core.mcp_client import StaticServerState
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "static-output"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
mgr._static_servers["static-srv"] = StaticServerState(
name="static-srv", session=fake_session
)
result = mgr.call_tool_sync(
"mcp__static-srv__t",
{"q": "hi"},
user_id="user-1",
timeout=5,
)
assert result == "static-output"
# No pool entries were created.
assert mgr._user_pool_entries == {}
-209
View File
@@ -4,8 +4,6 @@ from turnstone.core.metacognition import (
NUDGE_COMPLETION,
NUDGE_CORRECTION,
NUDGE_DENIAL,
NUDGE_IDLE_CHILDREN_DISPLAY_CAP,
NUDGE_IDLE_CHILDREN_WAIT_CAP,
NUDGE_REPEAT,
NUDGE_RESUME,
NUDGE_START,
@@ -13,7 +11,6 @@ from turnstone.core.metacognition import (
RepeatDetector,
detect_completion,
detect_correction,
format_idle_children_nudge,
format_nudge,
should_nudge,
)
@@ -379,209 +376,3 @@ class TestRepeatDetector:
def test_threshold_one_fires_immediately(self):
det = RepeatDetector(threshold=1)
assert det.record("a") is True
class TestFormatIdleChildrenNudge:
"""``format_idle_children_nudge`` renders the wake-driven idle_children
body no ``<system-reminder>`` envelope (the side-channel splice
wraps it at the wire boundary).
"""
def test_empty_list_returns_empty_string(self):
# Caller short-circuits on `if not text: return` — so empty
# input MUST produce empty output, not a header-only stub.
assert format_idle_children_nudge([]) == ""
def test_single_child_renders(self):
children = [{"ws_id": "ws-abc12345", "name": "research-task", "state": "running"}]
text = format_idle_children_nudge(children)
assert "ws-abc12" in text # short-id form (8 chars)
assert "research-task" in text
assert "running" in text
assert "wait_for_workstream" in text
assert "ws-abc12345" in text # full id appears in the suggestion's ws_ids list
def test_under_display_cap_no_overflow_line(self):
children = [
{"ws_id": f"ws-{i:08d}", "name": f"task-{i}", "state": "running"} for i in range(3)
]
text = format_idle_children_nudge(children)
assert "...and" not in text
for i in range(3):
assert f"task-{i}" in text
def test_over_display_cap_renders_overflow_line(self):
n = NUDGE_IDLE_CHILDREN_DISPLAY_CAP + 4
children = [
{"ws_id": f"ws-{i:08d}", "name": f"task-{i}", "state": "thinking"} for i in range(n)
]
text = format_idle_children_nudge(children)
assert f"...and {n - NUDGE_IDLE_CHILDREN_DISPLAY_CAP} more" in text
# First N children are inline; later ones are folded into "...and N more".
for i in range(NUDGE_IDLE_CHILDREN_DISPLAY_CAP):
assert f"task-{i}" in text
for i in range(NUDGE_IDLE_CHILDREN_DISPLAY_CAP, n):
# Names beyond the display cap aren't visible; only counted.
assert f"task-{i}" not in text
def test_over_wait_cap_truncates_suggestion_ws_ids(self):
n = NUDGE_IDLE_CHILDREN_WAIT_CAP + 5
children = [
{"ws_id": f"ws-{i:08d}", "name": f"task-{i}", "state": "running"} for i in range(n)
]
text = format_idle_children_nudge(children)
# The first WAIT_CAP ids appear in the suggestion; later ones don't.
first_in_suggestion = f"ws-{NUDGE_IDLE_CHILDREN_WAIT_CAP - 1:08d}"
first_excluded = f"ws-{NUDGE_IDLE_CHILDREN_WAIT_CAP:08d}"
assert first_in_suggestion in text
assert first_excluded not in text
def test_unnamed_child_falls_back(self):
children = [{"ws_id": "ws-deadbeef", "name": "", "state": "attention"}]
text = format_idle_children_nudge(children)
assert "(unnamed)" in text
assert "attention" in text
def test_newline_in_name_does_not_forge_extra_bullet(self):
"""A workstream name with embedded ``\\n`` / ``\\t`` / ``\\r`` MUST
NOT break the bullet structure :func:`sanitize_name`'s strict
regex strips control chars (incl. TAB/LF/CR) so the name stays
on a single line under its own bullet. Without this, a
malicious child name like ``"foo\\n - ws-fake (running): bar"``
would forge a fake sibling row in the rendered list.
"""
children = [
{"ws_id": "ws-real0001", "name": "real", "state": "running"},
{
"ws_id": "ws-evil0002",
"name": "evil\n - ws-fake (running): forged",
"state": "thinking",
},
{"ws_id": "ws-real0003", "name": "tail", "state": "running"},
]
text = format_idle_children_nudge(children)
bullet_rows = [ln for ln in text.splitlines() if ln.startswith(" - ")]
assert len(bullet_rows) == 3, (
f"expected 3 bullet rows; got {len(bullet_rows)}: {bullet_rows!r}"
)
evil_row = next(row for row in bullet_rows if "ws-evil" in row)
assert "\n" not in evil_row
assert "\t" not in evil_row
assert "\r" not in evil_row
assert "evil" in evil_row
assert "ws-real" in bullet_rows[2]
assert "tail" in bullet_rows[2]
def test_missing_state_renders_question_mark(self):
children = [{"ws_id": "ws-12345678", "name": "x"}]
text = format_idle_children_nudge(children)
# Defensive default — exotic state keys / partial dicts shouldn't crash.
assert "?" in text
def test_no_system_reminder_envelope(self):
# The side-channel ``_apply_reminders_for_provider`` splice
# adds ``<system-reminder>`` at the wire boundary; the formatter
# MUST NOT wrap, or the model would see a doubled envelope.
text = format_idle_children_nudge([{"ws_id": "ws-x", "name": "y", "state": "running"}])
assert "<system-reminder>" not in text
assert "</system-reminder>" not in text
def test_format_nudge_returns_empty_for_idle_children(self):
# The static map's idle_children entry is the empty string by
# design — format_idle_children_nudge produces the real body.
assert format_nudge("idle_children") == ""
def test_should_nudge_recognises_idle_children_type(self, monkeypatch):
# Type registration in ``_NUDGE_MAP`` makes ``should_nudge``
# recognise it for cooldown gating; without the entry it would
# silently return False on every call.
state: dict[str, float] = {}
# message_count > 1 to clear the first-message gate.
assert should_nudge("idle_children", state, message_count=4, memory_count=0) is True
# Cooldown set on success → second immediate call returns False.
assert should_nudge("idle_children", state, message_count=5, memory_count=0) is False
class TestSanitizeName:
"""Strict sanitiser for single-line user-controlled name fields
(used by :func:`format_idle_children_nudge` for the workstream
``name``). Strips ASCII control chars **including** TAB/LF/CR
plus Unicode steering vectors and angle-bracket tag breakers.
"""
def test_empty_input_returns_empty(self):
from turnstone.core.metacognition import sanitize_name
assert sanitize_name("") == ""
def test_strips_tab_lf_cr(self):
"""Strict variant: TAB/LF/CR are stripped so a hostile name with
an embedded newline can't break a bullet's one-line structure.
"""
from turnstone.core.metacognition import sanitize_name
# All three become spaces (then collapsed to one inline space
# by the trailing ``strip()``-on-leading/trailing-only step
# — interior runs stay as multiple spaces, that's fine for a
# one-line name).
assert sanitize_name("a\tb") == "a b"
assert sanitize_name("a\nb") == "a b"
assert sanitize_name("a\rb") == "a b"
def test_strips_other_ascii_control_chars(self):
from turnstone.core.metacognition import sanitize_name
assert sanitize_name("a\x07b\x0bc\x0cd") == "a b c d"
assert sanitize_name("a\x7fb") == "a b"
def test_strips_angle_bracket_tag_breakers(self):
from turnstone.core.metacognition import sanitize_name
assert sanitize_name("a</thinking>b") == "a/thinkingb"
class TestSanitizePayload:
"""Permissive sanitiser used by the ``watch_triggered`` producer.
Strips ASCII control chars (except TAB/LF/CR), Unicode steering
vectors (bidi, zero-width, BOM, tag chars), and angle-bracket
tag breakers keeps everything else intact, so multi-line shell
output retains its line structure.
"""
def test_empty_input_returns_empty(self):
from turnstone.core.metacognition import sanitize_payload
assert sanitize_payload("") == ""
def test_strips_ascii_control_chars(self):
"""``\\x00``-``\\x1f`` minus TAB/LF/CR plus ``\\x7f`` (DEL) become spaces."""
from turnstone.core.metacognition import sanitize_payload
# BEL (0x07), VT (0x0b), FF (0x0c) — all in strip set.
assert sanitize_payload("a\x07b\x0bc\x0cd") == "a b c d"
# DEL (0x7f).
assert sanitize_payload("a\x7fb") == "a b"
def test_preserves_tab_lf_cr(self):
"""TAB / LF / CR are intentionally preserved so multi-line shell
output keeps its line structure when sanitised as a watch payload.
"""
from turnstone.core.metacognition import sanitize_payload
# Newlines kept; only the leading + trailing strip happens.
out = sanitize_payload("line1\nline2\n\tindented\rline3")
assert out == "line1\nline2\n\tindented\rline3"
def test_strips_bidi_and_zero_width(self):
from turnstone.core.metacognition import sanitize_payload
# U+202E RIGHT-TO-LEFT OVERRIDE; U+200B ZERO WIDTH SPACE.
assert sanitize_payload("abc") == "a b c"
def test_strips_angle_bracket_tag_breakers(self):
from turnstone.core.metacognition import sanitize_payload
# "<" / ">" go away entirely (not replaced with space) so a name
# like "</thinking>" doesn't leave a hole the model can read as
# a structural marker.
assert sanitize_payload("a</thinking>b") == "a/thinkingb"
-166
View File
@@ -1,166 +0,0 @@
"""Tests for alembic migration 049 (OAuth-MCP schema).
Drives ``command.upgrade`` from a programmatic Alembic config against
an isolated SQLite database per test, then asserts:
* the two new tables (``mcp_user_tokens``, ``mcp_oauth_pending``) exist,
* the eight new ``mcp_servers`` columns exist,
* the post-upgrade ``UPDATE mcp_servers`` normalization rewrites rows
with empty / missing headers to ``auth_type='none'`` while leaving
rows with non-empty headers at ``auth_type='static'``.
"""
from __future__ import annotations
from pathlib import Path
import sqlalchemy as sa
from alembic import command
from alembic.config import Config
_MIGRATIONS_DIR = str(
Path(__file__).resolve().parent.parent / "turnstone" / "core" / "storage" / "migrations"
)
def _alembic_cfg(db_path: Path) -> Config:
cfg = Config()
cfg.set_main_option("script_location", _MIGRATIONS_DIR)
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{db_path}")
return cfg
class TestMigration049:
def test_creates_new_tables_and_columns(self, tmp_path: Path) -> None:
db_path = tmp_path / "049.db"
cfg = _alembic_cfg(db_path)
# Walk forward through 048 first, then explicitly to 049 so we
# exercise the *upgrade* function (not just the schema's `head`).
command.upgrade(cfg, "048")
command.upgrade(cfg, "049")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
inspector = sa.inspect(engine)
tables = set(inspector.get_table_names())
assert "mcp_user_tokens" in tables
assert "mcp_oauth_pending" in tables
mcp_cols = {c["name"] for c in inspector.get_columns("mcp_servers")}
new_cols = {
"auth_type",
"oauth_client_id",
"oauth_client_secret_ct",
"oauth_scopes",
"oauth_audience",
"oauth_registration_mode",
"oauth_authorization_server_url",
"oauth_as_issuer_cached",
}
assert new_cols.issubset(mcp_cols), new_cols - mcp_cols
# Index check on mcp_oauth_pending.
indexes = {ix["name"] for ix in inspector.get_indexes("mcp_oauth_pending")}
assert "idx_mcp_pending_created" in indexes
finally:
engine.dispose()
def test_normalizes_empty_headers_to_none(self, tmp_path: Path) -> None:
"""Streamable-http rows with NULL / '' / '{}' headers become
auth_type='none'; rows with non-empty headers stay 'static'.
Stdio rows always stay 'static' regardless of headers the
column value is opaque when there is no HTTP transport."""
db_path = tmp_path / "049-norm.db"
cfg = _alembic_cfg(db_path)
# Apply everything up to 048, seed rows, then apply 049.
command.upgrade(cfg, "048")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
with engine.begin() as conn:
conn.execute(
sa.text(
"""
INSERT INTO mcp_servers (
server_id, name, transport, command, args, url,
headers, env, auto_approve, enabled, created_by,
registry_name, registry_version, registry_meta,
created, updated
) VALUES (
:sid, :name, :transport, '', '[]',
'https://x', :headers, '{}', 0, 1, '', NULL, '',
'{}', '2026-05-04T11:00:00', '2026-05-04T11:00:00'
)
"""
),
[
{
"sid": "s-empty-str",
"name": "empty-str",
"transport": "streamable-http",
"headers": "",
},
{
"sid": "s-empty-obj",
"name": "empty-obj",
"transport": "streamable-http",
"headers": "{}",
},
{
"sid": "s-with-headers",
"name": "with-headers",
"transport": "streamable-http",
"headers": '{"Authorization":"Bearer x"}',
},
# Stdio rows must keep the 'static' default, even
# though their headers are empty — auth_type is
# opaque for stdio.
{
"sid": "s-stdio-empty",
"name": "stdio-empty",
"transport": "stdio",
"headers": "{}",
},
{
"sid": "s-stdio-null",
"name": "stdio-null",
"transport": "stdio",
"headers": "",
},
],
)
command.upgrade(cfg, "049")
with engine.connect() as conn:
rows = dict(conn.execute(sa.text("SELECT name, auth_type FROM mcp_servers")).all())
assert rows["empty-str"] == "none"
assert rows["empty-obj"] == "none"
assert rows["with-headers"] == "static"
# Stdio rows must remain at the 'static' column default even
# when headers are empty — the migration only touches HTTP
# rows where auth_type is semantically meaningful.
assert rows["stdio-empty"] == "static"
assert rows["stdio-null"] == "static"
finally:
engine.dispose()
def test_full_chain_to_head(self, tmp_path: Path) -> None:
"""Sanity: running ``upgrade head`` on a fresh DB yields the
same end-state column set as ``_schema.metadata``."""
db_path = tmp_path / "049-head.db"
cfg = _alembic_cfg(db_path)
command.upgrade(cfg, "head")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
from turnstone.core.storage._schema import mcp_servers
inspector = sa.inspect(engine)
actual = {c["name"] for c in inspector.get_columns("mcp_servers")}
expected = {c.name for c in mcp_servers.columns}
assert expected.issubset(actual), expected - actual
finally:
engine.dispose()
-70
View File
@@ -212,73 +212,3 @@ class TestModelDefinitionStorage:
m = db.get_model_definition(did)
assert m is not None
assert m["temperature"] is None
def test_reasoning_flags_default(self, db: SQLiteBackend) -> None:
"""surface_persisted_reasoning defaults True; replay_reasoning_to_model defaults False."""
did = _make_id()
db.create_model_definition(definition_id=did, alias="reason-default", model="gpt-5")
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is True
assert m["replay_reasoning_to_model"] is False
def test_create_with_explicit_reasoning_flags(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(
definition_id=did,
alias="reason-explicit",
model="claude-opus-4-7",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is False
assert m["replay_reasoning_to_model"] is True
# Same values must round-trip via the alias lookup too.
m_alias = db.get_model_definition_by_alias("reason-explicit")
assert m_alias is not None
assert m_alias["surface_persisted_reasoning"] is False
assert m_alias["replay_reasoning_to_model"] is True
def test_update_surface_persisted_reasoning(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(definition_id=did, alias="upd-persist", model="gpt-5")
ok = db.update_model_definition(did, surface_persisted_reasoning=False)
assert ok is True
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is False
assert m["replay_reasoning_to_model"] is False # untouched
def test_update_replay_reasoning_to_model(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(definition_id=did, alias="upd-replay", model="gpt-5")
ok = db.update_model_definition(did, replay_reasoning_to_model=True)
assert ok is True
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is True # untouched
assert m["replay_reasoning_to_model"] is True
def test_list_returns_reasoning_flags(self, db: SQLiteBackend) -> None:
db.create_model_definition(
definition_id=_make_id(),
alias="list-a",
model="gpt-5",
surface_persisted_reasoning=True,
replay_reasoning_to_model=False,
)
db.create_model_definition(
definition_id=_make_id(),
alias="list-b",
model="claude-opus-4-7",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
models = db.list_model_definitions()
by_alias = {m["alias"]: m for m in models}
assert by_alias["list-a"]["surface_persisted_reasoning"] is True
assert by_alias["list-a"]["replay_reasoning_to_model"] is False
assert by_alias["list-b"]["surface_persisted_reasoning"] is False
assert by_alias["list-b"]["replay_reasoning_to_model"] is True
+4 -133
View File
@@ -76,23 +76,6 @@ class TestModelConfig:
assert cfg.temperature == 0.0
assert cfg.temperature is not None
def test_reasoning_flags_default(self) -> None:
cfg = ModelConfig(alias="x", base_url="x", api_key="x", model="x")
assert cfg.surface_persisted_reasoning is True
assert cfg.replay_reasoning_to_model is False
def test_reasoning_flags_set(self) -> None:
cfg = ModelConfig(
alias="x",
base_url="x",
api_key="x",
model="x",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
assert cfg.surface_persisted_reasoning is False
assert cfg.replay_reasoning_to_model is True
# ---------------------------------------------------------------------------
# ModelRegistry
@@ -331,11 +314,7 @@ class TestLoadModelRegistry:
api_key="dummy",
model="local-model",
)
# The CLI ``"default"`` shim is suppressed once ``[models.*]``
# populates configs — only the explicit alias survives.
assert reg.count == 1
assert reg.has_alias("openai")
assert not reg.has_alias("default")
assert reg.count == 2 # "openai" + "default"
assert reg.default == "openai"
_, model, _ = reg.resolve()
assert model == "gpt-4o"
@@ -566,12 +545,7 @@ class TestLoadModelRegistryWithDB:
assert cfg.source == "config"
def test_db_only_models_coexist(self) -> None:
"""DB models coexist alongside config.toml models.
The CLI ``"default"`` shim is suppressed when DB / config models
already populate the registry see
``test_cli_default_shim_skipped_when_db_models_present``.
"""
"""DB models coexist alongside config.toml models."""
storage = _MockStorage(
[
{
@@ -595,7 +569,7 @@ class TestLoadModelRegistryWithDB:
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
assert reg.has_alias("db-only")
assert reg.has_alias("config-only")
assert not reg.has_alias("default")
assert reg.has_alias("default")
assert reg.get_config("db-only").source == "db"
assert reg.get_config("config-only").source == "config"
@@ -615,12 +589,10 @@ class TestLoadModelRegistryWithDB:
}
]
)
# The CLI default shim is suppressed when the DB row populates
# configs, so only the DB-sourced alias exists here.
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
assert reg.get_config("from-db").source == "db"
assert not reg.has_alias("default")
assert reg.get_config("default").source == ""
def test_disabled_db_models_excluded(self) -> None:
"""Disabled DB models are not loaded."""
@@ -714,53 +686,6 @@ class TestLoadModelRegistryWithDB:
assert cfg.max_tokens is None
assert cfg.reasoning_effort is None
def test_db_reasoning_flags_loaded(self) -> None:
"""Per-model reasoning flags from DB are carried in ModelConfig."""
storage = _MockStorage(
[
{
"alias": "anth-thinking",
"model": "claude-opus-4-7",
"provider": "anthropic",
"base_url": "",
"api_key": "sk-anth",
"context_window": 200000,
"capabilities": "{}",
"enabled": True,
"surface_persisted_reasoning": False,
"replay_reasoning_to_model": True,
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
cfg = reg.get_config("anth-thinking")
assert cfg.surface_persisted_reasoning is False
assert cfg.replay_reasoning_to_model is True
def test_db_reasoning_flags_default_when_absent(self) -> None:
"""Pre-052 rows without the columns degrade to dataclass defaults."""
storage = _MockStorage(
[
{
"alias": "legacy-row",
"model": "gpt-5",
"provider": "openai",
"base_url": "",
"api_key": "",
"context_window": 32768,
"capabilities": "{}",
"enabled": True,
# surface_persisted_reasoning + replay_reasoning_to_model intentionally absent
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
cfg = reg.get_config("legacy-row")
assert cfg.surface_persisted_reasoning is True
assert cfg.replay_reasoning_to_model is False
def test_db_default_alias_not_clobbered(self) -> None:
"""DB model with alias='default' is not overwritten by CLI args."""
storage = _MockStorage(
@@ -944,8 +869,6 @@ class _FakeUI:
self.infos: list[str] = []
self.errors: list[str] = []
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -1686,58 +1609,6 @@ class TestLoadModelRegistryDBOnly:
reg = load_model_registry(model="", storage=storage)
assert not reg.has_alias("default")
def test_cli_default_shim_skipped_when_db_models_present(self) -> None:
"""An auto-detected ``--model`` does NOT synthesise a ``default``
alias when the DB already contributes models.
Regression for the silent bypass of ``model.task_alias`` /
``model.plan_alias``: a synthesised ``default`` aliased to whatever
``--base-url`` was at boot leaks into the LLM-visible alias list,
and the LLM picks it for ``task_agent(model="default")`` which
then routes around the operator-configured per-role default.
"""
storage = _MockStorage(
[
{
"alias": "gh200",
"model": "deepseek-ai/DeepSeek-V4-Flash",
"provider": "openai",
"base_url": "http://gh200:8000/v1",
"api_key": "sk-gh200",
"context_window": 1048576,
"capabilities": "{}",
"enabled": True,
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry(
base_url="http://flatspark:8000/v1",
api_key="sk-flatspark",
model="qwen3.6-35B-A3B", # populated by ``detect_model``
storage=storage,
)
assert reg.has_alias("gh200")
assert not reg.has_alias("default")
def test_cli_default_shim_skipped_when_config_models_present(self) -> None:
"""Same shim suppression when only ``[models.*]`` populates configs."""
fake_cfg: dict[str, Any] = {
"models": {"local": {"model": "qwen3-32b"}},
}
with patch("turnstone.core.model_registry.load_config", return_value=fake_cfg):
reg = load_model_registry("http://x/v1", "x", "fallback-model")
assert reg.has_alias("local")
assert not reg.has_alias("default")
def test_cli_default_shim_still_fires_when_registry_empty(self) -> None:
"""Single-model CLI mode (no DB, no config.toml [models.*]) keeps
the back-compat ``default`` alias."""
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "lone-model")
assert reg.has_alias("default")
assert reg.get_config("default").model == "lone-model"
# ---------------------------------------------------------------------------
# server._effective_routing / _apply_routing_overrides
-1
View File
@@ -197,7 +197,6 @@ _EXPECTED_AFFECTING_KEYS = frozenset(
"coordinator.model_alias",
"coordinator.reasoning_effort",
"judge.model",
"channels.default_model_alias",
}
)
-400
View File
@@ -1,400 +0,0 @@
"""Tests for the console-side ``NotifyDispatcher``.
Exercises the dispatcher against the SQLite synthetic-sweep path so the
suite runs without a Postgres dependency. The PG path is shaped the
same way (same handler invocation semantics) the only difference is
the underlying stream's wake-up source, which is covered separately in
``test_storage_notify.py::TestPostgresNotify``.
"""
from __future__ import annotations
import threading
import time
import pytest
@pytest.fixture
def dispatcher_factory(storage):
"""Yield a factory that constructs + tracks dispatchers for teardown."""
from turnstone.console.notify_dispatcher import NotifyDispatcher
created: list[NotifyDispatcher] = []
def _make(*, channels: list[str]) -> NotifyDispatcher:
d = NotifyDispatcher(storage, channels=channels)
created.append(d)
return d
yield _make
for d in created:
d.stop(timeout=2.0)
def _wait_for(predicate, deadline_sec: float = 3.0) -> bool:
"""Poll ``predicate`` until True or timeout. Returns bool."""
deadline = time.monotonic() + deadline_sec
while time.monotonic() < deadline:
if predicate():
return True
time.sleep(0.02)
return False
def _start_ready(d, *, timeout: float = 5.0) -> None:
"""``d.start()`` + assert the listener is actually listening.
Closes the start-vs-notify race for backends where ``storage.listen``
blocks on the network (Postgres ``LISTEN`` over a fresh psycopg
connection): without the sync, a same-thread ``storage.notify`` can
fire before the LISTEN registers and the notification is lost.
"""
d.start()
if not d.wait_until_ready(timeout=timeout):
msg = f"dispatcher listener did not open within {timeout}s"
raise AssertionError(msg)
class TestSubscribe:
def test_subscribe_registers_handler(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen: list = []
d.subscribe("alpha", lambda n: seen.append(n))
_start_ready(d)
# Fire a notify via the storage layer — dispatcher delivers to handler.
storage.notify("alpha", "hello")
assert _wait_for(lambda: any(n.payload == "hello" for n in seen))
def test_subscribe_undeclared_channel_raises(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
with pytest.raises(ValueError, match="not declared"):
d.subscribe("beta", lambda n: None)
def test_subscribe_returns_unsubscribe_callable(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen: list = []
unsub = d.subscribe("alpha", lambda n: seen.append(n))
_start_ready(d)
storage.notify("alpha", "first")
assert _wait_for(lambda: any(n.payload == "first" for n in seen))
unsub()
# After unsubscribe, the handler no longer fires. Drain old hits
# so the next notify-vs-handler-count check is unambiguous.
seen.clear()
storage.notify("alpha", "second")
# Give the dispatcher a beat to deliver if it were going to.
time.sleep(0.2)
assert not any(n.payload == "second" for n in seen)
def test_construction_requires_at_least_one_channel(self, storage):
from turnstone.console.notify_dispatcher import NotifyDispatcher
with pytest.raises(ValueError, match="at least one"):
NotifyDispatcher(storage, channels=[])
def test_duplicate_channels_deduplicated(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha", "alpha", "beta"])
assert d.channels == ["alpha", "beta"]
class TestDispatch:
def test_multiple_handlers_each_invoked(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen_a: list = []
seen_b: list = []
d.subscribe("alpha", lambda n: seen_a.append(n))
d.subscribe("alpha", lambda n: seen_b.append(n))
_start_ready(d)
storage.notify("alpha", "shared")
assert _wait_for(lambda: seen_a and seen_b)
assert seen_a[0].payload == "shared"
assert seen_b[0].payload == "shared"
def test_handler_exception_does_not_break_dispatch(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
survived: list = []
def _broken(_n):
msg = "boom"
raise RuntimeError(msg)
d.subscribe("alpha", _broken)
d.subscribe("alpha", lambda n: survived.append(n))
_start_ready(d)
storage.notify("alpha", "after_broken")
# The second handler runs even though the first raised.
assert _wait_for(lambda: any(n.payload == "after_broken" for n in survived))
def test_dispatch_filters_by_channel(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha", "beta"])
seen_a: list = []
seen_b: list = []
d.subscribe("alpha", lambda n: seen_a.append(n))
d.subscribe("beta", lambda n: seen_b.append(n))
_start_ready(d)
storage.notify("alpha", "for_a")
storage.notify("beta", "for_b")
assert _wait_for(lambda: seen_a and seen_b)
assert all(n.payload == "for_a" for n in seen_a)
assert all(n.payload == "for_b" for n in seen_b)
class TestReconnect:
"""Reconnect + synthetic ``reconcile`` notify on stream-open success.
Uses a stub storage that owns its own listen stream so the test can
drive a controlled stream-error sequence the SQLite path can't
raise :class:`NotifyConnectionError`, and the PG path requires a
real database outage to exercise this code, neither of which fits a
unit test. The dispatcher's threading and reconcile-pending logic
are storage-agnostic the dispatcher sees the same
:class:`NotifyStream` Protocol regardless of backend.
"""
def test_reconcile_fires_after_reopen_not_before(self):
from turnstone.console.notify_dispatcher import NotifyDispatcher
from turnstone.core.storage._notify import Notify, NotifyConnectionError
# State machine: open -> first poll raises NotifyConnectionError
# -> dispatcher waits backoff then reopens -> second open's first
# poll blocks forever (test stops the dispatcher before then).
# The fix: synthetic reconcile fires AFTER the second open
# succeeds, not after the first open fails.
sequence: list[str] = []
reopen_event = threading.Event()
class _StubStream:
def __init__(self, fail_first_poll: bool):
self._fail = fail_first_poll
self._closed = False
def poll(self, _timeout):
if self._closed:
return []
if self._fail:
self._fail = False
sequence.append("poll_raises")
msg = "fake-disconnect"
raise NotifyConnectionError(msg)
sequence.append("poll_returns")
# Block until close to simulate a quiet steady-state.
time.sleep(0.5)
return []
def close(self):
self._closed = True
class _StubStorage:
def __init__(self):
self._open_count = 0
def listen(self, _channels):
import contextlib as _contextlib
@_contextlib.contextmanager
def _cm():
self._open_count += 1
sequence.append(f"open_{self._open_count}")
if self._open_count == 2:
reopen_event.set()
stream = _StubStream(fail_first_poll=(self._open_count == 1))
try:
yield stream
finally:
stream.close()
return _cm()
# Speed up backoff so the reopen happens promptly in the test.
import turnstone.console.notify_dispatcher as nd_mod
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
try:
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
got: list[Notify] = []
d.subscribe("alpha", lambda n: got.append(n))
d.start()
try:
# Wait for the second open (post-reconnect).
assert reopen_event.wait(3.0), "dispatcher did not reopen after disconnect"
# Reconcile should be delivered shortly after the reopen.
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if any(n.payload == "reconcile" for n in got):
break
time.sleep(0.02)
assert any(n.payload == "reconcile" for n in got), (
f"no reconcile delivered; sequence={sequence}, got={got}"
)
# The reconcile must NOT fire before the second open —
# if it did, the index of 'open_2' in sequence would
# come after any reconcile-emitting work. Check ordering:
# 'open_1' < 'poll_raises' < 'open_2' (synthesize happens
# inside the with-block of the SECOND open).
ix_open_1 = sequence.index("open_1")
ix_raises = sequence.index("poll_raises")
ix_open_2 = sequence.index("open_2")
assert ix_open_1 < ix_raises < ix_open_2
finally:
d.stop(timeout=2.0)
finally:
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
def test_generic_exception_path_also_synthesizes_reconcile(self):
"""Exceptions thrown during ``listen()`` (not via stream.poll) still trigger reconcile.
Models the ``psycopg.connect()`` / initial ``LISTEN`` failure
shape, which doesn't go through the stream's exception
translator and would hit the generic ``except Exception``
branch. Pre-fix, that branch emitted no reconcile.
"""
from turnstone.console.notify_dispatcher import NotifyDispatcher
reopen_event = threading.Event()
class _StubStream:
def __init__(self):
self._closed = False
def poll(self, _timeout):
if self._closed:
return []
time.sleep(0.5)
return []
def close(self):
self._closed = True
class _StubStorage:
def __init__(self):
self._open_count = 0
def listen(self, _channels):
import contextlib as _contextlib
self._open_count += 1
if self._open_count == 1:
# First open raises a generic exception (e.g.
# ``psycopg.OperationalError`` from a failed connect)
# — landing in the dispatcher's generic except branch.
msg = "fake-connect-failure"
raise RuntimeError(msg)
@_contextlib.contextmanager
def _cm():
reopen_event.set()
stream = _StubStream()
try:
yield stream
finally:
stream.close()
return _cm()
import turnstone.console.notify_dispatcher as nd_mod
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
try:
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
got: list = []
d.subscribe("alpha", lambda n: got.append(n))
d.start()
try:
assert reopen_event.wait(3.0), "dispatcher did not reopen after generic exception"
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if any(n.payload == "reconcile" for n in got):
break
time.sleep(0.02)
assert any(n.payload == "reconcile" for n in got), (
"no reconcile delivered after generic-exception recovery"
)
finally:
d.stop(timeout=2.0)
finally:
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
class TestCoalescing:
"""Same-channel burst collapses to one handler invocation per batch."""
def test_burst_coalesces_to_one_handler_call_per_channel(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
invocations: list = []
# Slow handler to ensure all bursts queue up before the first
# call returns — gives the dispatch loop time to coalesce.
coalesce_gate = threading.Event()
def _slow_handler(n):
invocations.append(n)
coalesce_gate.wait(0.05)
d.subscribe("alpha", _slow_handler)
_start_ready(d)
# Burst of 10 notifies on the same channel — should coalesce
# down to many fewer handler invocations.
for i in range(10):
storage.notify("alpha", str(i))
# Wait until the dispatch settles (handler is called at least once
# and the queue empties).
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if invocations and d._dispatch_queue.empty():
time.sleep(0.1) # allow any final coalesced call to land
break
time.sleep(0.02)
coalesce_gate.set()
# At least one handler call; well fewer than 10 (coalescing
# collapsed the burst). Exact count depends on timing — typical
# is 1-2 invocations per burst on a fast machine.
assert invocations, "handler never fired"
assert len(invocations) < 10, (
f"expected coalescing to collapse burst of 10; got {len(invocations)} invocations"
)
class TestLifecycle:
def test_start_is_idempotent(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
d.start() # No-op, no thread doubling
# Single listener + single dispatch thread are spawned regardless.
# Inspect by name so we don't depend on the exact thread count of
# the test runner.
listener_threads = [
t for t in threading.enumerate() if t.name == "notify-dispatcher-listener"
]
dispatch_threads = [
t for t in threading.enumerate() if t.name == "notify-dispatcher-dispatch"
]
assert len(listener_threads) == 1
assert len(dispatch_threads) == 1
def test_stop_is_idempotent(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
d.stop(timeout=2.0)
d.stop(timeout=2.0) # No-op, no error
def test_stop_without_start_is_noop(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.stop(timeout=1.0) # No-op, no thread to join
def test_stop_joins_threads(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
# Capture thread references then stop and assert they exited.
threads_before = [
t
for t in threading.enumerate()
if t.name in {"notify-dispatcher-listener", "notify-dispatcher-dispatch"}
]
assert threads_before
d.stop(timeout=3.0)
time.sleep(0.05)
for t in threads_before:
assert not t.is_alive(), f"{t.name} still alive after stop"
-562
View File
@@ -1,562 +0,0 @@
"""Unit tests for :class:`NudgeQueue`."""
from __future__ import annotations
import logging
import threading
import pytest
from turnstone.core.nudge_queue import TOOL_DRAIN, USER_DRAIN, NudgeQueue
class TestEnqueueDrain:
def test_enqueue_drain_fifo_order(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "any")
# Drain everything regardless of channel — preserves insertion order.
out = q.drain({"user", "tool", "any"})
# Drain returns ``(nudge_type, text, metadata)``; producers
# without ``metadata`` see ``None`` in the third slot.
assert out == [("a", "1", None), ("b", "2", None), ("c", "3", None)]
assert len(q) == 0
def test_drain_filter_keeps_non_matching(self):
q = NudgeQueue()
q.enqueue("a", "x", "user")
q.enqueue("b", "y", "tool")
# Drain only user → tool entry stays.
out = q.drain(USER_DRAIN)
assert out == [("a", "x", None)]
assert len(q) == 1
# Now drain tool — gets the remaining entry.
out = q.drain(TOOL_DRAIN)
assert out == [("b", "y", None)]
assert len(q) == 0
def test_any_channel_drains_on_either_seam(self):
q = NudgeQueue()
q.enqueue("c", "z", "any")
# User-seam drain pulls "any".
assert q.drain(USER_DRAIN) == [("c", "z", None)]
assert len(q) == 0
# Re-enqueue and prove tool-seam also drains "any".
q.enqueue("d", "w", "any")
assert q.drain(TOOL_DRAIN) == [("d", "w", None)]
assert len(q) == 0
def test_drain_empty_filter_no_op(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
# Empty filter drains nothing.
assert q.drain(set()) == []
assert len(q) == 1
def test_drain_empty_queue_returns_empty_list(self):
q = NudgeQueue()
# Fast-path: no items → no kept-deque allocation, just `[]`.
assert q.drain(USER_DRAIN) == []
assert q.drain({"user", "tool", "any"}) == []
assert len(q) == 0
def test_drain_preserves_order_across_partial_drain(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "user")
q.enqueue("d", "4", "tool")
# Drain user — should get "a" then "c" in order; "b","d" stay.
assert q.drain({"user"}) == [("a", "1", None), ("c", "3", None)]
# Tool drain follows insertion order on remaining.
assert q.drain({"tool"}) == [("b", "2", None), ("d", "4", None)]
class TestLenAndClear:
def test_len_does_not_mutate(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
assert len(q) == 1
assert len(q) == 1 # second call still 1; not consumed
assert q.pending() == [("a", "1")]
def test_len_empty_is_zero(self):
q = NudgeQueue()
assert len(q) == 0
def test_clear_returns_count(self):
q = NudgeQueue()
assert q.clear() == 0
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "any")
assert q.clear() == 3
assert len(q) == 0
def test_clear_empty_returns_zero(self):
q = NudgeQueue()
assert q.clear() == 0
class TestDropOldestByType:
def test_drop_oldest_by_type_removes_earliest_match(self):
"""Drop the FIRST entry of the matching type; later matches stay."""
q = NudgeQueue()
q.enqueue("other", "first", "any")
q.enqueue("target", "older", "any")
q.enqueue("target", "newer", "any")
# "older" is the earliest target — drop it.
assert q.drop_oldest_by_type("target") is True
assert q.pending() == [("other", "first"), ("target", "newer")]
def test_drop_oldest_by_type_no_match_returns_false(self):
"""Empty queue and unmatched-type cases both return False."""
q = NudgeQueue()
# Empty.
assert q.drop_oldest_by_type("target") is False
# Non-matching items only.
q.enqueue("other", "1", "any")
q.enqueue("other", "2", "tool")
assert q.drop_oldest_by_type("target") is False
# Queue is unaffected.
assert q.pending() == [("other", "1"), ("other", "2")]
def test_drop_oldest_by_type_only_drops_one(self):
"""Multiple matching entries → only the first is removed."""
q = NudgeQueue()
q.enqueue("target", "1", "any")
q.enqueue("target", "2", "any")
q.enqueue("target", "3", "any")
assert q.drop_oldest_by_type("target") is True
assert q.pending() == [("target", "2"), ("target", "3")]
def test_drop_oldest_by_type_channel_filter(self):
"""With ``channel`` set, drop walks only that channel. Pairs with
:meth:`count_by_type(..., channel=...)` so producer-side soft caps
operate on a consistent entry set.
"""
q = NudgeQueue()
q.enqueue("target", "user-1", "user")
q.enqueue("target", "any-1", "any")
q.enqueue("target", "any-2", "any")
# Drop the oldest "any"-channel target — leaves the user one
# untouched even though it's earlier in insertion order.
assert q.drop_oldest_by_type("target", channel="any") is True
assert q.pending() == [
("target", "user-1"),
("target", "any-2"),
]
# And a channel with no matches returns False without touching
# the queue.
assert q.drop_oldest_by_type("target", channel="tool") is False
assert q.pending() == [
("target", "user-1"),
("target", "any-2"),
]
class TestCapAtOrDropOldest:
def test_below_cap_no_drop(self):
q = NudgeQueue()
for i in range(3):
q.enqueue("target", f"t-{i}", "any")
# 3 entries, cap=5 → no drop.
assert q.cap_at_or_drop_oldest("target", 5, channel="any") is False
assert q.count_by_type("target") == 3
def test_at_cap_drops_oldest(self):
q = NudgeQueue()
for i in range(5):
q.enqueue("target", f"t-{i}", "any")
# 5 entries, cap=5 → drop the oldest ("t-0"), leaving 4.
assert q.cap_at_or_drop_oldest("target", 5, channel="any") is True
remaining = q.pending()
assert ("target", "t-0") not in remaining
assert len(remaining) == 4
assert remaining[0] == ("target", "t-1") # FIFO drop-oldest preserved
def test_above_cap_drops_only_one(self):
q = NudgeQueue()
for i in range(7):
q.enqueue("target", f"t-{i}", "any")
# 7 entries, cap=5 → drop only ONE per call (soft-cap regulates over time).
assert q.cap_at_or_drop_oldest("target", 5, channel="any") is True
assert q.count_by_type("target") == 6
def test_channel_filter_respected(self):
q = NudgeQueue()
for i in range(3):
q.enqueue("target", f"any-{i}", "any")
for i in range(3):
q.enqueue("target", f"user-{i}", "user")
# 3 "any"-channel entries; cap=3 on channel="any" → drop oldest "any" only.
assert q.cap_at_or_drop_oldest("target", 3, channel="any") is True
# User-channel entries untouched.
assert q.count_by_type("target", channel="user") == 3
assert q.count_by_type("target", channel="any") == 2
def test_other_types_ignored(self):
q = NudgeQueue()
for i in range(5):
q.enqueue("other", f"o-{i}", "any")
q.enqueue("target", "t-0", "any")
# Only one "target" entry; cap=1 on "target" → drop it. "other"
# entries are untouched even though queue holds 6 total.
assert q.cap_at_or_drop_oldest("target", 1, channel="any") is True
assert q.count_by_type("target") == 0
assert q.count_by_type("other") == 5
def test_zero_or_negative_cap_no_op(self):
q = NudgeQueue()
q.enqueue("target", "t-0", "any")
assert q.cap_at_or_drop_oldest("target", 0, channel="any") is False
assert q.cap_at_or_drop_oldest("target", -1, channel="any") is False
assert q.count_by_type("target") == 1
def test_no_match_returns_false(self):
q = NudgeQueue()
q.enqueue("other", "o-0", "any")
assert q.cap_at_or_drop_oldest("target", 1, channel="any") is False
assert q.count_by_type("other") == 1
class TestCountByType:
def test_count_by_type_no_channel(self):
"""Count across all channels with ``channel=None``."""
q = NudgeQueue()
q.enqueue("target", "1", "user")
q.enqueue("other", "x", "any")
q.enqueue("target", "2", "any")
q.enqueue("target", "3", "tool")
assert q.count_by_type("target") == 3
assert q.count_by_type("other") == 1
assert q.count_by_type("missing") == 0
def test_count_by_type_with_channel_filter(self):
"""Filter narrows the count to one channel — used by producer-side
soft caps that pair with ``drop_oldest_by_type(..., channel=...)``.
"""
q = NudgeQueue()
q.enqueue("target", "u-1", "user")
q.enqueue("target", "a-1", "any")
q.enqueue("target", "a-2", "any")
q.enqueue("target", "t-1", "tool")
assert q.count_by_type("target", channel="any") == 2
assert q.count_by_type("target", channel="user") == 1
assert q.count_by_type("target", channel="tool") == 1
def test_count_by_type_empty_queue(self):
q = NudgeQueue()
assert q.count_by_type("anything") == 0
assert q.count_by_type("anything", channel="any") == 0
class TestPending:
def test_pending_no_filter_returns_all_in_order(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "any")
# All three, in insertion order, as (nudge_type, text) tuples.
assert q.pending() == [("a", "1"), ("b", "2"), ("c", "3")]
def test_pending_channel_filter(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "user")
q.enqueue("d", "4", "any")
assert q.pending("user") == [("a", "1"), ("c", "3")]
assert q.pending("tool") == [("b", "2")]
assert q.pending("any") == [("d", "4")]
def test_pending_does_not_mutate(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
# Two pending calls return same content; nothing consumed.
first = q.pending()
second = q.pending()
assert first == second
assert len(q) == 2
class TestMetadata:
"""Producer-supplied ``metadata`` rides alongside ``(type, text)`` on
drain. Today only ``watch_triggered`` populates it; the wire shape
accommodates future producers (e.g. structured tool_error context)
without another schema bump.
"""
def test_drain_returns_metadata_when_set(self):
q = NudgeQueue()
meta = {"watch_name": "w1", "command": "ls", "poll_count": 2}
q.enqueue("watch_triggered", "$ ls\nfile.txt", "any", metadata=meta)
out = q.drain({"any"})
assert out == [("watch_triggered", "$ ls\nfile.txt", meta)]
def test_drain_returns_none_when_metadata_unset(self):
q = NudgeQueue()
q.enqueue("idle_children", "kids", "any") # no metadata kwarg
out = q.drain({"any"})
assert out == [("idle_children", "kids", None)]
def test_pending_with_metadata_projects_third_field(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("watch_triggered", "out", "any", metadata={"watch_name": "w"})
snapshot = q.pending_with_metadata()
assert snapshot == [
("a", "1", None),
("watch_triggered", "out", {"watch_name": "w"}),
]
# ``pending`` (without metadata) keeps the legacy 2-tuple shape.
assert q.pending() == [("a", "1"), ("watch_triggered", "out")]
def test_metadata_survives_partial_drain(self):
"""A ``user``-channel drain leaves an unaffected ``tool``-channel
entry its metadata must still be present on the next drain."""
q = NudgeQueue()
q.enqueue("user_thing", "u", "user")
q.enqueue("watch_triggered", "w-out", "tool", metadata={"watch_name": "w1"})
# User drain doesn't touch the tool entry.
assert q.drain({"user"}) == [("user_thing", "u", None)]
# Tool drain still has the metadata.
assert q.drain({"tool"}) == [
("watch_triggered", "w-out", {"watch_name": "w1"}),
]
def test_metadata_with_valid_until_predicate(self):
"""Metadata + ``valid_until`` co-exist on the same entry; the
predicate gate runs as before, and on a True result the metadata
rides the drained tuple.
"""
q = NudgeQueue()
q.enqueue(
"watch_triggered",
"w-out",
"any",
valid_until=lambda: True,
metadata={"watch_name": "w1", "is_final": True},
)
out = q.drain({"any"})
assert out == [
("watch_triggered", "w-out", {"watch_name": "w1", "is_final": True}),
]
class TestHasPending:
def test_has_pending_returns_false_on_empty_queue(self):
q = NudgeQueue()
assert q.has_pending({"user", "any"}) is False
assert q.has_pending({"tool"}) is False
def test_has_pending_short_circuits_on_first_match(self):
q = NudgeQueue()
q.enqueue("a", "1", "tool")
q.enqueue("b", "2", "user")
# First entry doesn't match, second does — true after walking 2.
assert q.has_pending({"user"}) is True
def test_has_pending_returns_false_when_no_match(self):
q = NudgeQueue()
q.enqueue("a", "1", "tool")
q.enqueue("b", "2", "tool")
assert q.has_pending({"user", "any"}) is False
def test_has_pending_matches_any_channel(self):
q = NudgeQueue()
q.enqueue("a", "1", "any")
# USER_DRAIN-shaped filter pulls "any" entries.
assert q.has_pending(USER_DRAIN) is True
# TOOL_DRAIN-shaped filter also pulls "any" entries.
assert q.has_pending(TOOL_DRAIN) is True
def test_has_pending_does_not_mutate(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
before = q.pending()
q.has_pending({"user"})
q.has_pending({"tool"})
q.has_pending(set())
assert q.pending() == before
class TestValidation:
def test_invalid_channel_raises(self):
q = NudgeQueue()
with pytest.raises(ValueError, match="channel"):
q.enqueue("a", "1", "wake") # type: ignore[arg-type]
with pytest.raises(ValueError):
q.enqueue("b", "2", "") # type: ignore[arg-type]
# Queue is unaffected by the failed enqueues.
assert len(q) == 0
def test_channel_is_required(self):
q = NudgeQueue()
# No default — caller MUST pick a seam consciously.
with pytest.raises(TypeError):
q.enqueue("a", "1") # type: ignore[call-arg]
class TestValidUntil:
"""``valid_until`` predicate: drain re-checks freshness. Falsy
predicates drop the entry without delivery and log at ``info``
(normal lifecycle outcome); raising predicates drop the entry and
log at ``warning`` with ``exc_info`` (misbehaving predicate).
"""
def test_valid_until_true_delivers(self):
q = NudgeQueue()
q.enqueue("a", "1", "any", valid_until=lambda: True)
out = q.drain({"any"})
assert out == [("a", "1", None)]
def test_valid_until_false_drops_with_info_log(self, caplog: pytest.LogCaptureFixture):
q = NudgeQueue()
q.enqueue("a", "1", "any", valid_until=lambda: False)
with caplog.at_level(logging.INFO, logger="turnstone.core.nudge_queue"):
out = q.drain({"any"})
assert out == []
# Already removed from queue (drain partition removes BEFORE
# predicate check — falsy doesn't return to queue).
assert len(q) == 0
# The drop emits a structured info record so a wiring
# regression (a predicate that always returns False) is still
# observable, without spamming ``warning`` for the routine
# lifecycle case where ``valid_until`` is doing its job.
# structlog renders the event name + extras into ``msg`` as a
# single rendered string, so substring-match like the
# ``watch_dispatch.queue_full`` assertion in
# tests/test_watch_dispatch.py.
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
assert len(drops) == 1
assert drops[0].levelno == logging.INFO
assert "predicate_false" in drops[0].getMessage()
assert "'nudge_type': 'a'" in drops[0].getMessage()
assert "'channel': 'any'" in drops[0].getMessage()
assert "'text_len': 1" in drops[0].getMessage()
def test_valid_until_exception_drops_with_warning(self, caplog: pytest.LogCaptureFixture):
q = NudgeQueue()
def boom() -> bool:
raise RuntimeError("predicate crash")
q.enqueue("a", "1", "any", valid_until=boom)
with caplog.at_level(logging.WARNING, logger="turnstone.core.nudge_queue"):
out = q.drain({"any"})
assert out == []
# Crash-on-predicate is treated as "no longer valid" — drop, not propagate.
assert len(q) == 0
# Stays at ``warning`` (with ``exc_info``) because a raising
# predicate is a bug, not a normal lifecycle outcome.
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
assert len(drops) == 1
assert drops[0].levelno == logging.WARNING
rendered = drops[0].getMessage()
assert "predicate_raised" in rendered
assert "RuntimeError" in rendered
assert "predicate crash" in rendered
def test_valid_until_evaluated_outside_lock(self):
"""The predicate may do non-trivial work (e.g. storage I/O)
without blocking other producers. Verify the predicate runs
outside the queue's internal lock by enqueueing from inside
the predicate would deadlock if the lock was still held.
"""
q = NudgeQueue()
def reentrant() -> bool:
# If the lock is held during predicate eval, this enqueue
# would block forever (RLock would let it through, but the
# queue uses a plain Lock).
q.enqueue("inner", "from-predicate", "any", valid_until=lambda: True)
return True
q.enqueue("outer", "1", "any", valid_until=reentrant)
out = q.drain({"any"})
# Outer's predicate ran outside the lock, enqueued "inner";
# outer's True return delivered "outer". "inner" was enqueued
# AFTER the partition snapshot, so it stays in the queue.
assert out == [("outer", "1", None)]
assert q.pending() == [("inner", "from-predicate")]
def test_valid_until_only_evaluated_for_matching_channel(self):
"""A non-matching entry's predicate must NOT fire — that would
be wasted work (or worse, a side-effecting predicate would run
when the entry is supposed to stay queued).
"""
q = NudgeQueue()
calls = []
def track() -> bool:
calls.append(1)
return True
# Tool-channel entry; we drain user-channel. Predicate must not run.
q.enqueue("a", "1", "tool", valid_until=track)
q.drain({"user", "any"})
assert calls == []
# Entry stays queued.
assert q.pending("tool") == [("a", "1")]
def test_valid_until_default_none_always_delivers(self):
# No predicate → entry behaves identically to pre-PR-3 entries.
q = NudgeQueue()
q.enqueue("a", "1", "any") # no valid_until kwarg
assert q.drain({"any"}) == [("a", "1", None)]
class TestConcurrency:
def test_concurrent_enqueue_drain_no_loss(self):
"""16 producer threads × 64 nudges = 1024 total; one consumer
drains in a loop until producers finish + queue empty. Verify
every produced item is observed exactly once.
"""
q = NudgeQueue()
producers = 16
per_producer = 64
total = producers * per_producer
produced: set[tuple[str, str]] = set()
produced_lock = threading.Lock()
observed: list[tuple[str, str]] = []
observed_lock = threading.Lock()
done_event = threading.Event()
def produce(pid: int) -> None:
for i in range(per_producer):
key = (f"p{pid}", f"i{i}")
with produced_lock:
produced.add(key)
q.enqueue(key[0], key[1], "user")
def consume() -> None:
while not done_event.is_set() or len(q) > 0:
drained = q.drain({"user"})
if drained:
with observed_lock:
# Drop the trailing ``metadata`` slot — every
# entry here was enqueued without metadata, so
# the comparison set / count matches the produced
# ``(type, text)`` shape.
observed.extend((nt, txt) for nt, txt, _meta in drained)
consumer = threading.Thread(target=consume, daemon=True)
consumer.start()
threads = [threading.Thread(target=produce, args=(i,)) for i in range(producers)]
for t in threads:
t.start()
for t in threads:
t.join()
done_event.set()
consumer.join(timeout=5.0)
assert not consumer.is_alive(), "consumer didn't finish in time"
# Every produced key observed; no duplicates.
assert set(observed) == produced
assert len(observed) == total
assert len(q) == 0
-172
View File
@@ -1,172 +0,0 @@
"""Direct tests for the shared SSRF helpers in :mod:`turnstone.core.oauth_ssrf`.
The OIDC test suite already exercises these via the OIDC adapter
(``OIDCError`` re-raises). This file pins the canonical
:class:`OAuthSSRFError` exception so callers that don't go through OIDC
(notably ``mcp_oauth``) can rely on a stable contract.
"""
from __future__ import annotations
import urllib.parse
from unittest.mock import patch
import pytest
from turnstone.core.oauth_ssrf import (
OAuthSSRFError,
effective_port,
is_localhost,
validate_discovered_endpoint,
validate_url_no_ssrf,
)
class TestIsLocalhost:
def test_loopback_names(self) -> None:
assert is_localhost("localhost")
assert is_localhost("127.0.0.1")
assert is_localhost("::1")
assert is_localhost("foo.localhost")
def test_non_loopback(self) -> None:
assert not is_localhost("example.com")
assert not is_localhost("internal.corp")
class TestEffectivePort:
def test_explicit_port(self) -> None:
p = urllib.parse.urlparse("https://idp.example.com:9443/foo")
assert effective_port(p) == 9443
def test_default_https(self) -> None:
p = urllib.parse.urlparse("https://idp.example.com/foo")
assert effective_port(p) == 443
def test_default_http(self) -> None:
p = urllib.parse.urlparse("http://idp.example.com/foo")
assert effective_port(p) == 80
def test_unknown_scheme(self) -> None:
p = urllib.parse.urlparse("ftp://idp.example.com/foo")
assert effective_port(p) is None
class TestValidateUrlNoSSRF:
_PUBLIC_ADDR = [(2, 1, 6, "", ("93.184.216.34", 0))]
_PRIVATE_ADDR = [(2, 1, 6, "", ("10.0.0.1", 0))]
_LOOPBACK_ADDR = [(2, 1, 6, "", ("127.0.0.1", 0))]
def test_valid_https(self) -> None:
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
parsed = validate_url_no_ssrf("https://idp.example.com/foo", allow_http=False)
assert parsed.scheme == "https"
assert parsed.hostname == "idp.example.com"
def test_rejects_http_when_not_allowed(self) -> None:
with pytest.raises(OAuthSSRFError, match="must use HTTPS"):
validate_url_no_ssrf("http://idp.example.com", allow_http=False)
def test_allows_http_localhost_with_flag(self) -> None:
with patch("socket.getaddrinfo", return_value=self._LOOPBACK_ADDR):
validate_url_no_ssrf("http://localhost:8080", allow_http=True)
def test_rejects_http_non_localhost_even_with_flag(self) -> None:
with pytest.raises(OAuthSSRFError, match="must use HTTPS"):
validate_url_no_ssrf("http://idp.example.com", allow_http=True)
def test_rejects_userinfo(self) -> None:
with pytest.raises(OAuthSSRFError, match="embedded credentials"):
validate_url_no_ssrf("https://user:pass@idp.example.com", allow_http=False)
def test_rejects_private_address(self) -> None:
with (
patch("socket.getaddrinfo", return_value=self._PRIVATE_ADDR),
pytest.raises(OAuthSSRFError, match="non-public address"),
):
validate_url_no_ssrf("https://corp.example.com", allow_http=False)
def test_rejects_unresolvable(self) -> None:
import socket
with (
patch("socket.getaddrinfo", side_effect=socket.gaierror("fail")),
pytest.raises(OAuthSSRFError, match="cannot be resolved"),
):
validate_url_no_ssrf("https://no.such.host.invalid", allow_http=False)
class TestValidateDiscoveredEndpoint:
_PUBLIC_ADDR = [(2, 1, 6, "", ("93.184.216.34", 0))]
def test_same_origin_passes(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
validate_discovered_endpoint(
"https://idp.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_third_party_host_rejected(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with (
patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR),
pytest.raises(OAuthSSRFError, match="not trusted"),
):
validate_discovered_endpoint(
"https://attacker.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_trusted_endpoint_host_passes(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
validate_discovered_endpoint(
"https://shard.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset({"shard.example.com"}),
)
def test_known_google_alias_passes(self) -> None:
"""The hard-coded Google alias map covers oauth2.googleapis.com."""
issuer = urllib.parse.urlparse("https://accounts.google.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
validate_discovered_endpoint(
"https://oauth2.googleapis.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_scheme_mismatch_rejected(self) -> None:
# When the issuer is http://localhost (allow_http=True), an
# https:// endpoint must still be rejected as a scheme mismatch.
issuer = urllib.parse.urlparse("http://localhost:8080")
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("127.0.0.1", 0))]),
pytest.raises(OAuthSSRFError, match="scheme"),
):
validate_discovered_endpoint(
"https://localhost:8080/token",
issuer,
allow_http=True,
trusted_endpoint_hosts=frozenset(),
)
def test_port_mismatch_rejected(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with (
patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR),
pytest.raises(OAuthSSRFError, match="port"),
):
validate_discovered_endpoint(
"https://idp.example.com:9443/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
+151 -1434
View File
File diff suppressed because it is too large Load Diff
+45 -259
View File
@@ -18,7 +18,10 @@ from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from tests.conftest import make_oidc_test_config as _make_oidc_config
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
from turnstone.console.server import (
admin_delete_oidc_identity,
admin_list_oidc_identities,
@@ -29,12 +32,34 @@ from turnstone.core.auth import (
handle_oidc_authorize,
handle_oidc_callback,
)
from turnstone.core.oidc import OIDCConfig, OIDCError, OIDCKeyNotFoundError
from turnstone.core.oidc import OIDCConfig, OIDCError
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_oidc_config(**overrides: Any) -> OIDCConfig:
"""Build a test OIDCConfig with sensible defaults."""
defaults: dict[str, Any] = {
"enabled": True,
"issuer": "https://idp.example.com",
"client_id": "my-client",
"client_secret": "my-secret",
"scopes": "openid email profile",
"provider_name": "TestIDP",
"role_claim": "",
"role_map": {},
"password_enabled": True,
"authorization_endpoint": "https://idp.example.com/authorize",
"token_endpoint": "https://idp.example.com/token",
"userinfo_endpoint": "https://idp.example.com/userinfo",
"jwks_uri": "https://idp.example.com/.well-known/jwks.json",
}
defaults.update(overrides)
return OIDCConfig(**defaults)
# ---------------------------------------------------------------------------
# Thin handler wrappers — match the pattern used in server.py / console
@@ -265,9 +290,9 @@ class TestOIDCCallback:
) -> None:
storage.create_oidc_pending_state(state, nonce, code_verifier, audience)
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.oidc.provision_oidc_user")
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
def test_happy_path(
self,
mock_exchange: AsyncMock,
@@ -354,7 +379,7 @@ class TestOIDCCallback:
assert resp.status_code == 302
assert "Login+session+expired" in resp.headers["location"]
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
def test_code_exchange_failure(
self,
mock_exchange: AsyncMock,
@@ -371,8 +396,8 @@ class TestOIDCCallback:
assert resp.status_code == 302
assert "Authentication+failed" in resp.headers["location"]
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
def test_token_validation_failure(
self,
mock_exchange: AsyncMock,
@@ -391,10 +416,10 @@ class TestOIDCCallback:
assert resp.status_code == 302
assert "Authentication+failed" in resp.headers["location"]
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.oidc.provision_oidc_user")
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
def test_jwks_key_rotation_retry(
self,
mock_exchange: AsyncMock,
@@ -404,13 +429,13 @@ class TestOIDCCallback:
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""First validate raises kid-not-found, fetch_jwks retried, second validate succeeds."""
"""First validate raises 'kid not found in JWKS', fetch_jwks retried, second validate succeeds."""
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
# First call raises kid-not-found; second call (after JWKS refresh) succeeds
mock_validate.side_effect = [
OIDCKeyNotFoundError("Signing key 'new-kid' not found in JWKS"),
OIDCError("Signing key 'new-kid' not found in JWKS"),
{"sub": "user123", "email": "u@example.com", "nonce": "test-nonce"},
]
mock_fetch_jwks.return_value = {"keys": [{"kid": "new-kid", "kty": "RSA"}]}
@@ -425,61 +450,9 @@ class TestOIDCCallback:
mock_fetch_jwks.assert_called_once()
assert mock_validate.call_count == 2
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_callback_uses_keynotfound_for_jwks_retry(
self,
mock_exchange: AsyncMock,
mock_fetch_jwks: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""Retry path keys off the OIDCKeyNotFoundError type, not message substring."""
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
# First raises subclass; rephrased message must not affect retry behaviour.
mock_validate.side_effect = [
OIDCKeyNotFoundError("rotated key absent from cached set"),
{"sub": "user123", "email": "u@example.com", "nonce": "test-nonce"},
]
mock_fetch_jwks.return_value = {"keys": [{"kid": "new-kid", "kty": "RSA"}]}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
mock_fetch_jwks.assert_called_once()
assert mock_validate.call_count == 2
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_callback_returns_authentication_failed_on_missing_id_token(
self,
mock_exchange: AsyncMock,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""A token endpoint response without id_token must redirect with auth-failed."""
self._seed_pending_state(storage)
mock_exchange.return_value = {"access_token": "x"}
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_error=Authentication+failed" in resp.headers["location"]
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.oidc.provision_oidc_user")
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
def test_no_users_after_oidc_success_redirects_setup(
self,
mock_exchange: AsyncMock,
@@ -534,193 +507,6 @@ class TestOIDCCallback:
assert "oidc_error" in resp.headers["location"]
assert "Too+many" in resp.headers["location"]
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_setup_gate_uses_count_users_not_full_scan(
self,
mock_exchange: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""Callback's setup-complete gate must call count_users, not list_users."""
from unittest.mock import patch as obj_patch
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_validate.return_value = {
"sub": "u1",
"email": "u@example.com",
"nonce": "test-nonce",
}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
with (
obj_patch.object(storage, "count_users", wraps=storage.count_users) as count_spy,
obj_patch.object(storage, "list_users", wraps=storage.list_users) as list_spy,
):
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
count_spy.assert_called_once_with()
list_spy.assert_not_called()
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_state_cleanup_is_gated(
self,
mock_exchange: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""Cleanup runs once per cleanup-interval window, not every callback."""
from unittest.mock import patch as obj_patch
# First call seeds the cleanup timestamp; subsequent calls within
# _OIDC_STATE_CLEANUP_INTERVAL_S must NOT trigger cleanup again.
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_validate.return_value = {
"sub": "u1",
"email": "u@example.com",
"nonce": "test-nonce",
}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
with obj_patch.object(
storage, "cleanup_expired_oidc_states", wraps=storage.cleanup_expired_oidc_states
) as cleanup_spy:
for state in ("s1", "s2", "s3"):
self._seed_pending_state(storage, state=state, nonce="test-nonce")
authorize_client.get(
f"/v1/api/auth/oidc/callback?code=c&state={state}",
follow_redirects=False,
)
assert cleanup_spy.call_count == 1
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_callback_uses_pending_audience_not_handler_audience(
self,
mock_exchange: AsyncMock,
mock_validate: Any,
mock_provision: Any,
storage: SQLiteBackend,
) -> None:
"""JWT ``aud`` claim must come from the audience stored at /authorize,
not the audience the callback handler was invoked with.
Regression for the cross-service audience-confusion concern: a
login flow opened against the server (audience ``"turnstone-server"``)
must not be silently re-targeted to ``"turnstone-console"`` when
the callback runs through the console's handler wrapper.
"""
import jwt as pyjwt
# Seed pending state with the SERVER audience.
storage.create_oidc_pending_state(
"audience-state",
"audience-nonce",
"audience-verifier",
"turnstone-server",
)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_validate.return_value = {
"sub": "user-aud",
"email": "u@example.com",
"nonce": "audience-nonce",
}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
# Wire a callback bound to the CONSOLE audience. After bug-3 the
# stored audience must take precedence.
async def _console_callback(request: Request) -> Response:
return await handle_oidc_callback(request, "turnstone-console")
jwt_secret = "test-jwt-secret-key-padded-32b!!"
app = Starlette(
routes=[Mount("/v1", routes=[Route("/api/auth/oidc/callback", _console_callback)])]
)
app.state.oidc_config = _make_oidc_config()
app.state.auth_storage = storage
app.state.jwt_secret = jwt_secret
app.state.jwks_data = {"keys": []}
app.state.login_limiter = None
client = TestClient(app, raise_server_exceptions=False)
resp = client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=audience-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
# Extract the JWT from the Set-Cookie header and decode it.
set_cookie = resp.headers["set-cookie"]
cookie_kv = set_cookie.split(";", 1)[0]
name, _, token = cookie_kv.partition("=")
assert name == "turnstone_auth"
assert token
# Decoding without audience verification first to inspect the claim.
claims = pyjwt.decode(
token, jwt_secret, algorithms=["HS256"], options={"verify_aud": False}
)
assert claims["aud"] == "turnstone-server"
assert claims["aud"] != "turnstone-console"
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_jwks_refetch_dedup_when_kid_appears(
self,
mock_exchange: AsyncMock,
mock_fetch_jwks: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""If a concurrent caller already refreshed JWKS, second caller skips fetch."""
from unittest.mock import patch as obj_patch
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
# First validate raises kid-not-found; second succeeds.
mock_validate.side_effect = [
OIDCKeyNotFoundError("Signing key 'k-rotated' not found"),
{"sub": "u1", "email": "u@example.com", "nonce": "test-nonce"},
]
# Pre-populate the JWKS cache so the rotated kid is already
# present — analog of a concurrent caller having won the lock.
# The retry path must short-circuit and skip the network fetch.
authorize_client.app.state.jwks_data = {"keys": [{"kid": "k-rotated", "kty": "RSA"}]}
with obj_patch("jwt.get_unverified_header", return_value={"kid": "k-rotated"}):
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
mock_fetch_jwks.assert_not_called()
# ---------------------------------------------------------------------------
# Admin OIDC identity endpoint tests
-293
View File
@@ -6,85 +6,6 @@ import time
import pytest
from turnstone.core.storage import StorageConflictError
# ---------------------------------------------------------------------------
# Atomic OIDC user provisioning
# ---------------------------------------------------------------------------
class TestCreateOIDCUser:
def test_create_oidc_user_success(self, db):
"""Both rows present after one atomic call."""
db.create_oidc_user(
user_id="u-new",
username="alice",
display_name="Alice",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-1",
email="alice@example.com",
)
user = db.get_user("u-new")
assert user is not None
assert user["username"] == "alice"
assert user["password_hash"] == "!oidc"
identity = db.get_oidc_identity("https://idp.example.com", "sub-1")
assert identity is not None
assert identity["user_id"] == "u-new"
assert identity["email"] == "alice@example.com"
def test_create_oidc_user_username_conflict_rolls_back(self, db):
"""Pre-existing username -> StorageConflictError; identity NOT inserted."""
db.create_user("u-existing", "alice", "Alice", "$2b$12$hash")
with pytest.raises(StorageConflictError, match="username"):
db.create_oidc_user(
user_id="u-new",
username="alice",
display_name="Alice2",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-1",
email="alice2@example.com",
)
# The new user_id row must not exist.
assert db.get_user("u-new") is None
# The identity row must not exist.
assert db.get_oidc_identity("https://idp.example.com", "sub-1") is None
# The pre-existing user is untouched.
existing = db.get_user("u-existing")
assert existing is not None
assert existing["password_hash"] == "$2b$12$hash"
def test_create_oidc_user_identity_conflict_rolls_back(self, db):
"""Pre-existing (issuer, subject) -> StorageConflictError; user row rolled back."""
db.create_user("u-other", "other", "Other", "!oidc")
db.create_oidc_identity("https://idp.example.com", "sub-1", "u-other", "other@example.com")
with pytest.raises(StorageConflictError, match="OIDC identity"):
db.create_oidc_user(
user_id="u-new",
username="bob",
display_name="Bob",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-1",
email="bob@example.com",
)
# The candidate user row was rolled back.
assert db.get_user("u-new") is None
assert db.get_user_by_username("bob") is None
# The pre-existing identity still points at the original user.
identity = db.get_oidc_identity("https://idp.example.com", "sub-1")
assert identity is not None
assert identity["user_id"] == "u-other"
# ---------------------------------------------------------------------------
# OIDC Identity CRUD
# ---------------------------------------------------------------------------
@@ -383,217 +304,3 @@ class TestOIDCPendingState:
.where(oidc_pending_states.c.state == "state-cleanup")
).scalar()
assert count == 0
# ---------------------------------------------------------------------------
# count_users / find_existing_usernames
# ---------------------------------------------------------------------------
class TestCountUsers:
def test_count_users_empty(self, db):
assert db.count_users() == 0
def test_count_users_after_inserts(self, db):
db.create_user("u1", "alice", "Alice", "h1")
db.create_user("u2", "bob", "Bob", "h2")
db.create_user("u3", "carol", "Carol", "h3")
assert db.count_users() == 3
class TestFindExistingUsernames:
def test_empty_input_returns_empty_set(self, db):
db.create_user("u1", "alice", "Alice", "h1")
assert db.find_existing_usernames([]) == set()
def test_returns_subset_present_in_db(self, db):
db.create_user("u1", "alice", "Alice", "h1")
db.create_user("u2", "bob", "Bob", "h2")
existing = db.find_existing_usernames(["alice", "bob", "carol", "dave"])
assert existing == {"alice", "bob"}
def test_no_matches_returns_empty_set(self, db):
db.create_user("u1", "alice", "Alice", "h1")
assert db.find_existing_usernames(["bob", "carol"]) == set()
# ---------------------------------------------------------------------------
# replace_oidc_roles
# ---------------------------------------------------------------------------
class TestReplaceOIDCRoles:
def _seed_role(self, db, role_id):
db.create_role(role_id, role_id, role_id, "perm.read", False, "")
def test_inserts_added_roles(self, db):
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
added, removed = db.replace_oidc_roles("u1", {"role-a", "role-b"})
assert added == {"role-a", "role-b"}
assert removed == set()
roles = {r["role_id"] for r in db.list_user_roles("u1")}
assert roles == {"role-a", "role-b"}
def test_removes_stale_oidc_roles(self, db):
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
db.assign_role("u1", "role-a", "oidc")
db.assign_role("u1", "role-b", "oidc")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == {"role-b"}
roles = {r["role_id"] for r in db.list_user_roles("u1")}
assert roles == {"role-a"}
def test_preserves_non_oidc_roles(self, db):
"""Manually-assigned and oidc-default rows are NOT touched."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-manual")
self._seed_role(db, "role-default")
self._seed_role(db, "role-oidc-old")
db.assign_role("u1", "role-manual", "admin-ui")
db.assign_role("u1", "role-default", "oidc-default")
db.assign_role("u1", "role-oidc-old", "oidc")
added, removed = db.replace_oidc_roles("u1", set())
# Only the oidc-assigned row was diffed
assert added == set()
assert removed == {"role-oidc-old"}
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {
"role-manual": "admin-ui",
"role-default": "oidc-default",
}
def test_no_op_when_desired_matches_current(self, db):
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "oidc")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
assert {r["role_id"] for r in db.list_user_roles("u1")} == {"role-a"}
def test_empty_user_no_oidc_history(self, db):
db.create_user("u1", "alice", "Alice", "h")
added, removed = db.replace_oidc_roles("u1", set())
assert added == set()
assert removed == set()
def test_desired_role_blocked_by_admin_ui_assignment(self, db):
"""Desired role already held via admin-ui: untouched, no PK conflict."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "admin-ui")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "admin-ui"}
def test_desired_role_blocked_by_oidc_default_assignment(self, db):
"""Desired role already held via oidc-default fallback: untouched."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "oidc-default")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "oidc-default"}
def test_desired_role_added_alongside_blocked_role(self, db):
"""Mixed case: one desired role is blocked (admin-ui), the other inserts cleanly."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
db.assign_role("u1", "role-a", "admin-ui")
added, removed = db.replace_oidc_roles("u1", {"role-a", "role-b"})
assert added == {"role-b"}
assert removed == set()
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "admin-ui", "role-b": "oidc"}
def test_revoke_only_oidc_assigned_roles(self, db):
"""OIDC-assigned roles get revoked when not in desired; admin-ui rows survive."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-manual")
self._seed_role(db, "role-oidc-old")
self._seed_role(db, "role-default")
db.assign_role("u1", "role-manual", "admin-ui")
db.assign_role("u1", "role-oidc-old", "oidc")
db.assign_role("u1", "role-default", "oidc-default")
added, removed = db.replace_oidc_roles("u1", set())
assert added == set()
assert removed == {"role-oidc-old"}
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-manual": "admin-ui", "role-default": "oidc-default"}
def test_replace_oidc_roles_no_op_steady_state(self, db):
"""Steady-state re-login: claims unchanged, function must short-circuit.
This pins the contract that drives the SQLite optimistic-read fast
path the common case (token refresh with identical role claims)
must not acquire a write lock.
"""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
db.assign_role("u1", "role-a", "oidc")
db.assign_role("u1", "role-b", "oidc")
added, removed = db.replace_oidc_roles("u1", {"role-a", "role-b"})
assert added == set()
assert removed == set()
# All rows still oidc-assigned with identical membership.
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "oidc", "role-b": "oidc"}
def test_replace_oidc_roles_returns_post_lock_diff(self, db):
"""Returned (added, removed) reflects the post-lock state, not the optimistic read.
The SQLite implementation re-reads under the write lock to defend
against races; the values returned must come from that re-read so
callers (apply_role_mapping audit logs) see the actual transition
that hit the table. Steady-state input must collapse to empty
sets and leave row timestamps unchanged.
"""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "oidc")
before = db.list_user_roles("u1")
assert len(before) == 1
original_created = before[0]["assignment_created"]
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
# No write occurred — the assignment row's timestamp is untouched.
after = db.list_user_roles("u1")
assert len(after) == 1
assert after[0]["assignment_created"] == original_created
-6
View File
@@ -11,12 +11,6 @@ from turnstone.core.session import ChatSession, _render_template
class NullUI:
"""UI adapter that discards all output."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
-125
View File
@@ -1,125 +0,0 @@
"""Tests for ``AnthropicProvider.extract_reasoning_text``.
Phase 1 of the optional-reasoning-persistence feature: provider-side
extractor that walks stored ``provider_blocks`` and returns the
concatenated thinking text, capped at the operator-friendly UI display
size.
These tests drive through the real ``AnthropicProvider`` instance no
mocks of the extractor itself using fixture-shaped blocks that match
what ``_iter_anthropic_stream`` actually accumulates at
``_anthropic.py:713-724`` (``thinking_delta`` + ``signature_delta``
combined into ``{"type": "thinking", "thinking": <text>, "signature":
<sig>}``).
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
from turnstone.core.providers._protocol import (
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
)
@pytest.fixture
def anthropic() -> AnthropicProvider:
return AnthropicProvider()
class TestExtractReasoningText:
def test_none_input_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
assert anthropic.extract_reasoning_text(None) == ""
def test_empty_list_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
assert anthropic.extract_reasoning_text([]) == ""
def test_no_thinking_blocks_returns_empty(self, anthropic: AnthropicProvider) -> None:
blocks: list[dict[str, object]] = [
{"type": "text", "text": "hello"},
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_single_thinking_block_returns_text(self, anthropic: AnthropicProvider) -> None:
blocks = [{"type": "thinking", "thinking": "Let me think about this.", "signature": "abc"}]
assert anthropic.extract_reasoning_text(blocks) == "Let me think about this."
def test_multiple_thinking_blocks_joined_with_newline(
self, anthropic: AnthropicProvider
) -> None:
blocks = [
{"type": "thinking", "thinking": "first thought", "signature": "s1"},
{"type": "thinking", "thinking": "second thought", "signature": "s2"},
]
assert anthropic.extract_reasoning_text(blocks) == "first thought\nsecond thought"
def test_mixed_blocks_extracts_only_thinking(self, anthropic: AnthropicProvider) -> None:
blocks = [
{"type": "thinking", "thinking": "reason A", "signature": "s"},
{"type": "text", "text": "visible answer"},
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
{"type": "thinking", "thinking": "reason B", "signature": "s"},
]
assert anthropic.extract_reasoning_text(blocks) == "reason A\nreason B"
def test_thinking_block_without_thinking_field_skipped(
self, anthropic: AnthropicProvider
) -> None:
blocks = [{"type": "thinking", "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_thinking_block_with_empty_text_skipped(self, anthropic: AnthropicProvider) -> None:
blocks = [{"type": "thinking", "thinking": "", "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_truncation_at_64kib_cap(self, anthropic: AnthropicProvider) -> None:
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
blocks = [{"type": "thinking", "thinking": long_text, "signature": "s"}]
result = anthropic.extract_reasoning_text(blocks)
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
def test_just_under_cap_not_truncated(self, anthropic: AnthropicProvider) -> None:
text = "y" * (_MAX_REASONING_DISPLAY_CHARS - 1)
blocks = [{"type": "thinking", "thinking": text, "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == text
def test_malformed_block_entry_skipped(self, anthropic: AnthropicProvider) -> None:
# A defensive sanity check — we should not crash if some
# entry isn't a dict (e.g. a corrupted JSON payload).
blocks = [
"not a dict", # type: ignore[list-item]
{"type": "thinking", "thinking": "good one", "signature": "s"},
]
assert anthropic.extract_reasoning_text(blocks) == "good one" # type: ignore[arg-type]
def test_non_list_input_returns_empty(self, anthropic: AnthropicProvider) -> None:
# Defensive against a corrupted provider_data payload.
assert anthropic.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
assert anthropic.extract_reasoning_text({"type": "thinking"}) == "" # type: ignore[arg-type]
class TestOtherProvidersDefault:
"""Non-Anthropic providers return "" for the same fixture shapes."""
def test_openai_chat_returns_empty(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
assert provider.extract_reasoning_text(blocks) == ""
def test_openai_responses_returns_empty(self) -> None:
provider = OpenAIResponsesProvider()
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
assert provider.extract_reasoning_text(blocks) == ""
def test_openai_responses_extracts_reasoning_summary(self) -> None:
# Phase 3: extractor now walks reasoning items captured via
# include=["reasoning.encrypted_content"] and returns the
# summary[*].text concatenation. Pre-Phase-3 this returned
# "" — the stub was replaced once the wire path landed.
provider = OpenAIResponsesProvider()
blocks = [{"type": "reasoning", "summary": [{"type": "summary_text", "text": "x"}]}]
assert provider.extract_reasoning_text(blocks) == "x"
-532
View File
@@ -1,532 +0,0 @@
"""Tests for Phase 2 wire-build replay flag + shape filter on AnthropicProvider.
Phase 2 of optional reasoning persistence wraps the verbatim
``_provider_content`` replay path at ``_anthropic.py:_convert_messages``
with two gates:
1. ``ANTHROPIC_VALID_BLOCK_TYPES`` per-block shape filter foreign-
shaped blocks (OpenAI Responses ``type="reasoning"``, Gemini thought
parts, the synthetic ``reasoning_text`` from path-3 capture) are
dropped individually; valid Anthropic blocks in the same message
still ride the verbatim path. When NO valid blocks survive, the
converter falls through to the text+tool_calls rebuild path.
2. ``replay_reasoning_to_model`` operator flag when False (the
``model_definitions`` server_default), thinking blocks are
stripped before the wire payload is built. Tool_use /
server_tool_use / web_search_tool_result blocks (which carry
web-search ``encrypted_content``) intentionally survive the
strip predicate is narrow by design.
Drives through the real ``AnthropicProvider._convert_messages`` with
fixture-shaped messages, no mocks of the converter. Edge cases come
from the briefing's "Edges & validation memo" sections.
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._anthropic import (
ANTHROPIC_REASONING_BLOCK_TYPES,
ANTHROPIC_VALID_BLOCK_TYPES,
AnthropicProvider,
)
@pytest.fixture
def provider() -> AnthropicProvider:
return AnthropicProvider()
def _assistant_with_thinking(content: str = "Final answer.") -> dict[str, object]:
"""Build an assistant message with a thinking + text + tool_use shape
matching what the streaming layer captures at ``_anthropic.py:713-724``."""
return {
"role": "assistant",
"content": content,
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "sig"},
{"type": "text", "text": content},
{
"type": "tool_use",
"id": "call_abc",
"name": "search",
"input": {"q": "x"},
},
],
"tool_calls": [
{
"id": "call_abc",
"type": "function",
"function": {"name": "search", "arguments": '{"q": "x"}'},
}
],
}
class TestReplayFlagStripsThinking:
"""``replay_reasoning_to_model=False`` strips thinking; ``True`` preserves."""
def test_replay_true_preserves_thinking_block(self, provider: AnthropicProvider) -> None:
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present
assert "text" in types_present
assert "tool_use" in types_present
def test_replay_false_strips_thinking_block(self, provider: AnthropicProvider) -> None:
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present
assert "text" in types_present # final answer survives
assert "tool_use" in types_present # tool dispatch survives
def test_replay_false_strips_redacted_thinking_too(self, provider: AnthropicProvider) -> None:
# Anthropic emits redacted_thinking blocks when the safety system
# rewrites a thinking block. Phase 2 strip predicate must include
# both shapes.
msg = {
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{"type": "redacted_thinking", "data": "redacted-blob"},
{"type": "text", "text": "Answer."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "redacted_thinking" not in types_present
assert "text" in types_present
def test_default_kwarg_preserves_existing_behaviour(self, provider: AnthropicProvider) -> None:
"""Pre-Phase-2 callers that don't pass the kwarg get the verbatim
replay (default True), matching the behaviour all production
Anthropic-with-thinking turns shipped with for months."""
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg]) # no kwarg
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present
class TestWebSearchBlocksSurviveStrip:
"""Edge 14: Anthropic web-search ``encrypted_content`` rides on
``server_tool_use`` / ``web_search_tool_result`` blocks (NOT
thinking blocks). Strip predicate is intentionally narrow."""
def test_server_tool_use_survives(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "From search: ...",
"_provider_content": [
{"type": "thinking", "thinking": "I should search", "signature": "s"},
{
"type": "server_tool_use",
"id": "stu_1",
"name": "web_search",
"input": {"query": "turnstone bird"},
},
{
"type": "web_search_tool_result",
"tool_use_id": "stu_1",
"content": [{"type": "web_search_result", "url": "https://e.com"}],
"encrypted_content": "abc123encrypted",
"encrypted_index": "idx456encrypted",
},
{"type": "text", "text": "From search: ..."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present # stripped
assert "server_tool_use" in types_present # survives
assert "web_search_tool_result" in types_present # survives
assert "text" in types_present # survives
# encrypted_content rides through intact — required for round-trip continuity
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
assert wsr["encrypted_content"] == "abc123encrypted"
def test_tool_use_block_survives(self, provider: AnthropicProvider) -> None:
# Plain tool_use (not server-side) — used by client-side function
# tools. Strip predicate must not touch these.
msg = {
"role": "assistant",
"content": "Calling tool",
"_provider_content": [
{"type": "thinking", "thinking": "I should call tool", "signature": "s"},
{"type": "tool_use", "id": "tu_1", "name": "f", "input": {"a": 1}},
],
"tool_calls": [
{
"id": "tu_1",
"type": "function",
"function": {"name": "f", "arguments": '{"a": 1}'},
}
],
}
# Provide the tool result so orphan-tool detection doesn't synthesize
msgs = [
msg,
{"role": "tool", "tool_call_id": "tu_1", "content": "ok"},
]
_, converted = provider._convert_messages(msgs, replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present
assert "tool_use" in types_present
def test_orphan_tool_use_synthesized_after_strip(self, provider: AnthropicProvider) -> None:
"""Pin the post-strip orphan-tool branch at _anthropic.py:397-433.
The implementation comment specifically calls out reading
``provider_content`` (not ``wire_blocks``) for the orphan-tool
ID walk after the strip keeping the read on the source-of-
truth list so a future refactor that swapped them would still
get the same set of tool_use IDs. This test exercises that
branch end-to-end: replay=False strips the thinking block,
AND the message has a tool_use whose result is missing. The
converter must synthesize a 'cancelled' tool_result for the
orphaned tool_use ID (matching the existing pre-Phase-2
behaviour for the verbatim path).
"""
msg = {
"role": "assistant",
"content": "Calling tool",
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "s"},
{"type": "tool_use", "id": "orphan_tu", "name": "f", "input": {"a": 1}},
],
"tool_calls": [
{
"id": "orphan_tu",
"type": "function",
"function": {"name": "f", "arguments": '{"a": 1}'},
}
],
}
# NO tool result follows — orphan branch must synthesize one.
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
# Synthetic tool_result lands as a user-role message immediately
# after the assistant turn (per existing behaviour at
# _anthropic.py:421-430).
assistant = next(m for m in converted if m["role"] == "assistant")
# Stripped: thinking gone, tool_use survives.
a_types = [b["type"] for b in assistant["content"]]
assert "thinking" not in a_types
assert "tool_use" in a_types
# Synthesized: cancelled tool_result for orphan_tu attached to a
# following user-role message.
user_msgs_after = [m for m in converted if m["role"] == "user"]
assert user_msgs_after, (
"Expected a synthetic user message carrying the cancelled "
"tool_result for the orphaned tool_use"
)
flat_results = [
block
for um in user_msgs_after
if isinstance(um["content"], list)
for block in um["content"]
if isinstance(block, dict) and block.get("type") == "tool_result"
]
synth = next(
(b for b in flat_results if b.get("tool_use_id") == "orphan_tu"),
None,
)
assert synth is not None, f"Expected synthetic tool_result for orphan_tu in {flat_results}"
assert synth.get("is_error") is True
assert "cancelled" in synth.get("content", "").lower()
class TestShapeFilterFallthrough:
"""Foreign / empty / mixed-shape ``_provider_content`` falls through
to the text+tool_calls rebuild path rather than reaching the wire
as a malformed block."""
def test_foreign_shape_openai_reasoning_falls_through(
self, provider: AnthropicProvider
) -> None:
# OpenAI Responses style block (Phase 3 will land this shape into
# _provider_content via include=["reasoning.encrypted_content"]).
# Mid-workstream model switch from OpenAI -> Anthropic must NOT
# reach the API with an OpenAI-shaped block (which would 400).
msg = {
"role": "assistant",
"content": "Final answer from openai turn.",
"_provider_content": [
{
"type": "reasoning",
"summary": [{"type": "summary_text", "text": "I reasoned..."}],
"encrypted_content": "openai-encrypted",
}
],
"tool_calls": [],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
# Rebuilt from text — no foreign block reached the wire.
for b in assistant["content"]:
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES, (
f"Foreign block type leaked through: {b}"
)
# And the foreign block specifically is NOT present.
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present
def test_mixed_shape_drops_foreign_keeps_valid(self, provider: AnthropicProvider) -> None:
# Per-block filter: a single foreign block in a mostly-Anthropic
# payload no longer forces fall-through. Valid Anthropic blocks
# ride the verbatim path; the foreign block is dropped.
msg = {
"role": "assistant",
"content": "Mixed.",
"_provider_content": [
{"type": "thinking", "thinking": "anth shape", "signature": "s"},
{"type": "text", "text": "Mixed."},
{"type": "reasoning", "summary": []}, # foreign
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present # foreign dropped
assert "thinking" in types_present # valid + replay=True kept
assert "text" in types_present
for b in assistant["content"]:
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES
def test_mixed_shape_preserves_web_search_encrypted_content(
self, provider: AnthropicProvider
) -> None:
# The motivating case for per-block (vs all-or-nothing) filter:
# cross-model resumption stamps a foreign ``reasoning`` block
# alongside Anthropic web-search blocks carrying encrypted
# citations. An all-or-nothing filter would discard the whole
# message and rebuild from text+tool_calls — silently losing
# the encrypted_content the API needs for round-trip continuity.
msg = {
"role": "assistant",
"content": "From search: ...",
"_provider_content": [
{"type": "reasoning", "summary": []}, # foreign (e.g. OpenAI)
{
"type": "server_tool_use",
"id": "stu_1",
"name": "web_search",
"input": {"query": "x"},
},
{
"type": "web_search_tool_result",
"tool_use_id": "stu_1",
"content": [{"type": "web_search_result", "url": "https://e.com"}],
"encrypted_content": "encrypted-blob-must-survive",
"encrypted_index": "encrypted-idx-must-survive",
},
{"type": "text", "text": "From search: ..."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present # foreign dropped
assert "server_tool_use" in types_present
assert "web_search_tool_result" in types_present
assert "text" in types_present
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
assert wsr["encrypted_content"] == "encrypted-blob-must-survive"
assert wsr["encrypted_index"] == "encrypted-idx-must-survive"
def test_all_foreign_blocks_fall_through_to_rebuild(self, provider: AnthropicProvider) -> None:
# When every block is foreign-shaped (no Anthropic-valid block
# survives the per-block filter), the converter still falls
# through to text+tool_calls rebuild rather than emitting an
# empty assistant turn.
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "reasoning", "summary": []},
{"type": "reasoning_text", "text": "synthetic"}, # path-3 shape
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
# Rebuild path: msg.content lifted into a single text block.
assert assistant["content"] == [{"type": "text", "text": "Final answer."}]
def test_empty_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "Plain text answer.",
"_provider_content": [],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
# Falls through to text rebuild
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
def test_none_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "Plain text answer.",
"_provider_content": None,
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
def test_provider_content_not_a_list_falls_through(self, provider: AnthropicProvider) -> None:
# Defensive against a corrupted provider_data deserialization.
msg = {
"role": "assistant",
"content": "Plain.",
"_provider_content": "not a list",
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Plain."}]
def test_non_dict_and_missing_type_blocks_are_dropped(
self, provider: AnthropicProvider
) -> None:
# Defensive branches in the per-block walk: a stray non-dict
# element (corrupted JSON) or a dict with no/None ``type`` key
# (provider drift) must be silently dropped without raising.
# Valid blocks in the same list still ride the verbatim path.
msg = {
"role": "assistant",
"content": "ok",
"_provider_content": [
{"type": "text", "text": "ok"},
"stray-string", # non-dict
{"type": None, "text": "huh"}, # None type
{"no_type_key": 1}, # missing type
{"type": "thinking", "thinking": "t", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b.get("type") for b in assistant["content"]]
assert types_present == ["text", "thinking"]
class TestLegacyAnthropicRowsNoRegression:
"""Critical property: rows persisted before Phase 2 carry valid
Anthropic-shape _provider_content (only Anthropic captured this lane
historically). They must stay in the verbatim path and keep their
thinking context across the migration boundary when replay=True
(the legacy default).
"""
def test_legacy_thinking_row_preserved_with_default_kwarg(
self, provider: AnthropicProvider
) -> None:
"""No kwarg passed (matches the pre-Phase-2 production call site)."""
msg = {
"role": "assistant",
"content": "Old answer from months ago.",
"_provider_content": [
{"type": "thinking", "thinking": "old reasoning", "signature": "s"},
{"type": "text", "text": "Old answer from months ago."},
],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present # preserved -> no regression
# The thinking block IS the same dict as the source (verbatim path).
assert assistant["content"][0]["thinking"] == "old reasoning"
def test_replay_false_only_strips_when_explicitly_requested(
self, provider: AnthropicProvider
) -> None:
# Operator flips persist+replay flags off. Strip fires.
# Pinning that the strip is gated on the explicit flag value,
# not silently triggered by some other condition.
msg = {
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{"type": "thinking", "thinking": "stripped", "signature": "s"},
{"type": "text", "text": "Answer."},
],
}
_, converted_default = provider._convert_messages([msg])
_, converted_strip = provider._convert_messages([msg], replay_reasoning_to_model=False)
default_types = [b["type"] for b in converted_default[0]["content"]]
strip_types = [b["type"] for b in converted_strip[0]["content"]]
assert "thinking" in default_types
assert "thinking" not in strip_types
class TestStripAllBlocksFallthrough:
"""When the message is 100% thinking (no text, no tool_use) and
replay=False strips everything, the message falls through to the
text+tool_calls rebuild path. If both are also empty, the assistant
turn is silently skipped correct: stripped reasoning has nothing
to replay."""
def test_only_thinking_strip_falls_to_rebuild_with_text(
self, provider: AnthropicProvider
) -> None:
# Provider_content = only thinking; msg.content has the spoken text.
# Strip drops thinking; rebuild path picks up the content as a
# text block. No information lost.
msg = {
"role": "assistant",
"content": "Spoken answer.",
"_provider_content": [
{"type": "thinking", "thinking": "internal", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Spoken answer."}]
def test_only_thinking_strip_with_no_content_skips_message(
self, provider: AnthropicProvider
) -> None:
# Edge: provider_content was 100% thinking AND msg.content is
# empty AND no tool_calls. The rebuild path sees nothing to
# emit — assistant turn silently skipped. Anthropic's API
# would reject an empty assistant content array anyway.
msg = {
"role": "assistant",
"content": "",
"_provider_content": [
{"type": "thinking", "thinking": "only", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
# Assistant turn skipped — no entry for it in `converted`.
assert all(m["role"] != "assistant" for m in converted)
class TestConstants:
"""Pin the constant contents so a future edit doesn't accidentally
widen the strip set or narrow the valid set."""
def test_reasoning_block_types_is_narrow(self) -> None:
# Strip predicate MUST cover only reasoning shapes. Adding
# tool_use here would break web-search round-trip.
assert frozenset({"thinking", "redacted_thinking"}) == ANTHROPIC_REASONING_BLOCK_TYPES
def test_valid_block_types_includes_web_search(self) -> None:
# Without server_tool_use / web_search_tool_result, Anthropic
# web-search results would fall through to the rebuild path
# and lose their encrypted_content.
assert "server_tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
assert "web_search_tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
assert "tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
assert "tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
def test_reasoning_subset_of_valid(self) -> None:
# The strip set must be a subset of the valid set — otherwise
# the strip predicate would never match anything (we only
# strip after shape validity passes).
assert ANTHROPIC_REASONING_BLOCK_TYPES.issubset(ANTHROPIC_VALID_BLOCK_TYPES)
@@ -1,320 +0,0 @@
"""Tests for OpenAI Responses reasoning capture + replay (Phase 3 path 2).
Phase 3 wires:
1. ``include=["reasoning.encrypted_content"]`` on the request when
the operator flag AND the model capability both allow.
2. ``_convert_messages`` round-tripping stored reasoning items as
``ResponseReasoningItemParam`` input items on subsequent turns.
3. ``OpenAIResponsesProvider.extract_reasoning_text`` walking
reasoning items and returning concatenated summary + content text.
All tests drive through the real ``OpenAIResponsesProvider`` no
mocks of the converter/build_kwargs themselves; only the SDK boundary
is mocked where relevant.
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._openai_responses import (
OpenAIResponsesProvider,
_reasoning_item_for_input,
)
from turnstone.core.providers._protocol import (
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
)
from turnstone.core.providers._protocol import ModelCapabilities
@pytest.fixture
def provider() -> OpenAIResponsesProvider:
return OpenAIResponsesProvider()
def _capable_caps() -> ModelCapabilities:
"""Capability fixture for a reasoning-replay-capable model."""
return ModelCapabilities(
context_window=400000,
max_output_tokens=128000,
supports_temperature=False,
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
supports_reasoning_replay=True,
)
class TestExtractReasoningText:
def test_none_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
assert provider.extract_reasoning_text(None) == ""
def test_empty_list_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
assert provider.extract_reasoning_text([]) == ""
def test_no_reasoning_items_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
blocks = [
{"type": "message", "role": "assistant", "content": "hi"},
{"type": "function_call", "call_id": "c1", "name": "x", "arguments": "{}"},
]
assert provider.extract_reasoning_text(blocks) == ""
def test_summary_text_extracted(self, provider: OpenAIResponsesProvider) -> None:
# Per ResponseReasoningItem (response_reasoning_item.py:31-62):
# summary is always present; content is optional.
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [
{"type": "summary_text", "text": "I considered X"},
{"type": "summary_text", "text": "then Y"},
],
}
]
assert provider.extract_reasoning_text(blocks) == "I considered X\nthen Y"
def test_content_text_extracted_alongside_summary(
self, provider: OpenAIResponsesProvider
) -> None:
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "summary line"}],
"content": [{"type": "reasoning_text", "text": "raw reasoning"}],
}
]
# Order: summary first, then content (matches the order the SDK
# surfaces them via streaming events).
result = provider.extract_reasoning_text(blocks)
assert "summary line" in result
assert "raw reasoning" in result
def test_truncation_at_64kib_cap(self, provider: OpenAIResponsesProvider) -> None:
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": long_text}],
}
]
result = provider.extract_reasoning_text(blocks)
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
def test_malformed_summary_entry_skipped(self, provider: OpenAIResponsesProvider) -> None:
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [
"not a dict",
{"type": "summary_text"}, # missing text
{"type": "summary_text", "text": ""}, # empty text
{"type": "summary_text", "text": "good"},
],
}
]
assert provider.extract_reasoning_text(blocks) == "good"
def test_non_list_input_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
assert provider.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
def test_other_block_types_skipped_in_walk(self, provider: OpenAIResponsesProvider) -> None:
# Mixed payload: only the reasoning block contributes.
blocks = [
{"type": "message", "role": "assistant", "content": "hi"},
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "thought"}],
},
{"type": "function_call", "call_id": "c1", "name": "x", "arguments": "{}"},
]
assert provider.extract_reasoning_text(blocks) == "thought"
class TestReasoningItemForInput:
"""``_reasoning_item_for_input`` projects a stored ``ResponseReasoningItem``
dict into ``ResponseReasoningItemParam`` shape (drops server-only
``status``)."""
def test_minimal_item_round_trip(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
"status": "completed",
}
result = _reasoning_item_for_input(stored)
assert result["type"] == "reasoning"
assert result["id"] == "r_1"
assert result["summary"] == [{"type": "summary_text", "text": "x"}]
# status NOT round-tripped (server-only field per
# ResponseReasoningItemParam at response_reasoning_item_param.py).
assert "status" not in result
def test_encrypted_content_round_trips_when_present(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
"encrypted_content": "opaque-blob",
}
result = _reasoning_item_for_input(stored)
assert result["encrypted_content"] == "opaque-blob"
def test_encrypted_content_omitted_when_absent(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
}
result = _reasoning_item_for_input(stored)
assert "encrypted_content" not in result
def test_content_round_trips_when_present(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "s"}],
"content": [{"type": "reasoning_text", "text": "raw"}],
}
result = _reasoning_item_for_input(stored)
assert result["content"] == [{"type": "reasoning_text", "text": "raw"}]
class TestBuildKwargsInclude:
"""``_build_kwargs`` adds ``include=["reasoning.encrypted_content"]``
when the resolved operator flag is True. The capability AND-gate
lives upstream in ``ChatSession._resolve_replay_reasoning_to_model``
(single source of truth across providers); the provider trusts the
bool it receives. See
``test_session_replay_reasoning.py::TestSessionToOpenAIResponsesBoundaryIntegration``
for the end-to-end gate test."""
def test_include_added_when_flag_true(self, provider: OpenAIResponsesProvider) -> None:
kwargs = provider._build_kwargs(
model="gpt-5",
messages=[{"role": "user", "content": "hi"}],
tools=None,
max_tokens=1024,
temperature=0.5,
reasoning_effort="medium",
deferred_names=None,
capabilities=_capable_caps(),
replay_reasoning_to_model=True,
)
assert kwargs.get("include") == ["reasoning.encrypted_content"]
def test_include_omitted_when_flag_false(self, provider: OpenAIResponsesProvider) -> None:
kwargs = provider._build_kwargs(
model="gpt-5",
messages=[{"role": "user", "content": "hi"}],
tools=None,
max_tokens=1024,
temperature=0.5,
reasoning_effort="medium",
deferred_names=None,
capabilities=_capable_caps(),
replay_reasoning_to_model=False,
)
assert "include" not in kwargs
class TestConvertMessagesReasoningReplay:
"""``_convert_messages`` round-trips stored reasoning items as input."""
def test_reasoning_item_emitted_before_assistant_when_replay_true(
self, provider: OpenAIResponsesProvider
) -> None:
messages = [
{"role": "user", "content": "explain"},
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "I thought"}],
"encrypted_content": "abc",
}
],
},
{"role": "user", "content": "follow up"},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
# Find the reasoning input item.
types = [it.get("type") for it in items]
# Expected: user, reasoning, message (assistant), user.
assert types == ["message", "reasoning", "message", "message"]
reasoning_idx = types.index("reasoning")
r_item = items[reasoning_idx]
assert r_item["id"] == "r_1"
assert r_item["encrypted_content"] == "abc"
# And the reasoning item appears immediately BEFORE the
# assistant message it belongs to.
assert items[reasoning_idx + 1]["role"] == "assistant"
def test_reasoning_item_dropped_when_replay_false(
self, provider: OpenAIResponsesProvider
) -> None:
messages = [
{
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "thought"}],
}
],
},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=False)
types = [it.get("type") for it in items]
assert "reasoning" not in types
def test_no_reasoning_items_when_provider_content_lacks_reasoning(
self, provider: OpenAIResponsesProvider
) -> None:
# Anthropic-shaped _provider_content reaching OpenAI Responses
# (cross-provider — operator switch from Anthropic to GPT-5):
# no type=="reasoning" items, so nothing emitted.
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "anth", "signature": "s"},
],
},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
types = [it.get("type") for it in items]
assert "reasoning" not in types
def test_default_replay_reasoning_false_omits_reasoning(
self, provider: OpenAIResponsesProvider
) -> None:
# Pre-Phase-3 callers (no kwarg) get the back-compat behaviour:
# reasoning items are silently dropped (sanitize_messages was
# already stripping _provider_content anyway).
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
}
],
},
]
_, items = provider._convert_messages(messages) # no kwarg
types = [it.get("type") for it in items]
assert "reasoning" not in types

Some files were not shown because too many files have changed in this diff Show More