Compare commits

...

194 Commits

Author SHA1 Message Date
Patrick Buckley 1de8f6b4c7 chore: bump version to 1.5.15 2026-05-12 21:21:15 -07:00
Patrick Buckley 4f89c1c3f3 docs(changelog): release 1.5.15 notes
Fixes the 1.5.14 admin.js syntax error that left the console UI
non-functional whenever an MCP server row had consented users.
2026-05-12 21:21:08 -07:00
Patrick Buckley 6898860f18 fix(mcp): admin.js syntax error in bulk-revoke button (Phase 9)
Line 3446 used double-quote string delimiters with an embedded ">
that terminated the string mid-attribute, leaving "bulk-revoke (" as
bare tokens.  The rest of the surrounding block uses single-quote
delimiters; switch the broken line to match so the embedded > and "
sit safely inside the string.

The parse error wiped out every global in admin.js, so showAdmin and
the rest of the admin entry points were undefined — the console was
non-functional whenever an MCP server row had consented_users_count > 0.
2026-05-12 21:20:45 -07:00
Patrick Buckley 3d07cad272 chore: bump version to 1.5.14 2026-05-12 21:17:35 -07:00
Patrick Buckley 3737ddf89f docs(changelog): release 1.5.14 notes
Backports OAuth-MCP Phase 9 (#516) to the stable/1.5 track. Introduces
forward-only migrations 054_mcp_pending_consent and
055_mcp_user_tokens_server_index.
2026-05-12 21:17:26 -07:00
Patrick Buckley 464450b9e2 feat(mcp): admin status, deferred-consent persistence, operator docs (Phase 9) (#516)
* feat(mcp): admin status, deferred-consent persistence, operator docs (Phase 9)

Completes the OAuth-MCP build-out (Phases 0-8 shipped) by closing the
operator + deferred-consent gaps:

1. **Per-(user, server) deferred-consent persistence** — when a
   non-interactive run (scheduled / channel) hits ``mcp_consent_required``
   or ``mcp_insufficient_scope``, the sync pool dispatchers now upsert a
   row into a new ``mcp_pending_consent`` table.  The dashboard hydrates
   the gear-icon badge from this table on load, so users who weren't
   online to see the in-flight SSE prompt still surface the deferred
   work on next login.  Cleared automatically by the OAuth callback
   handler on consent completion; manual user dismiss via new DELETE
   endpoints.  Composite PK ``(user_id, server_name)`` collapses repeat
   occurrences for the same server — no NULLs-not-distinct trap.

2. **Admin status pill + bulk-revoke** — the MCP Servers admin row now
   shows ``consented_users_count`` for ``auth_type=oauth_user`` rows
   when ≥1, with a two-step-confirm ``bulk-revoke`` button that drops
   every user's token for the server via the existing
   ``delete_mcp_oauth_rows_by_server_name`` primitive.  Upstream RFC
   7009 revoke is intentionally NOT attempted in bulk (avoids N
   upstream HTTP calls per admin click); audit detail records
   ``upstream_revoke_outcome=bulk_admin_no_upstream``.  A "last
   refresh" pill (age + outcome) renders on each row, sourced from a
   new ``_last_refresh`` dict populated by ``_refresh_server`` on every
   call (both manual ``refresh_sync`` and the ``_cb_auto_reconnect``
   follow-up).

3. **ClientType.SCHEDULED** added to the prompts module + scheduler
   passes it through to ``create_workstream``.  ``ChatSession`` now
   computes ``_is_interactive_for_consent`` at construction (WEB / CLI
   are interactive; CHAT / SCHEDULED are not) and plumbs the flag
   through ``call_tool_sync`` / ``read_resource_sync`` /
   ``get_prompt_sync`` to the three sync dispatchers.  The wrap at the
   ``_is_structured_error`` gate routes consent codes to the new
   ``_record_pending_consent_best_effort`` helper for non-interactive
   callers only; interactive sessions stay on the in-flight SSE path
   Phase 8 ships unchanged.

4. **Operator docs** — ``docs/mcp-oauth.md`` (operator guide, parallel
   to ``docs/oidc.md``: ``auth_type`` choice, OAuth client setup,
   encryption-key rotation, troubleshooting matrix) and
   ``docs/operations/mcp-oauth-headless.md`` (one-paragraph runbook
   per ``feedback_runbook_trust_llm.md``: pre-consent recipe for
   scheduled / channel-driven runs).

Schema
- Migration 054_mcp_pending_consent.py — composite PK
  ``(user_id, server_name)``, ``occurrence_count`` + ``first_seen_at`` /
  ``last_seen_at`` for recency metadata, ``idx_mcp_pending_consent_user``
  for the badge-load query.  No FKs (matches the rest of the
  oauth_user schema).
- Migration 055_mcp_user_tokens_server_index.py — adds
  ``idx_mcp_user_tokens_server`` on ``(server_name, expires_at)`` so
  the admin pill's ``count_mcp_consented_users_*`` queries don't
  full-scan against the leading-``user_id`` composite PK.
- Cross-backend: works on SQLite + PostgreSQL via dialect-specific
  ``on_conflict_do_update`` (PG ``postgresql.insert`` / SQLite
  ``sqlalchemy.dialects.sqlite.insert``).  No ``NULLS NOT DISTINCT``
  needed — the simplified PK eliminates the cross-version trap.

Endpoints
- ``GET /v1/api/mcp/oauth/pending`` — list deferred-consent records for
  the authenticated user.  Install-level gate via cached
  ``any_oauth_user_mcp_servers`` short-circuits to ``{pending: 0}`` on
  installs with no oauth_user MCP servers — local-auth deployments
  exercise zero new storage queries on this path.  The gate result is
  cached on ``app.state`` with a 60s TTL to spare repeat dashboard
  loads.
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` — single dismiss.
  Returns 204 in both existed-and-deleted and never-existed cases
  (no cross-tenant existence leak); audits
  ``mcp_server.oauth.pending_consent_dismissed`` with
  ``mode=single`` + ``cleared=0|1`` so a session-hijack attacker
  scrubbing breadcrumbs leaves an audit trail.
- ``DELETE /v1/api/mcp/oauth/pending`` — bulk dismiss; audits
  ``mode=bulk`` + ``cleared=N``.
- ``POST /v1/api/admin/mcp-servers/{name}/bulk-revoke`` — admin
  bulk-revoke for the named server's per-user tokens.  Requires
  ``admin.mcp`` permission + 400s when the row isn't ``oauth_user``.

All four registered on both ``turnstone-server`` and
``turnstone-console`` (mirrors the Phase 8 ``/connections`` endpoint
shape).

Performance
- Admin list handler now uses a single ``GROUP BY`` bulk-count query
  (``count_mcp_consented_users_grouped_by_server``) wrapped in
  ``asyncio.to_thread`` rather than N per-row sync DB round-trips
  inside the async handler.  Skipped entirely when no row is
  oauth_user.

Frontend
- ``ui/static/app.js``: ``loadPendingConsents()`` hydrates the
  existing ``_pendingConsentServers`` set on dashboard init + after
  the user opens the settings modal.  Endpoint failures stay silent
  — the badge will be re-driven by the next in-flight tool error.
- ``console/static/admin.js``: ``consented_users_count`` pill +
  ``bulk-revoke`` button on each MCP row (only when ≥1 consented),
  two-step confirm matching the existing delete pattern.  ``last-
  refresh`` age + outcome pill in the per-row status cell, sourced
  from the freshest per-node entry in ``status[*].last_refresh_at`` /
  ``last_refresh_outcome``.  CSS for the pills in ``style.css``.

Tests
- ``test_mcp_pending_consent_storage`` — 13 tests covering upsert
  idempotency, list ordering, per-user isolation, single/bulk delete,
  count-by-server + grouped variant, install-level gate.
- ``test_mcp_pending_consent_dispatch`` — 9 tests, including the
  boundary-cross gate per ``feedback_tests_through_boundaries.md``:
  drives the real ``call_tool_sync`` → ``_dispatch_pool_sync`` →
  ``_is_structured_error`` → ``_record_pending_consent_best_effort``
  with a mocked classified-lookup so the structural plumb-through is
  verified end-to-end.  Includes a storage-failure test that pins
  the docstring's "envelope unchanged on storage failure" promise.
- ``test_mcp_pending_consent_endpoints`` — 11 tests: install gate,
  list-for-self, no-cross-user-leak, single/bulk delete, idempotent
  not-found, audit emission on single + bulk + cross-tenant dismiss.
- ``test_chat_session_interactivity_flag`` — 7 tests pinning the
  ``ClientType`` → ``_is_interactive_for_consent`` mapping against
  the module-level ``INTERACTIVE_CONSENT_CLIENT_TYPES`` frozenset.
- ``test_mcp_admin_bulk_revoke`` — 7 tests covering admin.mcp
  permission gate, 404 on missing, 400 on non-oauth_user, 200 with
  ``rows_deleted`` + ``consented_users_before``, audit row with
  ``upstream_revoke_outcome=bulk_admin_no_upstream``, cross-server
  isolation.
- ``test_mcp_oauth_handlers`` — 2 new callback tests pin the post-
  callback ``delete_mcp_pending_consent`` invocation: success-clears
  + storage-failure-still-redirects.
- 636 tests pass on the impacted surface (47 new + Phase 0-8 OAuth-MCP
  + session + prompts + storage admin).  ruff + mypy clean.

Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}`` — the
  flag flows only through the pool dispatchers, which only fire when
  the row resolves to ``oauth_user``.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) preserved on every
  AS / SDK / pool-loop await — no new awaits added to the hot path.
- Install-level gate on the badge endpoint: cached
  ``any_oauth_user_mcp_servers`` returns False on a row-less
  deployment → endpoint short-circuits without touching the pending-
  consent table; 60s TTL bounds the staleness window after admin
  flips ``auth_type``.
- Operator-actionable codes (key-unknown, url-insecure, *_forbidden)
  explicitly filtered out of persistence — they're outside the
  user-facing consent badge scope.
- Best-effort write: the structured-error envelope returned to the
  agent is identical whether the persistence write succeeds or fails
  (storage exception is logged with type name only — no chained
  context that could carry an ``httpx.Request`` bearer header).
- No ``exc_info=True`` on any new path that can chain a bearer-bearing
  ``httpx.Request``.
- Defensive parsing: ``_parse_pending_consent_envelope`` mirrors
  ``_is_structured_error``'s ``isinstance(decoded, dict)`` guard plus
  filters scope tokens through ``is_valid_scope_token`` capped at
  ``MAX_INSUFFICIENT_SCOPE_REPORTED`` — defense-in-depth even though
  production callers already validate upstream.
- Audit events on every dismiss endpoint so a session-control attacker
  scrubbing dashboard breadcrumbs still leaves a trail.

Cross-backend
- Tested on SQLite via the conftest backend fixture.
- PostgreSQL path uses ``postgresql.insert(...).on_conflict_do_update``
  parallel to the existing ``mcp_user_tokens`` upsert in Phase 3.

Deferred (not Phase 9 blockers)
- Multi-node pool eviction on bulk-revoke: only local-node sessions
  would be evicted if we built it, and there's no bulk-by-server
  primitive on MCPClientManager today; remote nodes will surface as
  a 401 on next dispatch which refreshes through the (now empty)
  token row.
- RFC 8693 / Azure OBO ``auth_type=oauth_token_exchange`` — captured
  in the design doc as a future architectural direction (~600 LOC +
  IdP-side admin work); requires OIDC token capture and per-MCP-server
  resource-trust configuration that v1 does not ship.

* docs(mcp): address Copilot review feedback on Phase 9

- Fix misleading admin.js comment that claimed the refresh pill rendered
  "<short-relative> <outcome>" — the pill actually renders only the short
  age, with outcome reflected via CSS class and tooltip.
- Replace broken feedback_secrets_not_in_env.md repo-root link in
  mcp-oauth.md with the inlined rationale (env-borne secrets reachable
  via shell tools / os.environ; TOML secrets are not).
2026-05-12 20:55:01 -07:00
Patrick Buckley 1fba80a9b5 chore: bump version to 1.5.13 2026-05-11 21:24:09 -07:00
Patrick Buckley 28a5914be0 docs(changelog): release 1.5.13 notes
Adds notes for the 14 patches cherry-picked to stable/1.5 since 1.5.12:
reactive PG LISTEN/NOTIFY node discovery + event-driven wait_for_workstream,
memory tool audit trail, task_agent skill personas, plus fixes for the
LLM-visible default alias bypass, mermaid streaming parse errors,
proxy-prefixed re-auth, dashboard appbar visibility, and the PG test
backend on the notify dispatcher suite.

Introduces forward-only migration 053_services_notify_trigger.
2026-05-11 21:23:53 -07:00
Patrick Buckley bb2515eddf fix(task_agent): address Copilot feedback on skill parameter
- Put ``skill`` back in the access-denial list in the tool
  description with a clarification — TASK_AGENT_TOOLS does not
  include the skill tool, so sub-agents cannot switch personas
  mid-task.  Removing the disclaimer entirely created an ambiguity
  the LLM could misread.

- Minimize the skill_data carried on the approval item dict to
  ``name`` / ``content`` / ``risk_level`` only.  ``get_skill_by_name``
  returns the full ~30-column prompt_templates row including
  ``scan_report``, ``installed_by``, ``source_url`` — none of those
  flow through ``_exec_task`` / ``_evaluate_intent``, and they
  shouldn't ride along any future audit serializer that reads the
  approval item shape.

- Regression test for ``skill=""``, whitespace-only, and ``\t\n``
  values — pins the documented "empty value is acceptable" contract
  at the ``(args.get("skill") or "").strip()`` chokepoint.
2026-05-11 21:18:48 -07:00
Patrick Buckley 19978bfd89 feat(task_agent): add optional skill parameter for per-call personas
The task_agent tool now accepts an optional ``skill=<name>`` argument
that loads the named skill's content as the sub-agent's persona,
substituting the hardcoded "# Task Agent" identity statement.  The
operating-guidance numbered list (one-shot, tool-use over narration,
no follow-up questions) is layered on top of every persona and always
applies — those are sub-agent semantics that a persona should ride on
top of, not replace.

Validation lives in ``_prepare_task`` so the approval surface tells
the operator what they're consenting to: the validated skill dict
(including content) rides on the item dict from prepare to exec to
defeat TOCTOU between consent and execution.  An unknown skill
returns a clean error item with a hint pointing at
``skill(action='search')``; a disabled skill returns a distinct error
so the LLM's recovery path can tell "not found" from "quarantined",
mirroring the enabled gate that ``_exec_skill(action='load')`` and
skill-search already apply.

High and critical skills now surface their risk tier on the approval
header (``, risk: critical``) and emit a
``task_agent.high_risk_skill`` warning — same signal ``_load_skills``
emits for session-level skills, so the operator sees the same flag
whether the skill is loaded session-wide or per-call.  ``_exec_task``
emits a ``task_agent.skill_invoked`` info log on the skill branch for
forensic traceability — the approval row captures the choice at
consent time, this log captures it at exec time so post-incident
search doesn't have to cross-walk approval and exec tables.

The ``_evaluate_intent`` func_args projection now includes the skill
name — without it, heuristic ``arg_pattern`` rules targeting a risky
persona name on ``task_agent`` silently no-op and the audit row loses
the choice.  Mirrors the long-standing ``spawn_workstream``
projection.
2026-05-11 21:18:48 -07:00
Patrick Buckley 8eec44d809 fix(ui): attach settings menu keydown synchronously
Caught by Copilot on PR #514.  openSettingsMenu sets _settingsMenu
synchronously, but the menu's keydown handler was registered inside
setTimeout(0).  The previous-commit guard in the global keydown
handler returns early when _settingsMenu is set (so dashboard isn't
hidden by Escape over the menu), which created a window where
Escape had no handler at all — the global skipped, the menu's own
listener wasn't ready yet, and the menu got stuck open until the
next interaction.

Attach keydown synchronously; keep mousedown + initial focus in
setTimeout (mousedown to avoid the opening click triggering its own
outside-click close, focus because the menu DOM needs a tick to
settle layout).
2026-05-11 21:18:48 -07:00
Patrick Buckley f1cf516eb6 fix(ui): keep appbar visible on dashboard, gear-icon dropdown menu
Two related changes that surfaced when the user pointed out the proxy's
node-picker pill was unreachable from the proxied dashboard view: the
dashboard overlay was covering the entire appbar.

  - Dashboard overlay now starts at top: 48px so the appbar (with the
    proxy-injected node picker) stays visible and interactive while the
    dashboard is open.  showDashboard no longer marks ui-header inert
    (tab-bar and split-root still are).  The dashboard's role downgrades
    from dialog+aria-modal to region — the appbar being reachable above
    it would otherwise contradict aria-modal's "ignore everything else"
    semantics.

  - Gear icon converts from a direct openSettingsPanel() click into a
    dropdown menu with two items: "MCP connections" (existing modal) and
    "Logout".  Reuses the .ws-tab-dropdown shell for visual consistency
    with the workstream tab chevron menu and the proxy node-picker.
    Logout uses .destructive styling to reduce misclick risk.

Bug fixes caught by the merged code-review pipeline:

  - Global Escape handler skips when _settingsMenu is open, otherwise it
    fires hideDashboard() before the menu's own handler — wiping the
    composer text + staged attachments out from under the user.
  - Menu-item click refocuses the trigger before close, so
    openSettingsPanel captures the gear (not <body>) as the eventual
    return-focus target.
  - ArrowUp keyboard cycling uses idx <= 0 ? len - 1 : idx - 1 instead
    of (idx - 1 + len) % len so the no-focus case wraps to the last
    item rather than the second-to-last.  Same fix backported to
    showTabDropdown which had the identical modulo bug.
  - Position clamps reordered: right-edge override now runs before the
    left-edge floor so a menu wider than the viewport still clamps to
    mx >= 4 instead of going negative.
  - openSettingsMenu caches _settingsMenuTrigger so closeSettingsMenu
    can reset ARIA without re-querying the gear by id.
  - aria-controls lifecycle wired both ways (set on open, removed on
    close).
2026-05-11 21:18:48 -07:00
Patrick Buckley 1df1e739ef fix(session): prevent LLM bypass of per-role plan/task model overrides
The LLM was passing ``task_agent(model="default")`` (and the same for
plan_agent) and routing to whichever backend the auto-created
``default`` alias was attached to at boot — flatspark in the verified
case (ws_id 7dde674) — silently bypassing the operator-configured
``model.task_alias`` / ``model.plan_alias`` (gh200).

Root fix:

- ``load_model_registry`` only synthesises the back-compat ``default``
  alias when neither DB nor ``[models.*]`` populate the registry.  The
  shim was only ever meant for single-CLI-model setups; with a multi-
  model DB it became a phantom routing target aliasing ``LLM_BASE_URL``.
- ``_render_agent_tool_descriptions`` filters ``default`` out of the
  LLM-visible alias list.  The English reading of "default" trips the
  model into picking it explicitly even when the description tells it
  to omit ``model=`` for the per-role default.

Defense-in-depth at the validator chokepoint
(``_validate_agent_model_override``): explicit rejection of
``alias == "default"`` (post-strip) with corrective guidance;
``default`` filtered out of the unknown-alias retry list so an LLM
probing with a bogus alias can't enumerate it back; the no-alternatives
wording is distinguished from the no-registry-configured wording.  The
render path also always rewrites tool descriptions instead of returning
early on filter-empty, so a reload that drops the registry to only
``default`` clears stale alias names left over from a prior render.
2026-05-11 21:18:47 -07:00
Patrick Buckley fbbb21012a feat(audit): emit memory tool save/update/delete events
Previously only the admin-console DELETE route emitted memory.delete
audit rows, so a long-running session whose memory was deleted via
the admin UI had no log trail showing what happened — masking
out-of-band deletes as apparent tool bugs.

The save branch now stamps memory.save (new row) or memory.update
(upsert); the delete branch does a lookup-then-delete-by-id pair so
the audit can record the resolved memory_id and type. All emissions
are best-effort: failures log at debug and swallow so an audit hiccup
never breaks the tool call itself. Reads (get/search/list) remain
un-audited.
2026-05-11 21:18:47 -07:00
Patrick Buckley 617de2488f fix(console): make proxy_api auth dispatch single-sourced
Copilot review on #511 flagged that the dispatch chain and the tests
both claimed to be in lockstep with one another, but only the comment
text said so — the parametrize list and the if/elif chain were two
independent hand-maintained copies, and the comments still referenced
the (long-reverted) ``_PROXY_AUTH_LOCAL_HANDLERS`` symbol.

Make the lockstep guarantee real by collapsing both copies onto one
``_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str]`` mapping
``(method, path)`` to handler-name strings.  ``proxy_api`` resolves
the name through ``globals()`` at call time so ``patch(...)`` in
tests still observes the override — a dict of function refs would
have captured the originals at module load (which is why the first
attempt at this dispatch broke the tests and got reverted).  Test
cases now derive directly from ``_PROXY_AUTH_LOCAL_HANDLERS.items()``,
so adding or removing an entry in the dispatch table flows through
to the parametrize list automatically and the two can't drift.
2026-05-11 21:18:47 -07:00
Patrick Buckley 1d5189bb8d fix(console): allow re-auth from inside the proxy-prefixed UI
When the user is on a proxied node page (``/node/{id}/...``) and the
JWT expires, the in-page login modal POSTs to ``/v1/api/auth/login``
which the proxy shim rewrites to ``/node/{id}/v1/api/auth/login``.
Two latent bugs both had to be fixed for the user to be able to
re-authenticate from inside the proxied UI:

1. ``is_public_path`` didn't recognise the ``/node/{id}/`` prefix
   over a public path, so the console's ``AuthMiddleware`` 401'd the
   login POST before any handler ran.  Extended via the existing
   ``_extract_proxied_path`` helper so a proxied public path stays
   public.

2. Even if the path had been public, ``proxy_api`` would have
   forwarded the request to the upstream node.  The upstream mints
   ``JWT_AUD_SERVER`` tokens; the console's ``AuthMiddleware``
   (expecting ``JWT_AUD_CONSOLE``) would reject those on the next
   proxied call, and ``_proxy_post`` drops ``Set-Cookie`` when
   forwarding anyway.  ``proxy_api`` now dispatches every entry in
   ``_PROXY_AUTH_LOCAL_PATHS`` (login, logout, setup, refresh,
   status, whoami, oidc/authorize, oidc/callback) to the console's
   own auth handlers, and short-circuits non-canonical methods on
   those paths with 405 instead of letting them slip through with
   the service-token fallback.

Tests parametrize across all eight local-dispatch entries so a future
refactor that drops a branch (or routes it through ``_proxy_post``)
fails loudly, plus a no-auth-header reproduction for the original
lockout and a 405 regression guard for the method-mismatch surface.
2026-05-11 21:18:47 -07:00
Patrick Buckley 89a282f86b fix(renderer): mermaid streaming parser errors + progressive hljs (#510)
* fix(renderer): mermaid streaming parser errors + progressive hljs

Live streaming was rendering mermaid diagrams with `Parse error,
got 'PS'` messages — bare `(`, `[`, `{` inside unquoted edge / node
labels re-entered Mermaid's shape parser. Two unrelated streaming-
specific issues in the renderer pile-up here; this commit addresses
both plus a follow-on UX improvement for code highlighting.

## Mermaid label autoquoter

`_normalizeMermaidSource` wraps two label forms that Mermaid rejects
when they contain bare shape-delimiter chars:

  1. Edge labels:  `|content|`  →  `|"content"|`
  2. Rectangle node labels:  `ID[content]`  →  `ID["content"]`

Shapes whose syntax already nests delimiters — cylinders `[(...)`,
subroutines `[[...]]`, trapezoids `[/.../]` `[\...\]`, circles
`((...))`, hexagons `{{...}}`, diamonds `{...}` — are intentionally
left alone (their inner delimiters are part of the shape syntax;
quoting would corrupt them). Labels already wrapped in `"..."` are
also left alone. The rewrite is idempotent and runs before the
mermaid SVG cache lookup so identical malformed input hits the
cache on re-render rather than re-quoting per tick.

## Markdown fence-pair regex

The old fence regex `/(```+)([^\s`]*)\n([\s\S]*?)\1/g` would, mid-
stream, pair an unclosed ```mermaid open with the OPENING backticks
of a later ```python fence as the "close", handing mermaid a
truncated source. New regex:

    /(```+)([^\s`]*)\n((?:(?!\1)[\s\S])*?)\1[ \t]*(?=\n|$)/g

Two constraints close the gap:

  - `(?!\1)` inside the content quantifier blocks the lazy matcher
    from extending across another N-backtick run. Smaller inner
    counts (e.g. 3-backtick inner inside a 4-backtick outer) still
    pass since `\1` is the open's actual count.
  - `[ \t]*(?=\n|$)` after `\1` forces the close to a line
    boundary, so ```python (open with a language tag) can't
    masquerade as a previous fence's close.

Together: an unclosed fence stays as plain markdown until its true
close arrives, so neither mermaid nor hljs ever sees a mid-stream
truncated source.

## Progressive hljs

Extracted `postRenderHljs` from `postRenderMarkdown` with a source-
keyed `_hljsCache` (FIFO, cap 64, keyed on `language:source`) and
wired it into `_streamingRenderApply`. Closed code fences are now
syntax-highlighted as they stream in, matching the progressive
mermaid pattern from #426. Per-tick cost stays cheap because the
cache returns the pre-tokenized HTML synchronously on hit; only
unique (language, source) pairs pay `hljs.highlightElement`.

## Internal cleanup from the review pipeline

  - `_cacheFifoEntry(cache, key, value, max)` replaces the duplicated
    `_cacheHljsEntry` and `_cacheMermaidEntry`. Single tested
    implementation across four caches (hljs, mermaid svg, mermaid
    error, mermaid normalize memo). The "don't evict on overwrite"
    invariant is pinned per-cache in tests.
  - `_mermaidNormalizeCache` memoizes raw textContent → normalized
    output so the per-rAF-tick autoquoter split + regex doesn't
    repeat for unchanged diagrams. Eviction shares
    `_MERMAID_CACHE_MAX` with the SVG cache it feeds.

## Tests

The fake DOM in tests/test_renderer_js.py grew a few capabilities
to drive these paths:
  - `classList` is now array-like (length + indexed access) so the
    hljs language-extraction loop works.
  - `textContent` setter mirrors the real-DOM side effect of
    entity-escaping into innerHTML, so `escapeHtml()` round-trips
    (otherwise every `renderMarkdown` returns empty `<p>` tags).
  - `querySelectorAll` handles both `pre code.language-mermaid`
    and `pre code[class*='language-']`.

Added: 6 fence-pairing regression cases, 9 hljs-progressive cases
(cache hit / distinct sources / language separation / NO_HIGHLIGHT
langs / terminal class / eviction / overwrite / postRenderMarkdown
wraps hljs / _streamingRenderApply invokes hljs), 11 autoquoter
cases including both diagram sources from the live screenshot
encoded verbatim as parametrized regressions, and 3 normalize-memo
cases (populates on first call, consulted before normalize via
sentinel pre-seed, distinct sources cache separately).

Total: 104 renderer tests pass (was 67).

* fix(renderer): apply Copilot review feedback on #510

Two doc / harness adjustments from the PR review — no behavior
change in production code.

- The `_mermaidNormalizeCache` comment claimed eviction "stays in
  lockstep with the SVG cache". That was misleading: the two
  caches key on different things (raw textContent vs normalized
  source) and evict independently. Updated the comment to describe
  what they actually share (the cap, for memory footprint) and
  what they don't (positional coupling), and to note that the memo
  deliberately survives `_initMermaid` since normalize output is
  theme-independent.

- The fake DOM in tests/test_renderer_js.py had `innerHTML` setter
  clear `children` but leave `_textContent` intact, so subsequent
  `textContent` reads could return stale data after an innerHTML
  mutation (real DOM invalidates textContent on innerHTML write).
  No current test triggered this, but it would mask future bugs
  that depend on innerHTML/textContent consistency. Setter now
  clears `_textContent`; the children-derived fallback in the
  getter returns `''` after the wholesale replace.

All 104 renderer tests still pass; ruff + mypy clean.
2026-05-11 21:18:47 -07:00
renovate[bot] 59c943b83a chore(deps): lock file maintenance 2026-05-11 21:18:47 -07:00
Patrick Buckley 14b7516b3f fix(notify): unbreak PG test backend on the notify dispatcher suite
CI's postgres-backend run failed 11 of the new notify tests from #505.
Three independent issues:

1. Migration 053's ``services_notify`` trigger lives only in the
   alembic chain, but the test fixture in conftest.py calls
   ``init_storage(..., run_migrations=False)`` for speed.  That path
   skips migrations and relies on ``metadata.create_all`` for the
   table tree.  Previous alembic-only DDL (migrations 041 / 048
   ``CREATE INDEX CONCURRENTLY`` on workstreams) is performance-only,
   so tests never depended on it.  053's trigger is the first
   behaviorally-required alembic-only DDL in the project — without it
   ``register_service`` doesn't fire NOTIFY and the trigger-filter
   tests time out.

   Fix: declare the trigger function + trigger in ``_schema.py`` and
   attach them via ``sa.event.listen(services, "after_create", ...)``
   DDL events, gated on ``dialect == "postgresql"``.  The same SQL
   constants are imported by migration 053 so there's a single source
   of truth.  Test fixture stays unchanged — ``create_all`` now
   installs the trigger on fresh PG test DBs.  Migration covers the
   upgrade-on-existing-DB path; the two are mutually exclusive given
   ``create_tables = not run_migrations`` in ``init_storage``.

2. NotifyDispatcher tests fired ``storage.notify(...)`` immediately
   after ``d.start()`` and hit a race: the listener thread is
   concurrently calling ``psycopg.connect(listen_url)`` + ``LISTEN
   <channel>`` over the network, so the notify can land before any
   session is listening on the channel and PG drops it (pg_notify
   only routes to sessions LISTEN'ing at COMMIT time).

   Fix: dispatcher gains a ``_listener_ready: threading.Event`` set
   inside ``_listener_loop`` after each successful ``storage.listen``
   open and cleared on disconnect, plus a public
   ``wait_until_ready(timeout)`` method.  Tests use a new
   ``_start_ready(d)`` helper that calls ``start()`` + asserts ready.
   Production callers don't need this (real reactive traffic arrives
   well after startup), but it's the right primitive for any future
   "start dispatcher, immediately send" call site too.

3. ``TestSqliteNotify`` is misnamed — its tests run against whichever
   backend the ``storage`` fixture provides (PG by default in CI).
   Two of its assertions were SQLite-specific:
   ``assert got.pid == 0`` only holds for the synthetic in-process
   path (PG carries real backend PIDs), and
   ``test_synthetic_sweep_emits_after_interval`` is fundamentally
   SQLite-only (no sweep on the PG path).

   Fix: drop the pid assertion (channel + payload are the
   backend-agnostic invariants), add an ``_is_sqlite`` fixture mirror
   of ``_is_postgres``, and gate the sweep test on it.  The sweep
   test also moves from monkey-patching ``stream._sweep_interval`` to
   passing the ``sweep_interval`` kwarg that ``SQLiteBackend.listen``
   now accepts (from the earlier Copilot review fix).

Validated locally against a fresh ``turnstone_test`` PG DB: 263
storage + console + notify tests pass on PG, 257 on SQLite, mypy +
ruff clean.
2026-05-11 21:18:47 -07:00
Patrick Buckley 5d0ec99449 feat(console): event-driven wait_for_workstream + idle cleanup via ChildEventBus
Retire two polling patterns in coord that have clean event sources.
PR 2 of 3 in the coord-completion stack; sits on top of PR #505
(reactive node discovery via PG LISTEN/NOTIFY).

`wait_for_workstream` (coord's block-wait tool) polled storage every
0.5 s in a worker thread regardless of whether anything had changed —
a 600 s wait incurred ~2400 round-trips. Now subscribes to a new
in-process `ChildEventBus` (`turnstone/core/child_event_bus.py`) and
blocks on `threading.Event.wait(min(remaining, WAIT_HEARTBEAT_INTERVAL))`:

- `CoordinatorAdapter` owns the bus; `_dispatch_child_event` calls
  `bus.notify(child_ws_id)` after each `_enqueue_on_ui` for the
  state-class branch (cluster_state, ws_closed, ws_rename,
  intent_verdict, approval_resolved, approve_request).
- Wait loop clears the Event BEFORE the storage snapshot to close
  the subscribe/check race; a notify between clear and the next
  `wait()` leaves the Event set so the loop re-reads without
  losing the wake-up.
- 2 s heartbeat cap preserves the existing `wait_progress` SSE
  cadence for the sidebar UI while cutting SSE traffic ~4x vs the
  pre-bus 500 ms cadence in the quiescent case.
- Worst-case completion latency is 2 s (vs pre-bus 0.5 s) because
  `set_state` buffers non-ERROR writes through `StateWriter`
  (async-flushed) while `emit_state` fans out immediately — a
  bus-driven wake can beat the flusher and read pre-transition
  state, then re-block until heartbeat. Deliberate trade-off; the
  SSE-traffic reduction outweighs the regression on the most
  common terminal transition.
- Defense-in-depth: ownership-filter `cleaned` to own-subtree
  before `register_waiter` so a foreign ws_id passed by an
  untrusted coord LLM (prompt injection) can't observe wake-up
  timing as a side channel. Predicate (`_row_in_own_subtree`)
  requires both `parent_ws_id == coord_ws_id` AND `user_id ==
  coord_user_id` parity — same gate strength as the existing
  `_is_own_subtree` mutating-op guard, so a corrupted /
  cross-tenant `parent_ws_id` alone can't satisfy it. Shared with
  `_snapshot_all` so the snapshot's `denied` shape stays in
  lockstep with the bus filter (Copilot review on #506).

Coord idle-cleanup thread polled the storage scan every
`check_every` seconds (~30 s on default 2 h timeout) even when no
coord was anywhere near idle. Now subscribes to
`SessionManager._state_subscribers` with a `tick_now` event and
blocks on `tick_now.wait(check_every)` — any state change wakes
the sweeper without waiting a full interval, AND the timeout still
fires the periodic sweep for the DB-orphan-only case. A
`min_sweep_interval=5 s` floor bounds DB-call traffic at ~0.2/s
under sustained activity so the loop can't tight-spin `close_idle`
at the rate of its own DB latency (6x improvement over the
pre-refactor fixed 30 s cadence under any activity, and prompt
state-change-driven wakes when below the floor).

`CoordinatorClient` constructor takes `child_event_bus` as a
required kwarg — there's no external SDK shape to preserve and
keeping it optional would silently mask a wiring bug in any future
caller. Tests construct their own `ChildEventBus()` per fixture.

Tests: 16 unit tests for `ChildEventBus` (register / unregister
symmetry, multi-waiter fan-out, multi-child waiter, subscribe/check
race, concurrent register / notify smoke); 7 new adapter tests
(bus notify fires for all 6 state-class events, drops for unknown
child / wrong ws_id); 7 new coord-client wait tests (subscribe-
after-terminal, notify wakes, unrelated notify doesn't wake,
heartbeat fires without notify, unregister on exit, multi-waiter
independence, cross-tenant denial via the user_id-parity filter);
8 idle-cleanup tests (initial sweep, heartbeat cadence, exception
swallowing, stop_event clean exit, state-change wake, subscriber
cleanup, mid-sweep wake, `min_sweep_interval` floor). All pass;
ruff + mypy clean. Full non-live suite: 6227 passed (+2 vs prior
baseline).
2026-05-11 21:18:47 -07:00
Patrick Buckley 6dfd1b5c18 feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher (#505)
* feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher

Add a console-side `NotifyDispatcher` that holds a dedicated PostgreSQL
`LISTEN` connection and fans wake-ups out to per-channel handlers on a
separate dispatch thread. Cluster collector subscribes to a new
`services` channel and runs node discovery reactively — new-node /
graceful-deregister visibility drops from up-to-60 s to ~500 ms on
Postgres, with the 60 s discovery loop retained as the backstop for
crash-shaped node loss (NOTIFY only fires on real writes).

Storage layer gains a uniform `notify` / `listen` API:
- PostgreSQL: real `pg_notify` / `LISTEN` on a dedicated session-mode
  connection that bypasses pgbouncer (mandatory: pgbouncer is required
  in transaction-pool mode per docs, which is incompatible with LISTEN).
- SQLite: in-process fan-out + synthetic-sweep fallback so consumer
  code is identical across backends.

`TURNSTONE_DB_LISTEN_URL` (or `[database] listen_url` in config.toml)
points the dispatcher's connection direct-to-Postgres. Defaults to the
main DB URL when unset.

Migration 053 installs the `services_notify` trigger; it filters
heartbeat-only UPDATEs in-trigger so the 30 s × N-nodes heartbeat tick
stays quiet, while INSERT, DELETE, and url/metadata-changing UPDATE
still fire.

Dispatcher detail:
- Two threads: listener (drains stream → bounded queue) and dispatch
  (invokes handlers under exception suppression). Same-channel notifies
  coalesce per dispatch batch so an N-node deploy burst is one
  `_discover_nodes` per channel.
- Reconnect uses exponential backoff (1 s → 30 s cap). After any
  successful reopen — whether the prior failure was a stream-poll error
  or a connect / initial-LISTEN error — one synthetic Notify with
  payload="reconcile" is enqueued per channel so handlers re-read on
  the same code path they use for real events.

Future consumers (ConfigStore live reload, scheduler immediate
dispatch, audit live-tail) plug in by adding their channel to the
dispatcher's construction list.

Tests: 22 dispatcher tests (incl. reconnect + coalescing under stub
storage), 7 SQLite notify-stream tests, 4 PG-gated trigger-filter
tests, 4 collector wire-in tests. All pass; ruff + mypy clean.

* fix(notify): address Copilot review on #505

- _sqlite.py: SQLiteBackend.listen() now de-dupes channel names via
  dict.fromkeys before constructing the stream — duplicates would
  otherwise register the queue twice and double-deliver each notify.
- _sqlite.py: SQLiteBackend.listen() gains a keyword-only sweep_interval
  parameter (defaults to _SQLITE_NOTIFY_SWEEP_INTERVAL) — matches what
  the comment at the constant already promised, and lets future
  consumers without their own polling timer pick a tighter cadence
  without reaching into private stream attributes.
- _sqlite.py: documented the `except queue.Empty: pass` end-of-drain
  termination so it's not mistaken for swallowing an unexpected error.
- _postgresql.py: docstring referenced :func:`_pg_listen_url` which
  was renamed to _resolve_pg_listen_url during PR development.
- notify_dispatcher.py: module docstring referenced a non-existent
  _bootstrap_console_subsystem; wire-in is at console/server.py::main.

Refuted (no change, false positives from github-code-quality bot):
- 4× "Statement has no effect" on Protocol-method `...` ellipsis bodies
  (idiomatic Python Protocol declaration, not dead code).
- 2× "Mixed import style" in tests — `import ... as nd_mod` is
  intentional to allow attribute assignment for monkey-patching the
  module's `_RECONNECT_BACKOFF_INITIAL` constant inside try/finally.
2026-05-11 21:18:47 -07:00
renovate[bot] 658c65aee8 chore(deps): lock file maintenance 2026-05-11 21:18:47 -07:00
Patrick Buckley 641ce8e7f6 docs(changelog): catch up 1.5.0 through 1.5.12 release notes
Converts [Unreleased] to [1.5.0] and adds individual sections for
1.5.1 – 1.5.12. Covers: MCP OAuth 2.1 + PKCE (Phases 1–8), OIDC
hardening, metacog NudgeQueue + wake trigger, SSE refresh-resume,
reasoning persistence (Phases 1–4), structured watch-result cards,
skills unlock, inline child approvals, Stage 3 Children primitive
lift, coordinator composer parity, node capability auto-detection,
progressive mermaid rendering, and the full schema migration list
for each release.

Updates the track list to stable/1.4, stable/1.5, and main.
2026-05-10 22:34:14 -07:00
Patrick Buckley d76c57e687 chore: bump version to 1.5.12 2026-05-10 18:28:46 -07:00
Patrick Buckley 58bf811a0e fix(server): always emit history event on /rewind to unblock edit-and-resend
Editing the first message in a workstream sends /rewind N where N is the
total user turns, leaving session.messages empty. The handler guarded the
history event with `if history:`, so only clear_ui was emitted. The
frontend dispatches the queued edit-and-resend from the history event
handler (app.js _pendingEditSend), so an empty history orphaned the
pending text and left the composer stuck in busy.

replayHistory already handles the empty case via showEmptyState(), so
emitting the event unconditionally is safe and unblocks the dispatch.
2026-05-10 18:27:23 -07:00
Patrick Buckley df942e375e feat(session): enriched backend error messages with provider + URL
A bare ``httpx.ReadTimeout`` previously surfaced as ``ReadTimeout: timed
out`` — no provider, no base URL, no model — leaving the user with no
signal to tell whether a model server hung, the URL was wrong, or the
model isn't loaded on the backend.

``ChatSession._format_backend_error`` now rewrites known boundary
exceptions (httpx ``ReadTimeout`` / ``ConnectError`` / etc. and OpenAI /
Anthropic SDK ``APITimeoutError`` / ``APIConnectionError`` /
``NotFoundError`` / ``AuthenticationError`` / ``RateLimitError``) into
operator-actionable text that names the provider, base URL (query
string stripped before ``sanitize_error_text`` redacts credentials),
and model.  Matching is by class name so the helper carries no SDK
imports.  Unrecognised exceptions fall through to the legacy
``f"{type(exc).__name__}: {exc}"`` shape, preserving existing grep
targets.
2026-05-10 18:27:14 -07:00
Patrick Buckley 84fd5dc859 chore: bump version to 1.5.11 2026-05-09 17:29:46 -07:00
Patrick Buckley ba8b1d9126 fix(session): AND-gate replay_reasoning_to_model with model capability
The Anthropic call sites in session.py passed the operator-side
`replay_reasoning_to_model` flag through without checking the
model's static `supports_reasoning_replay` capability. The OpenAI
Responses path AND-gated both flags in `_build_kwargs` so a model
without a reasoning lane (gpt-4o, etc.) silently skipped replay even
when the operator flag was set. The Anthropic path had no such gate.

For all current Claude entries this was a no-op asymmetry - every
`_ANTHROPIC_CAPABILITIES` row sets `supports_reasoning_replay=True`,
so `True AND op == op`. But:

- The capability flag was dead code on the Anthropic path
- A future Claude entry (or any Anthropic-shaped surface) shipping
  with the cap left at its False default would have replay fire
  anyway, against the cap declaration
- The asymmetry made `supports_reasoning_replay` an unreliable
  signal - readers couldn't tell if it gated anything per-provider

Move the AND-gate into `_resolve_replay_reasoning_to_model` via a
new optional `caps=` kwarg. When caps is provided, the resolver
returns `operator_on AND caps.supports_reasoning_replay`; when
omitted (back-compat for any caller not yet updated), it returns
the operator flag unchanged.

Thread caps through the three call sites: `_utility_completion`
(non-streaming), `_try_stream` (streaming, hoisted resolution out
of the retry loop since caps are attempt-invariant), and the
agent `_api_call` closure in `_run_agent`.

With the AND-gate now living at the session resolver, the redundant
in-provider gate in `OpenAIResponsesProvider._build_kwargs` is
removed. The provider now trusts the resolved bool it receives,
matching the AnthropicProvider shape and giving the cap a single
source of truth across providers. The two provider-level tests
that pinned the in-provider gate
(`test_include_omitted_when_capability_false`,
`test_include_omitted_by_default`) drop out; the session-level
boundary test
`TestSessionToOpenAIResponsesBoundaryIntegration::test_capability_false_omits_include_even_when_flag_true`
already covers the same end-to-end invariant.

Tests added:
- 4 resolver-level tests pinning the AND-gate semantics +
  back-compat when caps is omitted
- 1 wire-boundary integration test mirroring the OpenAI Responses
  `test_capability_false_omits_include_even_when_flag_true` -
  drives session._try_stream through the real AnthropicProvider
  with operator flag True + capability False and asserts the
  thinking block does NOT reach the SDK boundary

Existing `TestUtilityCompletionPassesFlag` test had its caps mock
upgraded from `SimpleNamespace` to a real `ModelCapabilities`
instance to satisfy the new attribute read and stay robust to
future capability fields.
2026-05-09 17:23:56 -07:00
Patrick Buckley 685b1e3d9b fix(console): preserve cs=None fallback in /v1/api/models placeholder
Copilot review feedback on #500.  The original
``list_available_models`` had an implicit cs=None branch where the
placeholder still advertised ``registry.default`` (filtered against
enabled rows) when ``app.state.config_store`` was None but
``coord_registry`` was bound — useful in the rare degraded state
where lifespan wired the registry but the ConfigStore failed to
initialise.  The PR #500 refactor accidentally dropped that branch:
the helper requires a config_store, so the cs=None case fell out as
"blank coordinator default".

Add an explicit ``elif coord_registry is not None`` branch that
mirrors the helper's tier 3 with the placeholder's enabled-rows
filter applied.  New test exercises this path by passing
``config_store=False`` to the test fixture.
2026-05-09 17:23:56 -07:00
Patrick Buckley 91300060fa fix(console): unify coordinator alias resolution across placeholder + factory
Previously /v1/api/models (home composer placeholder) and
console/session_factory.py walked separate two-/three-tier chains for
the coordinator alias.  session_factory was missing the
``model.default_alias`` tier, so admins who set the system default in
the Models tab would see it advertised but new coordinator sessions
would silently keep launching on ``registry.default``.

This commit:

- Extracts the chain into ``turnstone/console/coordinator_alias.py``.
  ``resolve_coordinator_alias`` returns the effective alias under a
  shared three-tier policy: explicit pin → ``model.default_alias`` →
  ``registry.default``.  Tier 2 is validated against
  ``registry.has_alias`` and falls through to tier 3 with a logged
  warning if unknown.  Tier 1 is intentionally passed through
  unvalidated so an explicit operator pin surfaces as 503 at
  ``registry.resolve`` rather than being silently swapped out.
- Wires both call sites through the helper.  The placeholder supplies
  an ``alias_filter`` that restricts every tier to enabled DB rows so
  the home composer never advertises a model the workstream picker
  can't actually offer; the session factory uses no filter (matches
  prior 503-on-typo behaviour for explicit pins).
- Adds direct integration tests for the session factory's chain
  (``tests/test_console_session_factory.py``) and updates the
  placeholder tests' fixture to provide a stub coord_registry, since
  the helper now requires one.
2026-05-09 17:23:56 -07:00
Patrick Buckley 688f047ce1 docs(console-ui): clarify coordinator placeholder fallback comment
Light-review followup on 389400c8.

The "mirrors session_factory.py:109-110" claim was inaccurate —
session_factory's chain is two tiers (coordinator.model_alias →
registry.default) and skips model.default_alias entirely.  The
placeholder handler extends that chain with model.default_alias as
tier 2 so admins who set the default in the Models tab see it
advertised in the home composer.  Comment now lists the three tiers
explicitly and flags the session_factory-vs-placeholder drift case
(where model.default_alias ≠ registry.default) as a separate issue
to track.

Also lifts the ``from types import SimpleNamespace`` import in the
test fixture to module level — minor readability cleanup.
2026-05-09 17:23:56 -07:00
Patrick Buckley 87893aa4ab fix(console-ui): align coordinator placeholder fallback with session_factory
Two Copilot-review followups on /v1/api/models default resolution.

- console/server.py: coordinator_default_alias now mirrors the full
  fallback chain in console/session_factory.py:109-110 — explicit
  coordinator.model_alias → model.default_alias → registry.default.
  The registry tier was missing, so the home composer placeholder went
  blank whenever an operator never set model.default_alias in the admin
  UI even though new coordinator sessions still launch on
  registry.default (loaded from config.toml [model].default by
  load_model_registry).  Two new tests cover the registry-default
  branch and the disabled-alias guard.
- console/static/app.js: _resolveModelLabel returns "" (not the bare
  alias) when the alias isn't found in the dropdown's model list, so
  callers can rely on the documented "fall back to neutral placeholder"
  contract.  Matches the existing doc comment.
2026-05-09 17:23:56 -07:00
Patrick Buckley 0988142303 feat(console-ui): home composer placeholders, toggle component, admin polish
Bundles the click-around polish on the console admin UX.

Home composer + schedule modals
- /v1/api/models now exposes coordinator_default_alias + judge_default_alias,
  resolved through the same chain console/session_factory.py uses.  Both the
  home composer's MODEL / JUDGE MODEL placeholders and the schedule create /
  edit modal model placeholders rewrite to "Default — alias (model)" once
  the API responds.  The `models_changed` SSE refresh keeps placeholders
  current as operators edit per-role assignments.
- Composer.setOptionPlaceholder added so callers can update just the first
  option's text without disturbing the rest of the choice list.

Admin → Models → Roles
- Channel adapter row added (channels.default_model_alias) — the migration
  to the Roles sub-tab missed it.  Key added to
  _MODEL_AFFECTING_SETTING_KEYS so edits fire the SSE refresh, and to the
  settings-tab roleKeys skip-list so it only renders in one place.
- Plan/Task agent rows now display "(inherit)" instead of the misleading
  "(default — <alias>)" — those roles cascade through plan_model →
  agent_model → session model, not a single concrete default.
- coordinator.reasoning_effort accepts "" (inherit), matching
  model.plan_effort / model.task_effort.
- Blank options in each role's MODEL select now match the "alias (model)"
  shape used by the other rows.

Toggle-switch component
- New .toggle-switch component (visually-hidden native checkbox + styled
  track + label).  40×22 hit target meets WCAG 2.5.5 (AAA), inset ring on
  the off state for ≥1.5:1 contrast against the modal surface.
- .toggle-stack groups toggles in a column with .toggle-group-divider for
  conceptual grouping (used in the Add Model modal between "Active" and the
  paired Reasoning toggles).
- .toggle--flush modifier zeroes the default top margin for toggles that
  sit flush against a heading or a dynamically-rendered row.

Sweep — every admin-modal boolean checkbox is now a toggle:
schedule (cs/es-autoapprove, es-enabled), policy (ep/epp-enabled),
tool-mode (ctm/etm-default), skill (csk/esk-auto-approve, csk/esk-enabled),
MCP (mcp-auto-approve, mcp-enabled), Add Model (Active, surface-persisted-
reasoning, replay-reasoning), judge bool settings (cancel_on_approval et
al.), and the user-roles-modal role assignment list.  The two
ogp-cred / eogp-cred inline credential checkboxes stay as compact inline
boxes since they sit beside text inputs in tight horizontal rows.

Add Model modal — the "Enabled" toggle promoted to "Active" and moved to
the very top of the form.  Tooltip explains it gates dropdown visibility
without removing the definition.

MCP authorization — the three radio buttons replaced with a vertical
.segmented-control option list.  Selected row paints --accent-dim plus a
filled .segmented-indicator; focus ring uses --accent so it stays visible
on the currently-selected option.

Role permissions modal — the 19 permission checkboxes are now
.toggle-switch.perm-toggle (monospace lowercase identifiers preserved).
The permissions are split into Scopes / Admin / Workstreams & Tools
sections under caps-styled section headers so the row-flow grid no longer
slices `admin.*` mid-column.

Judge bool toggles use a static "Enabled" caption rather than flipping
text on `.checked`; flipping lagged 50–300 ms behind the slider position
because the caption was sourced from the post-save reload.

CSS cleanup — dead `.admin-checkbox` / `.perm-checkbox` rules removed.
Specificity audit (scripts/css_specificity_audit.py) returns no conflicts
on any new component class.

Tests — 525 pass on the affected slices; new tests/test_console_available_
models.py pins each branch of the resolution chain in /v1/api/models so the
home composer placeholder stays correct as precedence rules evolve.
2026-05-09 17:23:56 -07:00
Patrick Buckley 40ecebf012 refactor(judge): require alias for judge.model, drop session-provider raw-model fallback
`IntentJudge.__init__` previously had a 3-way resolution chain: registered
alias → raw model id pinned onto the session provider → session model.  The
middle branch was a footgun documented in `console/session_factory.py:130-137`
— pinning the literal `judge.model` string onto the coordinator's session
provider silently broke every verdict whenever that provider didn't recognise
the model id (e.g. coordinator on Anthropic, `judge.model = "gpt-5-mini"` →
uniform `llm_fallback`).

Tightens to alias-only, matching `coordinator.model_alias` /
`model.plan_alias` / `model.task_alias`.  An unknown `config.model` now logs
a warning and inherits the session model — same path as empty.  Help text on
`judge.model` updated to clarify the contract.

Adds two regression tests in `TestModelAliasResolution` covering the
session-model inheritance for unknown values and the empty-model self-
consistency case.
2026-05-09 17:23:56 -07:00
Patrick Buckley c91869c7e5 fix(reasoning): synthesize reasoning_text alongside non-reasoning provider_blocks
GoogleProvider attaches raw tool_call dicts as ``provider_blocks`` on
the finish chunk for ``thought_signature`` round-trip
(``_google.py:_iter_stream``).  When the same turn streamed Gemini's
``reasoning_content`` as ``reasoning_delta`` chunks, the prior
synthesizer bailed out the moment ``provider_blocks`` was non-empty
— so the captured reasoning was visible live but lost on page reload.

Replace the early-return-if-non-empty check with a reasoning-bearing
type test (``thinking`` / ``redacted_thinking`` / ``reasoning`` /
``reasoning_text``).  When none of those types appear, append the
synthetic ``reasoning_text`` block to the existing list rather than
replacing it — preserving Google's tool-call fidelity blocks.

Also addresses two doc-accuracy review findings:
- ``LLMProvider.extract_reasoning_text`` docstring no longer claims
  OpenAI Chat / Responses are unwired (Phase 3+4 shipped extractors).
- Add the method to the Protocol methods table in
  ``docs/architecture.md`` (was missing alongside the class diagram).
2026-05-09 17:23:56 -07:00
Patrick Buckley 53b52092f9 fix(reasoning): per-block ANTHROPIC_VALID_BLOCK_TYPES filter + review fixes
The earlier all-or-nothing shape check on ``_provider_content`` discarded
every valid Anthropic block in a message the moment a single foreign
block (OpenAI ``reasoning``, Gemini thought parts, the synthetic
``reasoning_text`` from path-3 capture) appeared.  In the cross-model
resumption edge case that meant ``server_tool_use`` /
``web_search_tool_result`` blocks lost their ``encrypted_content``
silently, breaking web-search round-trip continuity on subsequent turns.

Replaced with a per-block walk: foreign blocks are dropped individually,
valid blocks ride the verbatim path, and an identity-preserving fast
path reuses the source list reference when nothing was filtered or
stripped (pinned by the ``is`` assertions in test_providers.py).

Also addresses validation-pass review findings:
- Document the single-tier vs three-tier ``surface_persisted_reasoning``
  resolution divergence between server.py:_build_history and
  session_routes.make_history_handler.
- Document why OpenAIResponsesProvider._convert_messages defaults
  ``replay_reasoning_to_model=False`` while Anthropic's defaults True.
- Document the ``source`` metadata field on synthetic ``reasoning_text``
  blocks as reserved-for-future-use, not dead code.
- Add edge tests for non-dict / missing-type-key blocks in
  _provider_content (defensive branches in the per-block walk).
2026-05-09 17:23:56 -07:00
Patrick Buckley 0d1a009a4c test(reasoning): skip wire-boundary tests when anthropic extra missing
CI test job installs `[test]` extras, which omits `anthropic`. The two
TestSessionToWireBoundaryIntegration cases drive the real
AnthropicProvider.create_streaming, which calls _ensure_anthropic() and
raises ImportError. Match the repo convention (test_channel_discord,
test_channel_slack, test_tls_*) by gating the helper with
pytest.importorskip("anthropic").
2026-05-09 17:23:56 -07:00
Patrick Buckley e8352bd8e5 fix(reasoning): apply Copilot review feedback + docs sync
PR #498 round-robin review surfaced 5 findings.  4 applied; 1 rejected
with rationale.

Applied

* **Copilot finding 5** (history_decoration.py:341): dispatcher
  inspected only ``provider_content[0]['type']``.  OpenAI Responses
  captures EVERY ``output_item.done`` event into ``provider_blocks``
  (not just reasoning) — in practice the order is
  ``[reasoning, message, ...]`` but the API doesn't guarantee that;
  a hypothetical ``[message, reasoning]`` ordering would silently
  drop the reasoning under an index-only check.  Now walks the list
  for the first block whose type is in ``_BLOCK_TYPE_PROVIDER_FACTORY``,
  then dispatches the WHOLE list to that provider's extractor.  Each
  provider's extractor already filters internally by its own block
  type, so passing the full list is correct.  Regression test added
  (``test_dispatcher_scans_past_unrecognized_first_blocks``).

* **Copilot finding 3** (migration 052 docstring): the previous
  review-fix wave used sed to rename ``persist_reasoning`` →
  ``surface_persisted_reasoning`` everywhere, which mangled a
  historical reference in the migration docstring ("The earlier name
  ``surface_persisted_reasoning`` was renamed...").  Restored to
  point at the actual pre-rename name (``persist_reasoning``).

* **Copilot finding 4** (sdk/typescript/src/events.ts:26):
  ``HistoryEvent`` JSDoc still referenced ``persist_reasoning`` —
  the sed rename only walked ``turnstone/`` and ``tests/``, missing
  the TypeScript SDK.  Updated to ``surface_persisted_reasoning``.
  Also widened the comment to cover all three reasoning-bearing
  block types (Anthropic ``thinking``, OpenAI Responses ``reasoning``,
  synthetic ``reasoning_text``) instead of mentioning only Anthropic.

* **github-code-quality finding** (session.py:1120): ``_resolve_server_type``
  had a bare ``except Exception: pass``.  Replaced with a
  ``log.debug(..., exc_info=True)`` + explanatory comment.  Behaviour
  unchanged (still returns ``""`` on any lookup failure); failures
  are now observable under DEBUG triage.

Rejected (with rationale)

* **github-code-quality finding** (_protocol.py:265):
  ``extract_reasoning_text``'s body is ``...`` per ``LLMProvider``
  Protocol convention.  Every method in the file uses ``...`` (PEP
  544 idiomatic Protocol style).  Changing only this one to
  ``raise NotImplementedError`` would be inconsistent with the rest
  of the file.  CodeQL's "statement has no effect" warning is
  technically correct for ``...`` as a standalone expression but
  ignores the documented Python Protocol convention.  No fix.

Docs sync

* docs/api-reference.md: ``history`` SSE event message-shape table
  gains the optional ``reasoning`` field.
* docs/architecture.md: ``ModelCapabilities`` row in the type table
  gains ``supports_reasoning_replay``; ``StreamChunk`` and
  ``CompletionResult`` rows gain the existing ``provider_blocks``
  field (was missing pre-PR).  New "Per-model reasoning persistence"
  subsection under the Models config section, documenting the two
  flags + capability gate + three reasoning paths + cross-provider
  shape filter.
* docs/settings.md: new "Reasoning persistence (per-model)"
  subsection with the two-flag table and capability-gate note.
* docs/diagrams/03-core-engine-classes.puml: ``LLMProvider`` interface
  adds ``extract_reasoning_text`` + the new ``replay_reasoning_to_model``
  kwarg; ``ModelCapabilities`` class adds ``supports_reasoning_replay``.
  PNG regenerated.

Lint + test gate

* ruff check + ruff format clean.
* mypy clean (191 source files).
* pytest -m 'not live' — 6116 passed (3 deselected), +1 net new test
  (``test_dispatcher_scans_past_unrecognized_first_blocks``).
2026-05-09 17:23:56 -07:00
Patrick Buckley de4cc568c4 fix(reasoning): apply full-stack review findings
Multi-stage /review on the full Phase 1+2+3+4 stack surfaced 9 findings
(0 critical, 3 major, 5 minor, 1 nit, 1 uncertain).  All applied.

Major

* perf-1 (session_routes.py:2402): make_history_handler ran sync
  storage.load_workstream_config inside async def history on the cold-
  workstream path, blocking the event loop on every dashboard /history
  request for non-resident workstreams.  Every other storage call in
  the same handler correctly used asyncio.to_thread.  Wrap the sync
  call in asyncio.to_thread (preserving the existing try/except so a
  DB failure still degrades to the conservative-default branch instead
  of bubbling out).

* q-2 (test_reasoning_audit_log_discipline.py): the security-sensitive
  test (reasoning text never lands at INFO+ severity) only covered the
  4 Phase 1 surfaces.  Phase 2 added the strip predicate in
  AnthropicProvider._convert_messages and Phase 3 added 3 more code
  paths that touch reasoning text — none guarded.  Added 4 parallel
  tests using the existing capture-and-walk infrastructure:
  OpenAIResponsesProvider.extract_reasoning_text,
  OpenAIChatCompletionsProvider.extract_reasoning_text,
  ChatSession._stream_response (drives the synth-block stamp via a
  fake reasoning-emitting stream), AnthropicProvider._convert_messages
  with replay_reasoning_to_model=False (drives the Phase 2 strip
  predicate).

* q-1 (model_registry.py:42): the persist_reasoning flag name implied
  storage-control but actually gates UI rehydration only — operators
  flipping it could reasonably expect "stop persisting reasoning" but
  storage of reasoning bytes happens in provider_data regardless.
  Renamed everywhere to surface_persisted_reasoning: ModelConfig
  field, migration 052 column (renaming in-place since 052 is not yet
  on main), schema, MODEL_DEFINITION_MUTABLE allowlist, _postgresql.py
  + _sqlite.py CRUD impls, _protocol.py create_model_definition
  signature, 3 console_schemas Pydantic models, console/server.py
  admin POST + PUT, model_registry row mapper, history_decoration.py
  helper parameter, server.py _build_history local var,
  session_routes.py make_history_handler local var, sdk/events.py
  HistoryEvent docstring, admin.js form id + override pill label,
  index.html form input id + UI label + tooltip, coordinator.js (none
  needed), and every test that referenced the old field name.  The
  admin tooltip now reads "Storage of reasoning bytes is unaffected
  by this flag — they ride in provider_data regardless" so the
  decoupling stays explicit at the operator surface.

Minor

* bug-1 (history_decoration.py:336): dispatcher discriminated on
  provider_content[0]["type"] only.  Anthropic's redacted_thinking
  blocks (sealed by the safety system) can appear before, after, or
  interleaved with regular thinking blocks per the API docs.  When a
  redacted block lands first, the dispatcher returned "" and the UI
  silently lost the surrounding thinking text.  Registered
  "redacted_thinking" as a second key in _BLOCK_TYPE_PROVIDER_FACTORY
  pointing at the same AnthropicProvider factory — the existing
  extractor's type=="thinking" filter already correctly skips redacted
  blocks while walking the full list.  Regression test added.

* q-3 (_protocol.py:155): replay_reasoning_to_model defaults split
  across 9 sites — operator-side defaults to False (matches DB
  server_default), provider-API defaults to True (back-compat with
  direct callers).  Original "pick False everywhere" fix would have
  silently flipped behaviour for any direct provider caller.  Instead
  documented the intentional bifurcation in the Protocol's
  create_streaming docstring.

* q-4+q-5 (_protocol.py:107 + 3 providers): MAX_REASONING_DISPLAY_BYTES
  was enforced via Python str slicing which counts code points, not
  UTF-8 bytes — 4-byte CJK/emoji glyphs would blow past the byte
  ceiling.  Renamed to MAX_REASONING_DISPLAY_CHARS to match actual
  behaviour.  Hoisted the 4-line truncation pattern into a shared
  _join_reasoning_with_cap helper in _protocol.py; each provider's
  extractor becomes a single line at the tail.

* q-6 (tests/_session_helpers.py): _NullUI + _make_session were
  duplicated verbatim between test_session_replay_reasoning.py and
  test_session_synth_reasoning_block.py.  Hoisted to a shared
  tests/_session_helpers.py module (importable, leading underscore so
  pytest doesn't try to collect it).  test_model_registry.py's
  _make_session has a different signature (registry/model_alias args
  + _FakeUI) and is not a candidate for sharing.

Nit

* q-7 (history_decoration.py:286): _make_provider_factory used a
  dict-as-cell workaround for closure read-only scope.  Replaced with
  the more idiomatic nonlocal pattern.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6115 passed (3 deselected).  Net +5 tests
  (4 audit-log discipline + 1 redacted_thinking dispatcher).

Refinements vs the dedupe output (caught during sanity rendering
the report)

* perf-1 fix preserved the try/except wrapper.  The original "wrap in
  to_thread" one-liner would have let an OperationalError bubble out
  instead of degrading to the fallback branch.

* q-3 fix explicitly documented the bifurcation rather than
  collapsing both sides to False.  "Pick False everywhere" would
  silently flip back-compat behaviour for direct provider callers.

* q-1 fix included the admin.js:5292 fallback site
  (m.persist_reasoning !== false) that the original threaded-change
  list missed.

* q-6 fix verified the third _make_session in test_model_registry.py
  is structurally different (different signature + different UI
  helper) and intentionally NOT a dedupe target.
2026-05-09 17:23:55 -07:00
Patrick Buckley b477c85ddc feat(reasoning): OpenAI Responses + Chat Completions capture/replay (Phase 3+4)
Wire reasoning capture and (where the API supports it) replay for the
two remaining provider paths.  Phase 3 was originally scoped as
"OpenAI Responses + Gemini" but a spike against the OpenAI SDK source
revealed that Gemini routes through the OpenAI-compatible endpoint
(``/v1beta/openai/``), which is structurally identical to vLLM /
llama.cpp / any other Chat-Completions-shaped local model.  Phase 3
and Phase 4 collapse into one feature with two distinct sub-paths:

* **Path 2 (OpenAI Responses)** — full capture+replay.  ``include=
  ["reasoning.encrypted_content"]`` on the request makes the API
  surface ``encrypted_content`` on reasoning items in
  ``provider_blocks``; ``_convert_messages`` round-trips them as
  ``ResponseReasoningItemParam`` input items on subsequent turns.
  Verified against the OpenAI Python SDK 2.33.0 source
  (``response_reasoning_item.py:31-62``,
  ``response_reasoning_item_param.py:33-37``,
  ``response_create_params.py:70-74``).  Even with ``store=False``,
  ``encrypted_content`` round-trips correctly per the SDK's own
  documentation.

* **Path 3 (Chat Completions / vLLM / llama.cpp / Gemini-compat)** —
  persist-only.  Canonical OpenAI Chat Completions has no reasoning
  field on the wire, but several local-model servers tack on
  ``delta.reasoning_content`` as Pydantic extras.  ``ChatSession.
  _maybe_synth_reasoning_block`` stamps a synthetic ``{type:
  "reasoning_text", text, source?}`` block onto ``_provider_content``
  at end-of-stream when no native ``provider_blocks`` were emitted but
  ``reasoning_parts`` accumulated text.  The ``source`` field carries
  ``server_compat.server_type`` (vllm, llama.cpp, sglang, …) for
  diagnostic value — informational only, doesn't gate behaviour.
  Reasoning text NEVER replays back to the model on this path; it
  rides ``_provider_content`` only for ``/history`` UI rehydration
  and gets stripped from the wire by the existing
  ``sanitize_messages`` underscore-prefix strip on every request.

What this change does

* ``ModelCapabilities.supports_reasoning_replay: bool = False`` added
  to the dataclass.  Set True on every OpenAI reasoning model
  (gpt-5* + o-series via the Responses API) and every Anthropic
  Claude entry (default + 6 model-specific).  Path-2 wire-build does
  ``replay_active = bool(replay_reasoning_to_model and caps.supports_
  reasoning_replay)`` so an operator who flips the flag on a
  non-reasoning model (gpt-4o via Responses) silently no-ops rather
  than emit a malformed ``include=`` request.

* ``OpenAIResponsesProvider`` gains:
  - ``_build_kwargs`` accepts ``replay_reasoning_to_model: bool``
    (threaded from ``create_streaming``/``create_completion``);
    adds ``include=["reasoning.encrypted_content"]`` when active.
  - ``_convert_messages`` accepts the same flag, captures
    ``_provider_content`` reasoning items pre-sanitization, and
    emits them as input items immediately before the assistant
    message they belong to.  Position is tracked by ASSISTANT
    ORDINAL (not raw index) — ``sanitize_messages`` drops orphan
    tool results and inserts synthesized error tool messages, but
    NEVER drops or duplicates assistant messages, so the n-th
    assistant in the original list is invariably the n-th in the
    sanitized list.  Index-based lookup would have silently
    misrouted reasoning attachments after any tool-message repair.
  - ``extract_reasoning_text`` walks ``type=="reasoning"`` items and
    returns ``summary[*].text`` + ``content[*].text`` concatenation.
  - ``_reasoning_item_for_input`` projects a stored item into
    ``ResponseReasoningItemParam`` shape (drops server-only
    ``status``).  Returns ``None`` when ``id`` is missing or non-
    string per the SDK ``Required[str]`` schema; caller skips
    appending, preventing malformed input items from reaching the API.

* ``OpenAIChatCompletionsProvider`` gains:
  - ``extract_reasoning_text`` walks synthetic
    ``type=="reasoning_text"`` blocks and returns the concatenated
    text directly (no underlying provider semantics — the synth
    block IS the surface).

* ``ChatSession`` gains:
  - ``_resolve_server_type(alias)`` reads ``server_compat.server_type``
    from the active model's capabilities dict.
  - ``_maybe_synth_reasoning_block(provider_blocks, reasoning_parts)``
    creates the synthetic ``reasoning_text`` block when no native
    blocks were emitted but reasoning was captured.  Wired at the
    end of ``_stream_response`` immediately before the
    ``_provider_content`` stamp.

* ``history_decoration.py`` dispatcher collapses three near-identical
  lazy-init singleton getters (one per recognised block type) into a
  single ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict + helper.  Adding a
  fourth provider becomes a one-line dict entry.

* Constants hoist: ``MAX_REASONING_DISPLAY_BYTES = 64 * 1024`` moved
  from three sibling provider modules into ``_protocol.py`` so a
  tuning change propagates uniformly to every provider's display path.

Cross-provider safety

The synthetic ``reasoning_text`` block type is intentionally NOT in
``ANTHROPIC_VALID_BLOCK_TYPES`` (Phase 2 constant).  Cross-model
resumption (operator switches from a local model to Anthropic mid-
workstream) falls through Phase 2's shape filter cleanly to the
text+tool_calls rebuild path rather than reaching Anthropic with a
malformed block.  Pinned by ``test_synthetic_block_falls_through_
anthropic_shape_filter``.

Same protection applies in reverse: OpenAI Responses
``type=="reasoning"`` items reaching Anthropic mid-workstream fail
the shape filter and rebuild from text+tool_calls.

Tests (49 net new tests)

* ``tests/test_provider_openai_responses_reasoning.py`` (21 tests):
  - Extractor unit tests: empty/none/no-reasoning/single/mixed/
    truncation/malformed/non-list (8).
  - ``_reasoning_item_for_input`` projection (4 tests including the
    new None-on-missing-id guard).
  - ``_build_kwargs`` include= gating: flag+capability/flag-false/
    capability-false/default-omits (4).
  - ``_convert_messages`` reasoning round-trip: emit-before-assistant/
    drop-on-replay-false/foreign-shape-skipped/default-replay-false (5).

* ``tests/test_session_synth_reasoning_block.py`` (23 tests):
  - ``_maybe_synth_reasoning_block`` direct unit tests (6).
  - Cross-provider safety regression — synthetic block falls through
    Anthropic shape filter (2).
  - ``OpenAIChatCompletionsProvider.extract_reasoning_text`` for the
    new synthetic block type (6).
  - ``_resolve_server_type`` direct unit tests (5).
  - ``_stream_response`` integration tests driving fake reasoning-
    emitting streams through the actual session method (3 tests
    — added in response to a code-review finding that pinned the
    wire-up at session.py needs an integration test).

* ``tests/test_session_replay_reasoning.py`` extended with 4
  ``TestSessionToOpenAIResponsesBoundaryIntegration`` tests driving
  ``session._try_stream`` -> real ``OpenAIResponsesProvider`` ->
  captured ``client.responses.create`` SDK boundary call.  Negative-
  tested: temporarily reverting the ``include=`` step in
  ``_build_kwargs`` makes ``test_replay_true_adds_include_to_
  responses_request`` fail; restoring makes it pass.

* ``tests/test_history_decoration.py`` extended with the new
  ``reasoning_text`` dispatcher branch test, and the Phase 1 stub
  test for the OpenAI Responses dispatcher branch was tightened
  (it now asserts real text extraction instead of the empty-string
  stub).

* ``tests/test_provider_anthropic_reasoning.py`` had its Phase 1
  ``OpenAIResponses returns "" for reasoning blocks`` stub test
  retitled and updated to assert the real Phase 3 behaviour.

Code-review pass

Multi-stage ``/review`` pipeline (4 finders + verify + dedupe) ran
on this diff.  6 findings (1 major, 3 minor, 2 nit), 0 critical, 0
security, 0 performance.  All applied:

* MAJOR (bug-1+bug-4+q-1): ``_convert_messages`` enumerate-index
  lookup was unsound under ``sanitize_messages`` length changes.
  Fixed by switching to assistant-ordinal-keyed lookup.
* MINOR (q-2+q-3): ``_MAX_REASONING_DISPLAY_BYTES`` duplicated
  across three provider modules + declared after first use.
  Fixed by hoisting to ``_protocol.py``.
* MINOR (q-4): three near-identical singleton getters in dispatcher.
  Fixed by collapsing to ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict.
* MINOR (q-5): ``_maybe_synth_reasoning_block`` wire-up not pinned
  by integration test.  Fixed by adding three
  ``TestStreamResponseSynthBlockIntegration`` tests.
* NIT (bug-2): ``_reasoning_item_for_input`` fell back to ``id=""``;
  fixed to return ``None`` on missing/non-string id.
* NIT (q-6): four naming variants for the same concept; renamed
  ``_convert_messages`` kwarg to match the operator-flag name.
* REFUTED (bug-3): SDK distinguishes summary vs content as separate
  fields; no double-counting concern.

Briefing departures

The briefing's Phase 3 plan grouped Gemini with OpenAI Responses on
the assumption that Gemini reasoning had its own native shape (like
Anthropic's ``thinking``).  The spike confirmed Gemini-via-OpenAI-
compat is path-3 (Chat Completions shape, no native reasoning
items).  Phase 3+4 merger handles Gemini for free via the synthetic
``reasoning_text`` block — same mechanism used for vLLM and
llama.cpp.  Whether Gemini's specific endpoint actually emits
``reasoning_content`` deltas is server-dependent and not yet
empirically verified; capture is best-effort (server-emission-driven,
no flag gate).

The briefing's Phase 4 plan stamped reasoning as Anthropic-shaped
``thinking`` blocks ``{type: "thinking", thinking: <text>}``.  This
PR uses a distinct ``{type: "reasoning_text", text, source?}`` shape
to avoid a cross-model resumption hazard the briefing missed: an
unsigned synthetic Anthropic-shape block reaching Anthropic's wire
would 400 the API.  The distinct shape falls through Phase 2's shape
filter cleanly without needing signature validation in the filter.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6110 passed (3 deselected).  Phase 3+4
  added 49 net new tests.
2026-05-09 17:23:55 -07:00
Patrick Buckley 00bd80a658 feat(reasoning): wire-build shape filter + replay flag (Phase 2)
Make ``replay_reasoning_to_model=False`` actually suppress prior-turn
thinking blocks on the Anthropic wire (Phase 1 stored the operator
flag but the wire path always re-sent ``_provider_content``
verbatim).  As a side benefit, close a pre-existing latent bug where
foreign-shaped ``_provider_content`` (e.g. an OpenAI Responses
``type="reasoning"`` block reaching Anthropic on a mid-workstream
model switch, post-Phase-3) would have 400'd the API.

Why now: Phase 1 shipped the operator knob and UI rehydration but
the wire payload still always carried thinking blocks for
Anthropic-with-thinking turns.  Operators flipping replay=False saw
no behaviour change on the actual API call -- the flag only affected
``/history`` rendering.  Phase 2 closes that gap.

What this change does

* ``ANTHROPIC_VALID_BLOCK_TYPES`` (frozenset of 8 block types
  Anthropic's input boundary accepts) and
  ``ANTHROPIC_REASONING_BLOCK_TYPES`` (the strip subset) added at
  the top of ``_anthropic.py``.  The strip set is intentionally
  narrow: ``{"thinking", "redacted_thinking"}`` -- ``tool_use`` /
  ``server_tool_use`` / ``web_search_tool_result`` (which carry
  web-search ``encrypted_content``) MUST survive for round-trip
  continuity, and a regression test pins this.
* ``_convert_messages`` signature gains
  ``replay_reasoning_to_model: bool = True`` (back-compat default
  -- production call sites pass the resolved value explicitly).
  The verbatim ``_provider_content`` replay path is now wrapped by
  a shape-validity check using ``ANTHROPIC_VALID_BLOCK_TYPES``;
  foreign-shaped payloads fall through to the existing text+
  tool_calls rebuild path rather than reaching the API.  When
  shape is valid AND replay=False, a list comprehension drops
  thinking blocks from ``wire_blocks`` while preserving
  tool_use / web_search blocks.  When all blocks are stripped
  (message had only thinking, no text or tool_calls), the message
  also falls through to the rebuild path -- which silently skips
  if both content and tool_calls are empty (correct: stripped
  reasoning has nothing to replay).
* Orphan-tool detection still walks the ORIGINAL ``provider_content``
  (not ``wire_blocks``) so the strip cannot accidentally lose the
  source-of-truth tool_use IDs.  The implementation comment pins
  this invariant.
* Protocol surface grows the kwarg on both ``create_streaming`` and
  ``create_completion``.  ``OpenAIChatCompletionsProvider``,
  ``OpenAIResponsesProvider``, and ``GoogleProvider`` (via
  inheritance) accept the kwarg and ignore it -- they have no
  first-class reasoning shape on the wire today.  Phase 3 will use
  it on the OpenAI Responses adapter to gate
  ``include=["reasoning.encrypted_content"]``.
* ``ChatSession._resolve_replay_reasoning_to_model(alias)`` reads
  ``ModelConfig.replay_reasoning_to_model`` from the registry,
  defaulting to ``False`` on lookup failure (the conservative
  miss-fallback: replaying reasoning text against an unknown
  operator preference is worse than missing the strip).  Threaded
  into the three production call sites:
  ``ChatSession._try_stream`` (streaming), ``_utility_completion``
  (title gen / compaction / extraction), and the agent provider
  call site (plan / task agents).

Token calibration deferred to Phase 4

The briefing's optional Phase 2 step (extending ``_msg_text_chars``
to count ``_provider_content`` bytes that survive the strip)
required either invasive flag-threading through every call site
of the static method or a lossy approximation that picked the wrong
direction for the default case.  Per the briefing's ``pick a
phase'' guidance, this is bumped to Phase 4.  The pre-existing
silent under-count on Anthropic-thinking turns persists when
replay=True.  Strip-when-False naturally fixes the under-count by
keeping the bytes off the wire entirely; the residual case is the
opt-in replay path.

Tests (28 new, all driving through real boundary objects)

* ``tests/test_provider_anthropic_replay.py`` (19 tests):
  - Strip vs preserve under both flag values (3 tests including
    redacted_thinking).
  - Default-kwarg back-compat preserves verbatim replay (1 test).
  - Web-search tool_use + server_tool_use + web_search_tool_result
    survive strip with encrypted_content intact (2 tests, edge 14).
  - Orphan-tool synthesis after strip -- pins the
    ``provider_content`` source-of-truth read at lines 397-433
    (1 test).
  - Foreign-shape fallthrough: OpenAI ``type="reasoning"`` block
    rebuilds via text+tool_calls (1 test).
  - Mixed-shape fallthrough: even one foreign block forces
    rebuild (1 test).
  - Empty / None / non-list ``_provider_content`` fallthrough
    (3 tests).
  - Legacy Anthropic-thinking row pre-Phase-2 stays in verbatim
    path -- no regression on existing conversations (2 tests).
  - All-blocks-stripped fallthrough behaviour: rebuild from text
    if available, silently skip if not (2 tests).
  - Constants pinning: strip set is narrow, valid set includes
    web search, strip is subset of valid (3 tests).
* ``tests/test_session_replay_reasoning.py`` (12 tests):
  - Resolver: 6 tests covering miss / default / set / explicit /
    fallback alias / exception.
  - Streaming call site: 3 tests pinning the kwarg propagates
    through ``_try_stream`` to a stub provider.
  - Non-streaming call site: 1 test pinning
    ``_utility_completion`` propagates the flag.
  - End-to-end boundary integration: 2 tests driving
    ``_try_stream`` -> real ``AnthropicProvider`` -> captured
    Anthropic SDK ``client.messages.stream`` boundary, asserting
    on the ACTUAL wire payload shape.  Negative-tested:
    temporarily reverting the kwarg-thread at
    ``_anthropic.py:create_streaming`` makes the wire test fail
    with ``Strip predicate did not fire at wire boundary``;
    restoring makes it pass.

The boundary integration tests were added in response to a code
review finding that the bare-stub call-site tests would not catch
a regression where the provider stops reading the kwarg or
``_convert_messages`` silently drops the strip.  The integration
tests close that gap by inspecting what reaches the (mocked) SDK,
not just what the provider was called with.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6061 passed (3 deselected).  Phase 2
  added 28 net new tests.
2026-05-09 17:23:55 -07:00
Patrick Buckley 47df9d23c5 feat(reasoning): persist reasoning text on history payload (Phase 1)
Surface stored Anthropic thinking blocks on /history responses so
refreshing the page rehydrates the reasoning bubble. Wire payloads
unchanged. Per-model operator knobs added to model_definitions for
both UI rehydration and (Phase 2) wire-build replay.

Why now: reasoning is already round-tripped via _provider_content for
Anthropic-with-thinking turns, but never surfaces on the history wire,
so a tab reload showed only the final answer with no rationale.
Operators also have no per-model lever to opt out of UI display or to
opt in to replay-to-model on subsequent calls.

What this change does

* Migration 052 adds two boolean columns to model_definitions:
  persist_reasoning (default 1) controls UI rehydration; replay_
  reasoning_to_model (default 0) reserved for Phase 2's wire-build
  shape filter. Mirrors the enabled column pattern (NOT NULL +
  integer server_default).
* LLMProvider Protocol gains extract_reasoning_text(provider_blocks)
  with concrete impls on AnthropicProvider (walks type=='thinking'
  blocks, joins with newline, caps at 64 KiB) and no-op stubs on
  OpenAIChatCompletionsProvider + OpenAIResponsesProvider. Google
  inherits the no-op via OpenAIChat. Phase 3 will wire the OpenAI
  Responses extractor once include=['reasoning.encrypted_content']
  is requested.
* turnstone.core.history_decoration gains a structural dispatcher
  extract_reasoning_text_from_provider_content keyed off the first
  block's type field (Anthropic 'thinking' / OpenAI Responses
  'reasoning' / Gemini 'thought' are non-overlapping by API design).
  Both history surfaces use it: _build_history calls the dispatcher
  directly (the SSE-replay path builds entry dicts from scratch),
  and the lifted make_history_handler runs the list-helper variant
  in the existing to_thread block.
* make_history_handler resolves persist_reasoning via three tiers:
  live session -> workstream_config.model_alias (the same key
  SessionManager uses to rehydrate the original model after process
  restart) -> conservative True default. Operator flag-flip takes
  effect uniformly on both warm and cold workstreams.
* Frontend: app.js replayHistory and coordinator.js role==='assistant'
  branch each call the existing reasoning-bubble construction (for
  app.js, the document.createElement pattern from the live SSE
  handler; for coord, the appendMsg('reasoning') helper) when
  msg.reasoning is non-empty. Reasoning bubbles render before the
  content bubble, matching live SSE order.
* Admin UI: two checkboxes ('Persist reasoning', 'Replay reasoning
  to model') in the model edit modal, plus override-pill display in
  the model row when set to non-default values.

What is intentionally out of scope

* Phase 2 -- ANTHROPIC_VALID_BLOCK_TYPES shape filter at
  _anthropic.py:312-316, _convert_messages replay_reasoning_to_model
  parameter, thinking-strip branch, _msg_text_chars token-calibration
  extension. The replay flag is stored but not consumed on the wire.
* Phase 3 -- OpenAI Responses include=['reasoning.encrypted_content'],
  Gemini include_thoughts spike, ModelCapabilities.supports_
  reasoning_replay.
* Phase 4 -- Local-model / chat-template reasoning persistence
  (session.py:3486 reasoning_parts accumulator).

Tests

* AnthropicProvider.extract_reasoning_text -- 13 unit tests covering
  None / empty / mixed / multi-block / cap / malformed / non-list
  inputs plus other-provider no-op verification (real provider
  instances, no mocks).
* extract_reasoning_for_history -- 10 dispatcher tests including
  block-type discriminator routing (thinking vs reasoning vs
  unknown), strip-when-flag-false, empty / non-dict guards, and
  cross-role isolation.
* _build_history -- 6 boundary tests through the real Anthropic
  extractor with stub sessions, including the registry-lookup
  failure default-True branch.
* make_history_handler -- 5 round-trip tests through real storage:
  the storage layer's reconstruct_messages decodes provider_data
  into _provider_content, and the helper extracts through the real
  AnthropicProvider. Includes the live-session flag honoring path,
  the cold-workstream workstream_config lookup path, and the
  no-alias default-True fallback path.
* Audit-log discipline -- 4 structural mock-and-assert tests that
  capture every Logger.info / warning / error call across the
  pipeline (extractor, dispatcher, list-helper, _build_history)
  and assert no captured payload contains a marker reasoning string.
* model_definitions storage -- 6 round-trip tests: default flags,
  explicit create with both flags, individual update of each flag,
  and list-includes-flags assertion.
* model_registry -- 4 tests: dataclass defaults, dataclass with
  explicit flags, DB-row-mapping with both flags, and pre-052
  legacy-row default-fallback.

Edge cases pinned by the test suite

* Pre-052 DB rows missing the new columns degrade to dataclass
  defaults (test_db_reasoning_flags_default_when_absent).
* Live session in memory has its flag honored (test_history_handler_
  with_persist_flag_false_via_live_session).
* Cold workstream resolves the flag via workstream_config +
  app.state.registry (test_history_handler_cold_workstream_resolves_
  via_workstream_config) -- this closes the gap where a process
  restart would have silently un-honored an operator flag-flip.
* Cold workstream without persisted model_alias falls through to
  default True (test_history_handler_cold_workstream_no_alias_
  defaults_true).
* Foreign / unknown / missing block types degrade silently to no
  reasoning field rather than misroute or crash.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6030 passed (3 deselected).
2026-05-09 17:23:55 -07:00
Patrick Buckley 16fc7efce2 style(sse): align comments with always-advance seq invariant
Doc-debt cleanup flagged by /review on 9dc29db7. The cap+seq fix
flipped the seq-advance rule but left two doc sites describing the
old "incremented only on actual append" shape — exactly the buggy
invariant the previous commit removed. Future readers trusting the
stale docs would be one wrong assumption away from re-introducing
the silent-drop bug.

Updates the field-init comment block and the docstring on
register_listener_with_in_progress_snapshot (which sits at the
snap_seq capture site, so its contract is consumer-facing).

Also drops the now-dead `seq: int = 0` initializer in
on_reasoning_token and on_content_token — under the new shape, the
unconditional `seq = self._ws_inflight_seq` inside the lock makes
the initializer unreachable. Was load-bearing under the old
else-branch; harmless now but signals "some path leaves seq at 0"
to a reader.
2026-05-09 17:23:55 -07:00
Patrick Buckley e8eca2ec9b fix(sse): always advance _ws_inflight_seq on emit, even past cap
Copilot caught a real bug in the cap+seq interaction: the previous
shape only advanced ``_ws_inflight_seq`` when the buffer actually
appended, on the theory that "every _seq corresponds to a buffered
fragment" was a useful invariant. It wasn't — once the buffer hit
its cap, seq stalled at the high-water-pre-cap, so a subscriber that
registered AFTER the cap was hit would capture
``snap_seq == stalled_seq``, and every subsequent live token (also
tagged with the stalled seq) would be filter-dropped by the events
handler's ``seq <= snap_seq`` dedup. Silent loss of the entire
post-cap stream for refresh-past-cap tabs.

Fix: advance seq on every emit, regardless of buffer cap. The cap
is a buffer-size limit, not a stop-streaming signal. Past-cap tokens
are absent from the snapshot's text payload (the buffer was
truncated at cap) but the live stream past them is now correctly
delivered — refresh-after-cap renders snapshot-up-to-cap then live
tokens past it, with a visual gap equal to the past-cap chunk and
no silent drop of subsequent tokens.

Test ``test_inflight_seq_increments_only_on_actual_append`` enforced
the buggy invariant and is renamed/flipped to
``test_inflight_seq_advances_on_every_emit_even_at_cap``. Added
``test_subscriber_after_cap_hit_receives_subsequent_tokens`` (and
the reasoning equivalent) as direct regressions for the
silent-token-loss scenario.
2026-05-09 17:23:55 -07:00
Patrick Buckley 57563b0c12 docs(sse): document state_change + in_progress_snapshot events
Updates the docs that describe the per-workstream SSE event stream and
the SessionUI lifecycle to match the refresh-resume changes:

- api-reference.md: documented the `state_change` event (previously
  undocumented despite already being a live event) and the new
  `in_progress_snapshot` event; rewrote the multi-consumer fan-out
  paragraph to mention the kind-specific replay tail (state_change +
  optional in_progress_snapshot) so the "no catch-up needed" claim
  is no longer misleading.
- architecture.md: bumped the SessionUI Protocol stub to 16 methods
  (added `on_turn_start` / `on_turn_committed`) and pointed at the
  in_progress_snapshot section in the API reference.
- sdk.md: added rows for `state_change`, `in_progress_snapshot`, and
  `approval_resolved` (preexisting gap) to the per-workstream event
  table.
- coordinator-api-tour.md: added an `in_progress_snapshot` row to the
  event table and rewrote the reconnection-contract paragraph to
  cover mid-stream content/reasoning restoration.
- diagrams/04-conversation-turn.puml: added `on_turn_start()` before
  the thinking-start emit and `on_turn_committed()` immediately after
  `messages.append(assistant_msg)`, with notes explaining the inflight-
  buffer reset semantics. PNG regenerated.
2026-05-09 17:23:55 -07:00
Patrick Buckley 43e622840b feat(sse): refresh-resume for mid-stream page reloads
Refreshing a coordinator or interactive workstream pane while the LLM
is mid-stream now restores the partial assistant text + reasoning
immediately and flips the composer back to stop-mode, instead of
showing nothing until the response completes.

Per-turn inflight buffers (`_ws_inflight_content`, `_ws_inflight_reasoning`,
`_ws_inflight_seq`) on `SessionUIBase` are kept separate from the
existing multi-turn `_ws_turn_content` buffer that drives the
dashboard's IDLE-piggyback payload. New `on_turn_start` (top of
send-loop, defensive) and `on_turn_committed` (right after
`messages.append(assistant_msg)`, primary) lifecycle hooks reset
inflight at turn boundaries. The seq counter is monotonic across
turns so a long-lived subscriber's `snap_seq` cutoff stays valid for
the lifetime of the connection — resetting per-turn would silently
drop turn N+1's first M tokens (M = whatever was streamed pre-snapshot
in turn N).

`snapshot_and_consume_state_payload` also drains inflight at idle/error
so cancel and exception paths don't leak stale text. New
`register_listener_with_in_progress_snapshot` atomically registers a
listener and snapshots the inflight buffers; `make_events_handler`
emits a `state_change` event (so the JS busy machine flips to
stop-mode) followed by a one-shot `in_progress_snapshot` after the
kind-specific replay, then strips the internal `_seq` field from
yielded live events while filtering against `snap_seq`. A per-listener
shallow `dict` copy in the live drain prevents the multi-tab race
where one listener's `del event["_seq"]` would corrupt another
listener's filter view.

`_synthesize_cancelled_results` now emits synthetic `on_tool_result`
events for each cancelled tool so live coord tabs can drop the
newly-additive `coord-tool-batch--running` indicator cleanly. The
indicator now coexists with `--auto`/`--approved` (applied on
`tool_info` and `approval_resolved` approved; removed when every row
in the batch has a result), making live tool execution visually
parallel to the replay-time orphan rendering.

Frontend handlers in `app.js` (interactive) and `coordinator.js` (coord)
absorb EventSource auto-reconnect re-replays via a length-based
prefix check on the in-progress buffer. New `InProgressSnapshotEvent`
+ `StateChangeEvent` dataclasses in the Python and TypeScript SDKs
with type guards.

`_MAX_TURN_CONTENT_CHARS` lifted 256 KiB → 512 KiB (single constant
for both buffers — headroom for current commercial models).

Regression tests cover race-free composition under concurrent writers,
seq-filter dedup invariants, the cross-turn seq monotonic invariant,
idle/error inflight drain, synthesized `on_tool_result` on cancel
(including UI-hook failure isolation), and the multi-listener
shared-dict invariant.
2026-05-09 17:23:55 -07:00
Patrick Buckley 584437b98e chore: bump version to 1.5.10 2026-05-08 15:18:46 -07:00
Patrick Buckley 3abe1c0058 fix(skills): apply Copilot review feedback on PR #495
Two findings, both confirmed against the source:

1. Migration 051's downgrade rewrote every '[]' row back to '{}',
   which would (a) destroy operator-written empty arrays and
   (b) reintroduce the known-invalid sentinel that every consumer
   rejects. Pre-migration '{}' rows and operator-authored '[]' rows
   are indistinguishable after upgrade — there is no clean inverse
   for the data state. Made downgrade an explicit no-op with the
   rationale documented inline; '[]' is the correct shape under any
   consumer's interpretation, so leaving the data untouched on
   downgrade is strictly safer than reversing it. Updated the
   module docstring to call this out.

2. admin_update_skill's notify_on_complete validator short-circuited
   on empty string: `if nc and nc != "[]":` skipped the JSON-parse
   branch when nc=="" and persisted the empty string straight to
   storage, leaving a non-JSON value behind. Folded the empty case
   into the existing "{}" coercion so any blank/whitespace/legacy
   value normalises to "[]" before the array-validation gate.

Tests: three new regressions in TestSkillAPI — empty-string
normalises, "{}" sentinel coerces, non-array JSON 400s. The third
locks in the array-only validator that the previous "valid JSON"
gate would have accepted.
2026-05-08 15:18:10 -07:00
Patrick Buckley e24d8b9597 fix(skills): notify_on_complete default is "[]" not "{}"
Every consumer of prompt_templates.notify_on_complete treats it as a
JSON-array string (the admin form's array editor, the JSON.isArray
validator in submitEditTemplate, _validate_notify_targets in
server.py, the documented "list of channel/contact identifiers"
shape). But the column's server_default — set in migration 011 and
inherited through 021's lift into prompt_templates — has been "{}"
(an empty JSON object) since day one.

Newly-installed remote skills inherit the schema default, so every
unlock-then-edit flow trips the array validator on the inherited
"{}" and the request never leaves the browser. The user-visible
symptom was "click Save, nothing happens"; the latent symptom was
silent shape divergence between every install and every operator-
authored skill.

Migration 051: rewrites every legacy "{}" row to "[]". Operator-
edited values (anything that's neither "{}" nor NULL) are left
intact. Downgrade restores "{}" only on rows still holding the
post-migration "[]" so any later operator edits stick.

Server-side defaults flipped to "[]" in the same PR so new rows
land correct without depending on the column's server_default:

- _schema.py prompt_templates.notify_on_complete server_default
- StorageBackend protocol create_prompt_template kwarg
- sqlite + postgres create_prompt_template kwargs
- console_schemas.py SkillCreateRequest / SkillUpdateRequest /
  SkillInfo Pydantic defaults
- core/session.py ChatSession._notify_on_complete initial value
- server.py initial-message worker fallback when skill_data omits
  the field

admin_update_skill validator now also rejects non-array JSON (was
"valid JSON" only — would have accepted "{}" or "{\"a\": 1}").
_skill_to_response coerces legacy "{}" rows to "[]" on read so the
admin UI sees a consistent shape even before migration 051 runs.
The frontend's `tmpl.notify_on_complete || "[]"` fallback already
handled empty-string but not "{}" — the read-side coercion makes
it moot.
2026-05-08 15:18:10 -07:00
Patrick Buckley e11e6f6b70 fix(ui): aria-atomic on modal error elements + drop stale inline display
Designer-review follow-up to the .is-visible sweep. With role=alert
+ aria-live=assertive, AT engines re-announce when the element's
text content changes — but without aria-atomic some engines only
read the diff between old and new content. With aria-atomic=true
the entire updated message is read each time, which matters when a
validation error is replaced by a server error on retry (or
vice-versa).

Added aria-atomic=true to all 24 modal error elements (every
role=alert with aria-live=assertive). Same accessibility uplift
across the board — no per-modal exceptions.

Also dropped the stale `style="display: none"` attribute from the
three MCP error elements (mcp-create-error, mcp-import-error,
mcp-install-error). The CSS rule

  .admin-modal [role="alert"] { display: none; }

already hides them by default — the inline attribute was redundant
and would have overridden the .is-visible toggle if the class-based
contract is ever changed.
2026-05-08 15:18:10 -07:00
Patrick Buckley b8d728fd79 fix(ui): convert remaining modal-error toggles to .is-visible class
Sweep of the latent bug PR #494 fixed for the skill modals: the
project's CSS contract for modal errors is

  .admin-modal [role="alert"]            { display: none; }
  .admin-modal [role="alert"].is-visible { display: block; }

…but ~30 sites across governance.js and admin.js were toggling
`style.display = ""` instead of the .is-visible class. The "show"
side broke silently — clearing the inline style fell back to the
CSS `display: none` so the error never rendered, and any
validation failure looked like an unresponsive button.

Mechanical conversion of every show/hide site for these modal
error elements:

  governance.js
    create-role-error, edit-role-error
    create-policy-error, edit-policy-error
    github-import-error
    cpp-error, epp-error  (custom + eval prompt policies)
    create-hr-error, edit-hr-error  (heuristic rules)
    create-ogp-error, edit-ogp-error  (output-guard patterns)

  admin.js
    mcp-create-error, mcp-import-error, mcp-install-error

Plus the global `_showModalError` helper in admin.js — its
`style.display = "block"` happened to work today (inline display
beats the CSS rule), but normalising it to .is-visible keeps every
modal on a single canonical path. The five modals that route their
show side through that helper (create-user, create-token,
create-channel, create-schedule, edit-schedule) had their hide
sides converted in lockstep.

Added a comment on `_showModalError` documenting the contract so
the next contributor doesn't reintroduce the bug.

Out of scope: model-create-error (already canonical), home-coord-error
(not in .admin-modal), edit/create-template-error (fixed in #494).
No CSS or HTML changes; behaviour-equivalent for hide sides; show
sides go from broken-silent-no-render to correct-render-with-AT-
announcement.
2026-05-08 15:18:10 -07:00
Patrick Buckley 2138c19821 fix(skills-ui): clear prior error at submit-start so it doesn't go stale
Once edit-template-error is actually visible (the visibility fix in
this same PR), a stale error now persists across resubmit cycles:
the user sees a red message, fixes the input, clicks Save, the
validator passes, the PUT goes out — and the previous error stays
on-screen the whole time, only clearing when the modal closes on
success.

Fix at the start of submitEditTemplate / submitCreateTemplate:
clear .is-visible AND empty textContent. Cheaper than tracking
every validator branch and every .catch path; a fresh submit is a
clean slate.
2026-05-08 15:18:10 -07:00
Patrick Buckley 627bf06ced fix(skills-ui): show validation errors via .is-visible, not style.display
Smoke-testing the unlock flow surfaced a latent bug: clicking Save
on the edit-skill modal silently no-op'd whenever the
notify-on-complete field had non-JSON content. The error div was
DOM-correct (text content set, role=alert, aria-live=assertive),
but invisible — because the project's modal-error CSS contract is:

  .admin-modal [role="alert"]              { display: none; }
  .admin-modal [role="alert"].is-visible   { display: block; }

…and the JS in submitEditTemplate / submitCreateTemplate was
clearing the inline `display: none` via `el.style.display = ""`.
That falls back to the CSS rule, which still says `display: none`,
so the error never rendered. The user saw no error and the click
felt unresponsive (compounded by the early-return before the
disabled-state reset, which also made Save look broken).

Fixed both skill-modal flows (create + edit) by toggling the
canonical `.is-visible` class instead. Six sites in governance.js:
the two early-return show paths, the two .catch show paths, and
the two modal-open hide-resets.

Scope note: this same bug pattern exists in ~20 other modal error
sites across governance.js and admin.js (create-role, edit-role,
create-policy, edit-policy, github-import, cpp, epp, create-hr,
edit-hr, create-ogp, edit-ogp, mcp-create, mcp-import, mcp-install,
plus admin.js sites that don't go through _showModalError). All
pre-existing, broken silently for who knows how long. Out of scope
for this PR — recommend a follow-up sweep that also normalises
_showModalError's `style.display = "block"` to the same convention.
2026-05-08 15:18:10 -07:00
Patrick Buckley b67da0f48a fix(skills): apply designer review on lock-icon UX
Designer review of the cb5fa1b lock-icon iteration flagged five
items; four are addressed here, one was a deliberate trade-off
documented below.

- Glyph hardening (#2): the lock character is now 🔒︎ — U+1F512 with
  the U+FE0E text variation selector — paired with the existing
  font-variant-emoji: text rule. font-variant-emoji shipped late
  and isn't universal yet (Chrome 131+, Safari 16.4+, Firefox 132+);
  the explicit text VS is belt-and-braces so older Chromium / most
  Linux don't fall back to a coloured emoji that would clash with
  the monochrome instrument-panel aesthetic.
- Accent-line de-conflict (#3): top:14px → 18px so the lock button
  sits below the modal's ::before accent-line decoration's visual
  band rather than competing with it horizontally. h2's
  padding-right reservation (44px) still gives the title clearance.
- Mobile touch target (#4): @media (max-width: 700px) bumps the
  button to 44×44 (WCAG 2.5.5 / Apple HIG / Material minimum) and
  shifts it to top:8px right:8px, with h2 padding-right widened to
  56px to match.
- Keyboard discoverability (#6): on readonly open, focus lands on
  the lock button instead of Cancel. Keyboard users hit the unlock
  affordance immediately instead of having to Tab past every
  disabled spec input to reach it. Cancel is one Shift-Tab away.

Deferred:
- (#1) Reviewer flagged top-right placement as risking confusion
  with the universal × close-button convention. Keeping the
  icon-only design per product direction; the bordered chip styling
  + accent-coloured hover make it visually distinct from the
  thin-stroke unbordered × pattern, and the confirm dialog catches
  any misclick safely.
- (#5) Optional empty-corner indicator after unlock — the
  "Customized from upstream" badge text already carries the signal;
  not adding new chrome.
2026-05-08 15:18:10 -07:00
Patrick Buckley e05b6adc67 fix(skills): unlock UX — lock icon top-right, save reset, confirm z-index
Three issues from manual smoke-testing the unlock flow:

1. Confirm dialog rendered behind the edit-skill modal. Both
   overlays sat at z-index 600, and confirm-overlay is earlier in
   the DOM than edit-template-overlay — so DOM order put the parent
   modal on top of its own confirm. Bumped confirm-overlay to 650
   (still below toasts at 700) since confirm dialogs are launched
   FROM other overlays and need to sit above them.

2. Save button stayed disabled (or non-functional) after unlock.
   submitEditTemplate disables etm-submit on click and re-enables in
   .finally, but a stale disabled=true survives the mutate-in-place
   re-render that runs after unlock. Always reset
   submitBtn.disabled = false in showEditTemplateModal so the
   re-render path can never inherit a stuck disabled state.

3. UX redesign — moved the unlock affordance from a "Customize…"
   button at the bottom of the footer to a 🔒 icon button at the
   top-right of the modal. The lock glyph is the universal "this is
   locked, click to unlock" affordance and reads more clearly than
   a footer button next to Cancel/Save. font-variant-emoji: text
   keeps it monochrome on browsers that support it (instrument-panel
   aesthetic) with graceful fallback to coloured emoji elsewhere.
   admin-modal-skill h2 reserves padding-right so a long title can
   never collide with the absolute-positioned button.

Cleanup: removed the now-unused .modal-secondary and
.modal-buttons-spacer rules; the bottom etm-unlock button + flex
spacer are gone from the modal footer.
2026-05-08 15:18:10 -07:00
Patrick Buckley 40355d8303 fix(skills): match readonly column int idiom in postgres unlock_skill
Copilot caught that prompt_templates.readonly is an Integer column
(_schema.py: sa.Column("readonly", sa.Integer, nullable=False,
server_default="0")) and create_prompt_template stores it as 1/0,
but unlock_skill in the postgres backend was passing a Python bool
(readonly=False). The sqlite impl already uses 0; this aligns the
two backends and matches the 0/1 idiom used for the sibling flag
columns (is_default, auto_approve, enabled).

The other Copilot findings on this PR (loadGovSkills race, NBSP
double-space, list_skill_versions O(history_size), ignored
set_skill_readonly return value + None re-read) were all closed by
the prior review-feedback commit (eea795d): the snapshot+flip is
now an atomic unlock_skill() that uses SELECT MAX(version)+1
internally, the handler guards both the unlock_skill return and the
post-flip get_prompt_template re-read, the JS chains
showEditTemplateModal off loadGovSkills's promise, and the badge
NBSP matches the sibling pattern.
2026-05-08 15:18:10 -07:00
Patrick Buckley 2cbd926b9d fix(skills): apply review feedback on unlock action
Code review caught a race + a missing None guard; designer review
caught a window.confirm regression and a button-hierarchy issue.

Backend:
- Race fix (bug-2): replace set_skill_readonly+create_skill_version
  with a single atomic unlock_skill(template_id, snapshot, changed_by)
  -> int|None on the storage protocol (sqlite + postgres). Snapshot
  insert + readonly flip happen in one transaction; the next version
  number is computed via SELECT MAX(version)+1 inside the txn rather
  than len(list)+1 outside, closing the (skill_id, version)
  collision window where two concurrent admin actions could both pick
  the same version.
- None guard (bug-3): check the post-flip get_prompt_template re-read;
  return 404 instead of letting _skill_to_response(None) raise.
- Audit body: also record snapshot_version, and harden None-vs-empty
  with `or ""` on the existing.get(...) calls.

Frontend:
- D-1: replace window.confirm with the existing showConfirmModal
  (admin.js:2350) — themed dialog, focus-trap, can render the source
  URL with consistent typography. The native dialog could collapse
  the multi-paragraph copy depending on browser.
- D-2: mutate-in-place on success rather than hide → reload → reopen.
  loadGovSkills now returns its fetch promise so unlockSkill can
  chain showEditTemplateModal after the cache refresh — no flicker,
  no focus bounce, and it kills bug-1 (the reopen was reading stale
  _govSkills before loadGovSkills resolved). showEditTemplateModal
  is idempotent when already open: it skips the trigger-element
  capture and the focus-trap reinstall.
- D-3: button hierarchy. Drop flex:1 from .modal-secondary so the
  Save button keeps a stable width whether or not Customize is
  rendered; insert a flex-spacer between Customize and Save so the
  destructive-ish detach groups left next to Cancel and the primary
  action floats right.
- D-4: NBSP normalized to match the existing   escape pattern
  on the sibling badge line (was an actual NBSP byte).
- D-5: success toast now reads "Skill unlocked — fields are now
  editable" so the operator gets a positive affirmation that the
  edit affordance is live.
- D-10: aria-describedby="etm-origin-badge" on disabled spec inputs
  so screen-reader users get the same "this came from upstream"
  context that sighted users see in the cyan badge.

Tests: + test_unlock_skill_versions_after_existing_history seeds an
out-of-order version (3) and asserts unlock picks 4, defending
against the len()-based version computation regressing.
2026-05-08 15:18:10 -07:00
Patrick Buckley 40c0ab5a58 feat(skills): unlock action lets operators customize installed skills
skills.sh / GitHub installs land with readonly=True so admins can only
tune runtime config (model, temperature, etc.); the SKILL.md spec is
locked. In practice, upstream skills aren't always tuned for turnstone,
so locking the spec adds friction without a real safety win — every
edit is audited and version-snapshotted regardless.

This adds an explicit unlock so the boundary stays visible (multi-user
audit trail benefits from a discrete event, vs. silently dropping the
gate). Behaviour:

- POST /v1/api/admin/skills/{id}/unlock — flips readonly=False on a
  readonly row. Snapshots the pre-unlock state into skill_versions so
  the upstream-pristine version is recoverable from the History tab.
  Records skill.unlock audit with {name, source_url, origin}. 400 on
  already-unlocked, 404 on missing.
- origin stays "source" after unlock so the UI keeps a "Customized
  from upstream" provenance badge — the readonly flag is the gate, the
  origin field is the lineage.
- Storage: dedicated set_skill_readonly writer on the protocol +
  sqlite + postgres backends. readonly is intentionally absent from
  SKILL_MUTABLE so the generic update path can't piggyback on a
  provenance flip — the dedicated writer pattern matches what's
  already used for set_mcp_oauth_client_secret_ct.
- Frontend: "Customize…" button in the edit modal (visible only when
  readonly), with a confirm dialog explaining the upstream-detach.
  Once unlocked the existing edit-skill flow handles spec edits with
  no other changes. Origin badge updates to show "Customized from"
  the upstream URL when a source-origin row is unlocked.

Tests cover: unlock flips readonly + persists, pre-unlock snapshot
written to skill_versions, 400 on already-unlocked, 404 on missing,
post-unlock PUT can edit name/content/description (the readonly gate
no longer fires).
2026-05-08 15:18:10 -07:00
Patrick Buckley a5e8b8b17e fix(skills): apply PR #491 review feedback (size cap + dedup + conflict mapping)
Three issues caught by Copilot on the initial PR:

1. SKILL.md size cap was measured in code points, not UTF-8 bytes.
   `len(str)` is a *lower* bound on encoded byte length — multi-byte
   chars (emoji, CJK) inflate up to 4×, so a 100k-emoji SKILL.md
   (400KB encoded) would slip past the 256KB cap. Switch to
   `len(contents.encode("utf-8"))` and surface lone-surrogate failures
   as SkillSourceError instead of dropping them silently. New
   regression test feeds emoji content.

2. _skills_sh_source_url did not normalize the skill_id, so a sloppy
   id from `/api/search` (whitespace, surrounding slashes) would pass
   `_split_skills_sh_id`'s charset check (which strips first) and
   produce a malformed persisted source_url that broke the
   discover-UI dedup contract. Strip the id inside the helper, and
   reconstruct the canonical id from validated parts in
   download_skill's listing so downstream callers never see the raw
   input.

3. The catch-all `except Exception:` around create_prompt_template
   relabeled every storage failure (DB connection, disk full,
   permission errors) as "conflict", masking operational issues.
   Translate IntegrityError → StorageConflictError at the storage
   shim (matching the pattern already used for OIDC user
   provisioning) in both sqlite and postgres backends, then catch
   StorageConflictError specifically in the install handler. Real
   conflicts → "conflict" + warning; other exceptions → new
   "internal error" reason + log.exception.

Tests: +3 (oversized multibyte SKILL.md, source_url normalization,
storage-layer conflict translation). 226 passing.
2026-05-08 15:18:10 -07:00
Patrick Buckley bed691c62a fix(skills): switch skills.sh install to /api/download endpoint
The skills.sh install path was failing with 404s because their public
API surface changed: /api/skills/{id} is gone, replaced by
/api/skill/[owner]/[repo]/[skill] (auth-walled) and
/api/download/[owner]/[repo]/[skill] (unauthenticated, returns the
SKILL.md + bundled resources inline as JSON). The error was not
surfacing in logs because admin_skill_install had a silent
`except Exception:` around create_prompt_template that relabeled every
storage failure as "conflict" with no log entry.

- Replace SkillsShClient.resolve_github_url with download_skill that
  hits /api/download/{owner}/{repo}/{skill} and returns a SkillPackage
  directly. No GitHub round-trip; no rate-limit surface.
- Add _split_skills_sh_id with strict per-segment charset validation
  ([A-Za-z0-9._-]+) so URL-hostile content can't produce a malformed
  request or divergent persisted source_url.
- Use len(contents) instead of len(contents.encode("utf-8",
  errors="ignore")) for the SKILL.md size cap — errors='ignore' was
  silently dropping invalid units, making the cap bypassable.
- Extract _accept_resource(rel_path, byte_size) gate predicate; share
  it between download_skill and the GitHub _find_resource_files helper.
- Have search() derive a deterministic source_url from the skill id
  when /api/search omits one (which it currently always does), so the
  discover-UI "already installed" check matches what download_skill
  persists.
- Add structured logging across admin_skill_install and
  admin_skill_discover: a shared _log_install_failure helper for the
  four except branches (was four near-duplicate log calls with one
  drift), plus per-resource failure tallying — partial-resource
  installs now surface failed_resources in the response and audit
  record instead of silently committing the skill row with missing
  assets.

Tests: 7 new — empty/non-list files, oversized SKILL.md, resource
cap, non-text extension filtering, plus _split_skills_sh_id charset
rejection (whitespace, query chars). Verified end-to-end against
live skills.sh with tavily-search.
2026-05-08 15:18:10 -07:00
Patrick Buckley c930078f3d chore: bump version to 1.5.9 2026-05-07 22:56:06 -07:00
Patrick Buckley f148c4b423 fix: apply repair=False to all display-read load_messages call sites 2026-05-07 22:48:13 -07:00
Patrick Buckley 8ce4c8e737 chore: bump version to 1.5.8 2026-05-07 17:36:12 -07:00
Patrick Buckley 95e67dc768 fix(replay): apply PR #488 review findings
Four Copilot findings on c6041c6 — all confirmed valid, all bounded
to authenticated-user prompt-injection scenarios but worth closing
before merge.

Wrapper-detect bypass (string + list branches of
``_apply_reminders_for_provider``):

The round-2 fix used ``content.startswith("<tool_output>\\n")`` to
detect already-wrapped content and skip ``escape_wrapper_tags``.  A
tool whose RAW output starts with that prefix (e.g. ``echo
'<tool_output>'``) would match and have its escape skipped, letting
literal ``<tool_output>`` / ``<system-reminder>`` tags reach the model
and impersonate a system envelope.  Replace the prefix check with
``extract_advisories_from_tool_envelope(content) is not None`` —
parsing requires the open AND matching close tags AND a structurally
valid envelope, raising the bypass bar significantly.

Mirror fix in the list-content branch so a tool emitting an unmatched
envelope as a text part can't bypass the per-text-part escape.

``_build_history`` legitimate-envelope drop:

The list-content drop path previously removed any text part starting
with ``<tool_output>\\n``.  A tool that legitimately outputs a
well-formed envelope (documentation viewer, code analyzer demoing the
wrapper, an echo tool) would have that part silently disappear on
replay.  Tighten the drop heuristic to require BOTH ``cleaned_text ==
""`` AND at least one extracted advisory — the structural signature of
the injected ``wrap_tool_result("", advisories)`` carrier we produce
in ``session.py`` for list-typed tool output.  A legitimate envelope
has non-empty inner body or no advisory blocks and survives the
projection.

Empty advisory body:

``queue_message`` accepts any non-None text including ``""`` and
whitespace-only strings.  ``_classify_advisory`` would return a
``user_interjection`` advisory with empty / whitespace body, which
``replayAdvisoriesAfterTool`` then renders as a featureless empty user
bubble.  Filter empty / whitespace-only bodies at classification time
so the wire-shape contract is uniform: no empty advisories ever ride
the wire.

Tests:

* ``test_apply_reminders_escapes_tool_output_starting_with_envelope_prefix``
  pins the structural-parser bypass close: a string starting with the
  envelope prefix but lacking a close tag still gets escaped.
* ``test_apply_reminders_escapes_list_text_part_with_unmatched_envelope_prefix``
  mirrors for the list-content branch.
* ``test_build_history_keeps_legitimate_envelope_text_part_with_body``
  pins that legitimate envelope output stays in the projected list.
* ``test_decorate_suppresses_empty_advisory_body`` and
  ``test_decorate_suppresses_whitespace_only_advisory_body`` pin the
  empty-body filter in ``_classify_advisory``.

Tests: 5923 passed, 3 deselected.  Lint + format + mypy clean.
(cherry picked from commit c2cb6a7ea5)
2026-05-07 17:35:23 -07:00
Patrick Buckley dc35cbc7bf fix(replay): seam 1 splice + storage symmetry for queued user messages
Reverses the seam-2-only design from the prior commits on this branch.
Queued user messages arriving DURING a tool batch (Seam 1) splice into
the last tool result's envelope as ``UserInterjection`` advisories via
``wrap_tool_result``.  Messages arriving BETWEEN turns (Seam 2) drain
as a single trailing user row via ``_flush_queued_messages`` with
``user_feedback`` (operator text alongside an approval, e.g. "y, use
full path") folded in as a prefix.  Cancel/exception drains (Seam 3)
keep the existing ``_flush_queued_messages()`` call unchanged.

Why all three seams:

* Strict-template providers (Mistral, Llama via vLLM with stock chat
  templates) reject role-alternation violations.  A literal ``user``
  row mid-tool-batch breaks ``assistant(tool_calls) → tool → ... →
  assistant``; back-to-back ``user → user`` rows on the wire also fail.
* The seam-2-only design produced back-to-back ``user`` whenever
  ``user_feedback`` and queued items both fired — bug-1 from the round-1
  review.  Folding ``user_feedback`` as a prefix to the queue-drain
  collapses the two into one row.
* During-batch arrivals couldn't ride seam 2 — the splice was the only
  way to deliver same-turn without violating role alternation.

Storage symmetry:

Tool DB rows now store the wrapped ``output`` (envelope + advisories)
unconditionally — ``self.messages[i]['content']`` and
``conversations.content`` match exactly.  List-typed output (image /
structured MCP results) uses ``wrap_tool_result(raw_joined_text,
advisories)`` at save time so the persisted string is anchored on
``<tool_output>\n`` for the replay parser.  ``TOOL_RESULT_STORAGE_CAP``
is removed entirely; tools are responsible for bounding their own
output, storage faithfully represents in-memory.  Removing the cap
also simplifies the parser — no truncated-envelope edge case.

Replay extraction:

``decorate_history_messages`` (REST ``/history``) and ``_build_history``
(SSE replay, resume, rewind, retry, post-load, rename re-replay) both
call the public ``extract_advisories_from_tool_envelope`` helper to
pull the envelope back into structured ``advisories`` for JS replay.
Both string content and list-typed content (image+queued-message
combo) covered.  JS renders extracted advisories as normal user
bubbles after the tool block via the shared ``replayAdvisoriesAfterTool``
helper in ``shared_static/utils.js``.

Wrapper-tag escape and provider splice:

``escape_wrapper_tags`` now encodes pre-existing ``&`` first using an
``&amp;`` sentinel so tool output containing literal entity strings
(documentation viewers, code analyzers, web scrapers returning entity-
encoded markup) round-trips correctly.  Both encode and decode helpers
short-circuit on absence of ``<`` / ``&``.

``_apply_reminders_for_provider`` detects already-wrapped content
(string body and list text-part) by ``startswith("<tool_output>\n")``
and skips re-escape so existing envelopes survive intact when a tool
message also carries ``_reminders`` (the queued-message + tool-error
co-occurrence case is now common).

``decorate_history_messages`` runs in ``asyncio.to_thread`` to keep
MB-scale string work off the event loop.

Other cleanup:

* ``_collect_advisories`` delegates the queue drain to a named helper
  ``_drain_queued_messages_to_advisories`` so the swap-and-clear pattern
  lives next to ``_flush_queued_messages``'s identical pattern and the
  side-effect is documented at the call site.
* Preamble strings + body marker for ``UserInterjection`` round-trip
  detection moved to module-level constants in ``tool_advisory.py``;
  imported by ``history_decoration.py`` so a producer-side rephrase
  can't silently desync the parser.
* ``_send_with_mocks`` ctxmgr extracted in ``test_session.py`` — the
  six new send-driven tests share an 8-deep ``patch.object`` block.
* ``replayAdvisoriesAfterTool`` shared helper in
  ``shared_static/utils.js``; ``app.js`` and ``coordinator.js`` both
  invoke it.
* Dead truncation-pill CSS removed (``.tool-output-truncated`` and
  ``.coord-tool-truncated``); the JS that added these elements went
  away with ``TOOL_RESULT_STORAGE_CAP``.
* Tautological tests (``TestBuildHistoryAdvisoryPropagation``)
  replaced with production-realistic round-trip tests built from
  ``wrap_tool_result(...)`` envelopes — REST and SSE-replay surfaces
  pinned to the same wire shape; full DB round-trip pinned end-to-end.

Negative-tested:

* Reverting the prefix-merge in ``_flush_queued_messages`` produces
  back-to-back ``user`` rows, breaking
  ``test_user_feedback_and_queued_coexistence_single_row_with_prefix``.
* Reverting the ``extract_advisories_from_tool_envelope`` call in
  ``_build_history``'s tool branch leaves the envelope verbatim in
  wire content, breaking the round-trip tests.
* Reverting the wrapper-detection in ``_apply_reminders_for_provider``
  entity-encodes the existing envelope's literal tags, breaking both
  the string-content and list-content envelope-preservation tests.
* Reverting the ``wrap_tool_result(raw_text, advisories)`` projection
  at the DB save site produces a string starting with the original
  raw text, breaking
  ``test_tool_db_row_round_trips_list_output_with_advisories``.

Tests: 5918 passed, 3 deselected.  Lint + format + mypy clean on
touched files.

(cherry picked from commit eca4bb79e4)
2026-05-07 17:35:23 -07:00
Patrick Buckley 79f4d0030d fix(replay): apply review findings q-2 through q-7
Round-1 ``/review`` apply-pass.  Drops stale ``UserInterjection``
references from comments and docstrings that no longer describe the
post-PR drain shape, asserts the two-stream invariant in the new
queued-message persistence test, and pins the ``content.trim()`` +
``renderAssistantToolBatch`` invariants on coord-side so a future
refactor can't silently regress the Qwen3 phantom-card fix or the
chronological-order render fix.

Deferred:

* **bug-1** (back-to-back ``user`` row when ``user_feedback`` from the
  approval-prompt UI callback coexists with a queued-message drain).
  Reachable on strict OpenAI-compatible local templates (Anthropic and
  Anthropic-via-merge-consecutive collapse fine; vLLM-hosted Mistral /
  Llama enforcing role alternation can reject).  The pre-PR splice
  guarded against this case by riding queued items inside the tool
  result envelope; that guard is what motivated the original
  UserInterjection design, so the fix lane needs a deliberate decision
  rather than a quick patch.  Sleeping on it.

* **q-1** (delete dead ``UserInterjection`` class + tests).  Held for
  the bug-1 decision — if the chosen fix is to resume the splice for
  the ``user_feedback``+queue coexistence case, the advisory shape
  stays load-bearing.  Class now carries a docstring note marking it
  retained-pending-decision so a passing reader doesn't grep for
  producers and assume it's actually dead.

Apply-pass content:

* ``q-2``: drop "queued user interjections" from the persistent-
  advisory parenthetical in ``send``'s tool-result loop comment;
  rewrite to point at ``_flush_queued_messages`` for the queue path.
* ``q-3``: ``__init__`` channel-routing comment loses "and
  ``UserInterjection``" — only ``GuardAdvisory`` remains.
* ``q-4``: ``_queue_tool_advisory`` docstring + the tool-error nudge
  comment lose the user-interjection mentions; the docstring also now
  describes the side-channel + ``_apply_reminders_for_provider``
  splice path (the actual mechanism).
* ``q-5``: ``AttachmentsNotQueueableError`` docstring rewritten to
  describe the post-PR ``_flush_queued_messages`` flow — the
  single-combined-turn ``\n\n``-join shape can't carry image / file
  blocks, and per-item separate user turns would expand the strict-
  template role-ordering surface that the post-batch drain already
  balances.
* ``q-6``: the new ``test_queued_message_persists_as_user_row_after_tool_batch``
  in ``test_session.py`` now asserts ``stream_idx == 2`` so a future
  regression where the post-batch flush runs but the send-loop short-
  circuits before the next iteration surfaces in CI rather than
  manual repro.
* ``q-7``: ``test_coordinator_page.py`` gets two new string-grep pins
  mirroring the existing ``test_app_js.py`` shape — ``content.trim()``
  on coord's assistant-replay branch and ``renderAssistantToolBatch``
  for the hoisted helper that orders content card before tool batch.

## Test plan

- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] Affected test surface (``test_session.py`` +
  ``test_tool_advisory.py`` + ``test_app_js.py`` +
  ``test_coordinator_page.py``) — 240 passed

(cherry picked from commit a032e71ff3)
2026-05-07 17:35:23 -07:00
Patrick Buckley 14e504db1f fix(replay): coord render order + blank assistant cards + queued message persistence
Three independent rehydrate / replay regressions reported on long
multi-turn conversations after the pull-model wake stack landed.

**1. coord history replay rendered tool_calls above the assistant
narration that announced them.**

In ``coordinator.js``'s loadHistory loop, the ``role === "assistant"``
``tool_calls`` branch sat above the role switch — every assistant turn
with both narration AND tool dispatch produced ``[tool batch][content
card]`` in the DOM, even though chronological order is content first.
On a parallel fan-out (e.g. four ``close_workstream`` calls in one
turn) operators saw the assistant text "Let me close them out and
summarize" with NO tool batch between it and the next assistant
message — the four-row batch had been rendered above the announcing
text and was scrolled out of view.

Hoisted the ``tool_calls`` synthesis into a local
``renderAssistantToolBatch(m)``, called from inside the assistant
branch AFTER the content card.  Live SSE order (text → dispatch →
results) now matches replay order.

**2. Whitespace-only assistant content rendered as a blank card on
replay.**

Models with vLLM's ``--reasoning-parser`` (Qwen3 in production)
strip ``<think>…</think>`` and emit only the trailing ``"\n\n"`` as
``content`` before a tool call.  ``content_parts = ["\n\n"]`` saves
``content = "\n\n"`` to the conversations row.  Live the user only
sees ``.msg.reasoning`` (the thinking content) — the empty
``.msg.assistant`` card lives next to it but reads as a thin
divider.  On rehydrate the reasoning bubble is gone (not persisted)
and the empty assistant card is the only thing left, surfacing as
"blank cards where the assistant message was."

Both UIs now check ``content && content.trim()`` before rendering
the body — whitespace-only content skips the card entirely instead
of showing a phantom row.  Live render unchanged.

**3. Queued user messages disappeared on reconnect.**

PR #474 routed queued user messages into the tool-result envelope
via ``UserInterjection`` advisories — same-turn delivery, but no
persisted user row.  On page reload / cross-tab replay the
optimistic ``.msg-queued`` bubble vanished: there was no DB row to
rehydrate it.

Dropped the ``UserInterjection`` splice in ``_collect_advisories``;
the queue drains through ``_flush_queued_messages`` AFTER the tool
batch completes instead.  Sequence becomes
``assistant(tool_calls) → tool … tool → user(drained)``, which is
valid for Mistral and Anthropic strict role validators (the only
forbidden shape was user injected mid-batch BEFORE the tool result,
which this still avoids).  Persists a real user row → bubble survives
reconnect, and stays in the session's wire-side context window on
the next turn.

## Test plan

- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] ``pytest -m "not live"`` — 5798 passed, 3 deselected
- [x] Updated ``test_collect_advisories_does_not_drain_queued_messages``
  (was pinning the old UserInterjection shape)
- [x] Added ``test_queued_message_persists_as_user_row_after_tool_batch``
  (drives ``send`` end-to-end with a queued message arriving during
  the tool batch; asserts the user row lands in self.messages AND
  hits ``save_message``)
- [x] Updated ``test_replay_history_renders_content_before_tool_block``
  to tolerate the new ``msg.content && msg.content.trim()`` guard
- [ ] Live browser pass on coord (close_workstream parallel fan-out
  rehydrates with the 4-row batch BETWEEN the announcing assistant
  text and the summary) and interactive (Qwen3 ``"\n\n"`` rows no
  longer paint blank cards on reload; queued bubble survives a tab
  refresh)

(cherry picked from commit c11692b327)
2026-05-07 17:35:23 -07:00
Patrick Buckley 0abe0cb77d fix(mcp): apply PR #489 review feedback + de-flake pool reuse 401 retry
PR #489 review feedback (Copilot + github-code-quality):
- closeSettingsPanel now closes nested revoke modal first on close-button
  path (Escape was already handled by the parent keydown trap deferring
  to the inner trap; missing-modal-on-close-button was an orphan-modal
  hazard).
- _refreshConsentBadge now updates the settings button's aria-label +
  title dynamically with the pending-consent count for screen readers
  (badge stays aria-hidden — the count is in the label).
- _MAX_INSUFFICIENT_SCOPE_REPORTED promoted to public
  MAX_INSUFFICIENT_SCOPE_REPORTED in mcp_http_parsers; drops cross-module
  private import in mcp_oauth's /start handler.
- Stale test comment in test_session_mcp_dispatch_error.py corrected:
  _exec_read_resource does not log with exc_info=True (bearer-leak
  invariant).
- Rejected the protocol-method ellipsis warning: rest of _protocol.py
  uses ... consistently per Protocol convention.

Lint:
- ruff format applied to test_mcp_pool_auth_integration.py and
  test_mcp_pool_auth_resource_integration.py (combined `with` grammar —
  pure formatting).

Flake fix — test_integration_pool_reuse_401_refresh_and_retry_succeeds
on Python 3.11 / resource-constrained CI:

Same cross-task scope hazard f6a3b66 fixed at the close side, surfacing
at the connect side. asyncio.wait_for at mcp_client.py:1206 wraps
streamablehttp_client.__aenter__ in a fresh asyncio.Task. That fresh
task enters anyio cancel scopes, completes, and dies. The eventual
stack.aclose() during eviction or auth_401 retry runs from a different
task and tries to exit scopes whose entering task is dead — anyio
raises RuntimeError, the wedged anyio state blocks the retry's stack
teardown + reconnect, and the call exceeds the 15s budget on slow
workers.

Fix: replace asyncio.wait_for with `async with asyncio.timeout(...)` so
the streamablehttp_client.__aenter__ runs in the dispatch task itself,
no fresh-task scope ownership. Aligns with invariant 18 (asyncio.timeout
not asyncio.wait_for for any SDK / AS / pool-loop await crossing anyio
scopes).

Static path (_connect_one) at lines 905 and 1000 deliberately retains
asyncio.wait_for — auth_type ∈ {none, static} is byte-identical
(invariant 1) and the narrow connect-once / no-eviction-then-reuse
pattern doesn't trigger the cross-task hazard. Anchor comments pin
both directions: a future migration there would break invariant 1; a
future revert at 1206 would re-introduce the flake.

The cited test is the symptom (non-deterministically times out under
load), not a structural gate (no deterministic asyncio.timeout
assertion exists). The comment block at line 1206 records this so a
maintainer who reverts and finds green on a fast machine doesn't
conclude the fix is unneeded.

Verified on Python 3.11.14 (/tmp/venv311) and 3.13.7 (.venv): ruff
format clean, ruff check clean, mypy clean. 368 unit tests + 30 pool
integration tests pass on both interpreters; the previously-flaky test
passed 20× in isolation on 3.11.

Multi-stage /review (4 finders × verify × dedupe): bug/security/perf
returned zero findings; quality returned 3 confirmed minor/nit items
all of which are applied here (q-1 anchor comments at 905+1000, q-2
symptom-vs-gate clarification at 1206, q-3 module-docstring sentence
in mcp_http_parsers).

(cherry picked from commit 4a3e3607be)
2026-05-07 17:35:23 -07:00
Patrick Buckley 610513398b feat(mcp): per-user MCP server consent UX (Phase 8)
Wires the structured-error envelopes produced by Phase 7b's pool
dispatcher (mcp_consent_required / mcp_insufficient_scope /
mcp_*_forbidden / mcp_token_undecryptable_key_unknown /
mcp_oauth_url_insecure) through to the user-facing dashboard, and
adds a per-user settings panel for managing MCP server consents.

Changes
- ``_dispatch_pool_sync`` and ``_dispatch_pool_resource_sync`` wrap
  structured-error string returns as ``RuntimeError(json_str)`` via
  ``_is_structured_error()`` so the session-layer ``except Exception``
  branch fires uniformly across tool / resource / prompt dispatchers
  (the prompt path's ``isinstance(result, str)`` shortcut works only
  because prompts return ``list[dict]`` on success). Without this,
  the consent UX silently does not render for tool / resource calls.
- ``_structured_error`` extended with an optional ``consent_url``
  field; ``_build_consent_url`` produces ``/v1/api/mcp/oauth/start``
  query strings (path-relative; the dashboard appends ``return_url``
  at click time). Wired to all 12 ``mcp_consent_required`` and the
  ``mcp_insufficient_scope`` emit sites.
- New endpoints ``GET /v1/api/mcp/oauth/connections`` and
  ``DELETE /v1/api/mcp/oauth/connections/{server_name}`` registered
  on both ``turnstone-server`` and ``turnstone-console``. The DELETE
  handler runs local delete + audit + 204 first, then schedules the
  RFC 7009 upstream revoke as a fire-and-forget ``asyncio.create_task``
  with strong-ref tracking via ``_revoke_upstream_tasks`` (mirrors
  the ``_pg_refresh_drain_tasks`` pattern). Soft cap of 256 concurrent
  in-flight revokes prevents pile-up under coordinated mass-revoke;
  the audit detail records ``upstream_revoke_outcome`` as
  ``scheduled | no_refresh_token | no_http_client | shed_by_cap``.
- ``ASMetadata`` extended with ``revocation_endpoint`` parsed from
  RFC 8414 metadata. ``revoke_token_at_as`` helper posts the form
  body under ``asyncio.timeout`` (not ``asyncio.wait_for``) and
  never raises; ``_attempt_upstream_revoke`` is wrapped in an outer
  ``try/except Exception`` so unhandled exceptions don't surface as
  ``Task exception was never retrieved``.
- ``/v1/api/mcp/oauth/start`` accepts an optional ``scopes=`` query
  param; tokens are validated against RFC 6749 §3.3 grammar via
  ``is_valid_scope_token`` (promoted to ``mcp_http_parsers``),
  capped at ``_MAX_INSUFFICIENT_SCOPE_REPORTED`` (32), and unioned
  with the configured server scopes for the step-up consent flow.
- Storage primitive ``list_mcp_user_token_metadata_by_user`` projects
  the metadata columns at the SQL boundary so ciphertext blobs never
  cross the wire on the settings-list path. New
  ``MCPUserTokenMetadataRow`` TypedDict in ``_protocol.py``;
  ``MCPTokenStore.list_user_token_metadata`` re-types to the existing
  ``MCPUserTokenMetadata`` shape.
- Dashboard renderer (``app.js``): ``tryParseMcpError`` detects the
  envelope shape on ``tool_result`` SSE events with ``is_error=True``
  and ``buildMcpErrorEmbed`` renders an action card mirroring the
  existing ``buildMediaEmbed`` pattern. Three categories: actionable
  (consent_required / insufficient_scope) with a ``Connect`` button
  that opens ``/v1/api/mcp/oauth/start`` in a popup with a scheme
  guard, forbidden (mcp_*_forbidden) with a static notice, operator
  (key-mismatch / url-insecure) with an operator-action notice.
- New gear button in the appbar opens an MCP-connections settings
  modal driven by ``loadMcpConnections`` / ``confirmRevokeMcp``
  (two-step revoke confirmation matching the existing delete-ws
  pattern). Pending-consent badge tracks unresolved consent prompts
  in this tab; cleared after the connections list returns. Console
  proxy collision-checked: the IIFE only prepends a node-id pill to
  ``header.firstChild``, so the right-anchored gear button is safe.

Bearer-leak invariant
- No ``exc_info=True`` on any new path that can carry a chained
  ``httpx.Request`` (revoke handler, dispatch sites, exec sites).
  The two pre-existing ``exc_info=True`` calls in
  ``_exec_read_resource`` / ``_exec_use_prompt`` were replaced with
  structured-field logs as a Phase 8 sibling fix.

Tests
- 440 pytest passes on both Python 3.13 (.venv) and 3.11
  (/tmp/venv311); ruff + mypy clean.
- 5 new test files: ``test_mcp_consent_url_sibling_audit`` (structural
  gate that every ``code="mcp_consent_required"`` / ``mcp_insufficient_scope``
  site carries ``consent_url=``), ``test_mcp_oauth_connections``,
  ``test_mcp_oauth_revoke``, ``test_mcp_token_store_metadata``,
  ``test_session_mcp_dispatch_error``.
- End-to-end regression coverage for the bug-1 sibling pattern:
  ``test_call_tool_sync_raises_on_structured_error_envelope``,
  ``test_read_resource_sync_raises_on_structured_error_envelope``,
  ``test_get_prompt_sync_raises_on_structured_error_envelope``, plus
  ``test_call_tool_sync_does_not_wrap_non_structured_string`` as the
  defensive gate (only ``mcp_*`` envelopes are wrapped).

Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}``: the
  wrap fires only when the dispatcher returns a structured-mcp-error
  string, which only happens on the oauth_user pool path.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) on every new
  AS / SDK / pool-loop await per Python 3.11 anyio cancel-scope
  hazard.
- Scope cap ``_MAX_INSUFFICIENT_SCOPE_REPORTED = 32`` enforced at
  every output / merge site.
- Cross-user isolation on the revoke endpoint: a non-owner DELETE
  returns 404 with the same body shape as a never-existed row;
  ``http_client_mock.post.assert_not_called()`` pins this in 3 tests.

Deferred (not Phase 8 blockers)
- perf-2 (``asyncio.gather`` parallelisation in revoke handler) —
  superseded by perf-1's fire-and-forget pattern.
- q-4 (prompt-path ``isinstance(str)`` vs sibling ``_is_structured_error``
  asymmetry) — already documented in the function docstring.
- q-9 (``_pendingConsentServers`` → ``_serversNeedingConsent``
  rename) — pure naming taste.

(cherry picked from commit 5a3f46a1fa)
2026-05-07 17:35:23 -07:00
Patrick Buckley 2d6519f9a8 fix(storage): sanitize NUL bytes on _source + _reminders columns
Apply sanitize_text() to the new _source and _reminders columns in
both save_message and save_messages_bulk on SQLite + PostgreSQL,
mirroring the existing pattern used for content and provider_data.

Producers (sanitize_payload on the watch dispatch path,
format_nudge constants on the standard nudge path) already strip
NUL bytes today so nothing in production reaches this clamp — but
the storage layer is opaque to those invariants, and PostgreSQL
TEXT columns reject NUL outright.  Without this clamp, a future
producer that forgets sanitize_payload (or hand-builds the column
string) hard-fails the chat-loop persist path on PostgreSQL.

Cost is negligible — sanitize_text early-exits on the common
no-NUL case via 'if value and "\x00" in value'.

Surfaced by Copilot's PR #486 review.

(cherry picked from commit fc8bd6ca33)
2026-05-07 17:35:23 -07:00
Patrick Buckley a99ce49311 revert(memory): drop dormant limit kwarg from load_messages
Closes round-2 review finding q-7 (nit).

The kwarg was added to close round-1 perf-2 cosmetically — the
storage backend's signature already accepted ``limit``, but the
single in-tree caller (``ChatSession.resume``) doesn't pass it and
other tail-load consumers go direct to ``storage.load_messages``.
Adding signature surface to mark a perf finding closed without an
actual consumer is API-surface bloat.

When a tail-load consumer is written (e.g. a heuristic in
``session.resume`` to skip ancient wake rows), the kwarg can come
back — at that point with a real caller driving the contract.

(cherry picked from commit 14af6f464e)
2026-05-07 17:35:22 -07:00
Patrick Buckley 46f3571c93 refactor(watch): rename _WATCH_REMINDER_OPTIONAL_KEYS public + hoist import
Closes round-2 review findings q-6 (nit) and perf-1 (nit).

* **q-6:** ``_WATCH_REMINDER_OPTIONAL_KEYS`` carried a leading
  underscore (Python's module-private convention) but was imported
  from two other modules — clearly a public contract between
  ``build_watch_reminder`` and its consumers
  (``ChatSession._dispatch`` + ``server._build_history``).  Drop the
  underscore so the import sites match the constant's documented
  cross-module role.

* **perf-1:** The dispatch closure imported the constant inside its
  body, paying ``IMPORT_NAME`` + ``IMPORT_FROM`` bytecode on every
  watch fire.  ``server.py`` already imports at module scope; hoist
  the same way in ``session.py``.  Microsecond savings per dispatch,
  but the in-closure form was just an oversight from the apply-pass.

(cherry picked from commit 668da26dce)
2026-05-07 17:35:22 -07:00
Patrick Buckley 53cabe7e20 fix(session): trim tombstone refs + WHAT-narration in apply-pass comments
Closes round-2 review findings q-1 (minor), q-3 (nit), q-4 (nit), q-5
(nit).

* **q-1:** Drop the ``post-migration 050`` clause from the fork-block
  comment — the apply-pass relocated rather than removed the
  tombstone-style temporal reference round-1 q-2 was supposed to fix.
  The bulk-row dict shape and ``_encode_reminders`` are
  self-explanatory; the WHY is pinned by
  ``test_fork_preserves_source_and_reminders``.

* **q-3:** Replace ``DOES persist now`` framing on the wake-row save
  comment with a present-tense invariant.  The ``now`` implies the
  reader knows the prior state, same family as the temporal
  tombstones.

* **q-4:** Trim the 12-line WHAT-narration block above the
  resume-time ``_reminders_delivered = True`` loop to two lines
  stating the WHY only.  The new regression test pins the contract.

* **q-5:** Reframe ``test_fork_preserves_source_and_reminders``
  docstring as a forward-looking invariant; drop the
  ``Dropping them was the original bug`` and ``post-migration 050``
  fix-narration.

Project convention: invariant statements, present tense; don't
reference the current task / fix / migration number.

(cherry picked from commit b120ee2fd7)
2026-05-07 17:35:22 -07:00
Patrick Buckley 1d9fd94e23 fix(session): byte-clamp REMINDER_TEXT_STORAGE_CAP + drop local-only doc citation
Closes round-2 review findings bug-1 (minor) and q-2 (minor).

* **bug-1:** ``_encode_reminders`` clamped each entry's ``text`` field
  with Python ``str`` slicing, which counts codepoints.  Multi-byte
  UTF-8 input (CJK, emoji) could land 4 bytes per character past the
  cap, defeating the row-width / FTS5-index protection by up to 4x.
  Switch to UTF-8 byte clamping with ``errors="ignore"`` on the
  decode boundary so a slice mid-codepoint drops the partial
  character cleanly.

* **q-2:** Both the constant block-comment and the ``_encode_reminders``
  docstring referenced ``docs/design/watch-card-ux-briefing.md`` —
  local-only per project convention (``feedback_no_design_doc_commits``)
  so the canonical repo reads as a dead reference.  The cap value
  stands by itself; the row-width / FTS5 WHY is enough.

(cherry picked from commit 779ec638a5)
2026-05-07 17:35:22 -07:00
Patrick Buckley eb89ddab1e fix(metacog): cleanup batch — share watch-key constant, sanitize metadata, drop tombstones
Closes round-1 review findings q-2 (minor), q-5 (minor), q-6 (nit), q-7
(nit), sec-1 (nit), perf-4 (nit).

* **q-5:** Export ``_WATCH_REMINDER_OPTIONAL_KEYS`` from
  ``turnstone/core/watch.py`` and import in the dispatch closure
  (session.py) and the replay filter (server.py:_build_history).  The
  three-place duplication of the literal tuple
  ``("watch_name", "command", "poll_count", "max_polls", "is_final")``
  is gone; future field adds touch one constant.

* **sec-1:** Run ``sanitize_payload`` over string-typed metadata fields
  (``watch_name`` / ``command``) before they enter the queue.  Today's
  consumers all use ``textContent``, but the asymmetry — sanitised
  ``text`` alongside unsanitised metadata — would survive forever in
  DB rows and resurface if a future consumer used a non-textContent
  sink (aria-label, copy-to-clipboard, markdown render).

* **q-7:** Drop the per-iteration ``isinstance(reminder, dict)`` from
  the dispatch closure's metadata comprehension.  By the time the
  block runs, ``text = reminder.get("text", "") if isinstance(...)``
  + the ``if not sanitized: return`` guard above already established
  ``reminder`` is a non-empty dict.

* **q-2:** Strip tombstone-style references — "post-#482", "post-#484",
  "Step 7 of the watch-card UX plan", "Post-Step-7 dispatch surface",
  and the brittle line-anchor "session.py:2685-2686" — across
  ``session.py``, ``test_session.py``, ``test_watch.py``,
  ``test_watch_dispatch.py``, ``test_watch_integration.py``.  Comment
  intent preserved; historical anchors gone.

* **q-6:** Drop the ``del source`` line in ``cli.py``'s
  ``on_user_reminder``; the parallel ``on_tool_reminder`` ignores
  ``tool_call_id`` without ``del`` and the comment alone is enough.

* **perf-4:** Document the SQLite ``render_as_batch=True`` recreate
  cost in migration 050's docstring — first deployment after upgrade
  copies the conversations table twice (one per ``add_column``).
  PostgreSQL is unaffected.

5734 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 7e35050b68)
2026-05-07 17:35:22 -07:00
Patrick Buckley eb92e61755 fix(ui): wrap interactive reminder spans in .msg-body + exclude system-nudge from anchor lookup
Closes round-1 review findings q-3 + q-4 (minor, merged) and bug-3 + bug-4
(nit, merged).

* **q-3 + q-4:** The new ``.msg.user-reminder .msg-body { white-space:
  pre-wrap }`` rule was a no-op on the interactive UI because that
  frontend's ``_buildDefaultReminderBubble`` appended label + text spans
  directly to the outer ``.msg.user-reminder`` element with no
  ``.msg-body`` wrapper.  Coord rendered the same shape with a wrapper.
  The two implementations diverging on DOM structure also meant a
  shared-helper extraction was harder than necessary.  Reconciled by
  wrapping interactive's spans in ``.msg-body`` to match coord; the CSS
  rule now applies to both UIs and the shared-extraction follow-up to
  ``shared_static/cards.js`` is mechanical (deferred per the review
  report — out of scope for this commit).

* **bug-3 + bug-4:** The reminder anchor lookup ``.msg.user`` also
  matched ``.msg.user.system-nudge`` markers because the marker carries
  both classes.  A non-wake reminder fired between a wake marker and
  the next real user message would anchor below the wake marker rather
  than the previous real user message.  Edge case (``/history`` reload
  corrects), but the fix is mechanical: change the selector to
  ``.msg.user:not(.system-nudge)`` in both files.

(cherry picked from commit 869135d97a)
2026-05-07 17:35:22 -07:00
Patrick Buckley 0e2ea122eb fix(memory): wire limit kwarg through load_messages
Closes round-1 review finding perf-2 (minor).

Storage backends accept ``*, limit: int | None = None`` (see
:meth:`StorageBackend.load_messages` at storage/_protocol.py:146) but
the in-memory wrapper at memory.py:82-85 dropped the kwarg, so
callers that wanted to tail-load (e.g. ``session.resume`` against a
long-running coord with hundreds of wake rows + persisted reminder
JSON) were forced to pull every row through the wrapper anyway.

Wraparound is mechanical: signature widens, default leaves existing
callers unaffected.

(cherry picked from commit 885f6a9185)
2026-05-07 17:35:22 -07:00
Patrick Buckley eb9dd2402a fix(session): delete stale 'reminders stay in-memory' comment
Closes round-1 review finding q-1 (major).

The comment block above ``self._attach_pending_user_reminders(user_msg)``
asserted that reminders "stay in-memory only and don't persist across
reloads" — directly contradicted by the comment block immediately below
(at the save_message call site) that explains the new persistence
semantics, plus the actual code that now writes ``_source`` and
``_reminders`` to the conversations row.  Future readers hitting both
blocks would lose trust in the surrounding comments.

The lower block already documents the persistence contract, so the
upper block is just deleted rather than rewritten.

(cherry picked from commit 81502c962f)
2026-05-07 17:35:22 -07:00
Patrick Buckley 0c58910c4b fix(session): preserve _source/_reminders on fork + cap persisted reminder text
Closes round-1 review findings bug-2 (major), perf-1 (minor), perf-6 (nit).

* **bug-2:** ``ChatSession.resume(..., fork=True)``'s bulk-row builder
  silently dropped the ``_source`` and ``_reminders`` side-channel
  data the source workstream had persisted via ``_append_user_turn``.
  Both backends' ``save_messages_bulk`` already accept these keys
  (the columns exist post-migration 050) — the bulk builder just
  didn't supply them.  The fork's resumed transcript would then look
  like the assistant turn answered out of nowhere: every wake marker
  and every reminder bubble that survived to disk on the source got
  dropped on the fork.  New regression test
  ``test_fork_preserves_source_and_reminders`` pins the contract.

* **perf-6:** Extracts ``_encode_reminders(reminders) -> str | None``
  near ``_apply_reminders_for_provider`` so the user-turn save path,
  the tool-turn save path, and the new fork bulk builder share one
  encoder.  Eliminates the drift risk between three near-identical
  ``json.dumps(..., separators=(",", ":")) if X else None`` patterns.

* **perf-1:** The new helper clamps each entry's ``text`` field at
  ``REMINDER_TEXT_STORAGE_CAP = 8192`` characters before encoding so
  a single rogue producer (a watch streaming unbounded shell output,
  a corruption-class steering payload) can't blow the conversations
  row width or the FTS5 index.  The in-memory side-channel keeps the
  full body — only the persisted JSON is clamped.  Mirrors
  ``TOOL_RESULT_STORAGE_CAP`` on tool result rows.

5734 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 91e7f2daca)
2026-05-07 17:35:22 -07:00
Patrick Buckley 2e393d76b4 fix(session): flag persisted reminders delivered on resume
Persisted ``_reminders`` survive ``load_messages`` but the in-memory
``_reminders_delivered`` flag does not (it's session-scoped — set by
``_mark_reminders_delivered`` after each successful provider stream,
never persisted alongside the JSON column).  Without a re-splice
guard at resume time, ``_apply_reminders_for_provider`` would walk
every loaded message, see ``_reminders`` set + the flag falsy, and
splice every historical ``<system-reminder>`` envelope onto the wire
on the very next user turn — leaking each reminder a second time, the
turn after it had already advised.

Mirror the post-stream hook in ``resume()``: every loaded message
that carries reminders has already been delivered (it survived to
disk), so flag it accordingly so ``_apply_reminders_for_provider``
short-circuits on the pass-through path.

Test pins the contract end-to-end — stage a workstream with a
persisted reminder, resume into a fresh session, append a live user
turn, run the wire transform, and assert the historical reminder
body does NOT land in the rendered output.

(cherry picked from commit f1466ca7e3)
2026-05-07 17:35:22 -07:00
Patrick Buckley dec175f176 feat(ui): structured watch-result card + system-nudge marker on replay
User-visible slice of the watch-card UX workstream — combines the
replay-path widening, both frontend renderers, the CSS, and the
cross-cutting Python tests.

server._build_history widens the reminder filter from {type, text} to
project on a known set of optional fields (watch_name, command,
poll_count, max_polls, is_final) and surfaces _source as
entry["source"] when set.  The known-key filter narrows the blast
radius if a future producer accidentally stuffs sensitive fields
into the dict.

SessionUIBase.on_user_reminder takes a new source: str | None kwarg
that rides on the SSE event when set.  _attach_pending_user_reminders
forwards user_msg["_source"] so non-originating tabs see the wake's
"system_nudge" tag and render the thin marker.  Protocol + cli + eval
implementations widen accordingly.

Frontend (coordinator.js + app.js — touched in lockstep per project
memory's "logic that lands in BOTH UIs must touch both files"):
* Branch on r.type === "watch_triggered" for a structured
  .msg.watch-result card with header / $ command / <pre> body /
  poll N/M [· final] footer.
* New addSystemNudgeMarker (interactive) + appendSystemNudgeMarker
  (coord) renders a thin .msg.user.system-nudge anchor for
  wake-driven reminders, both live (source === "system_nudge" on the
  SSE event) and replay (msg.source === "system_nudge").
* Default .msg.user-reminder rendering preserved for every other
  metacog nudge type.

CSS (shared_static/chat.css):
* New .msg.watch-result rules — full-width treatment, cyan accent,
  monospace body with word-break: break-word for mobile.
* New .msg.user.system-nudge rule — thin yellow marker.
* Bonus newline-collapse fix: .msg.user-reminder .msg-body now sets
  white-space: pre-wrap so multi-line shell output / bulleted lists
  stay readable inside the advisory bubble.

Plan reference: docs/design/watch-card-ux.md §4 Steps 9-12 + bonus
CSS §11 (Commit 4).

(cherry picked from commit 6ae6877acc)
2026-05-07 17:35:22 -07:00
Patrick Buckley 592433b46d feat(metacog): structured watch reminders carry watch metadata onto NudgeQueue
WatchRunner._dispatch_result now takes a structured reminder dict
produced by build_watch_reminder() — text matches format_watch_message
verbatim (so compaction / channel adapters / wire splice keep their
behaviour), and watch_name / command / poll_count / max_polls /
is_final ride alongside as queue-entry metadata.

The dispatch closure registered in ChatSession.set_watch_runner pulls
the optional fields out of the dict and passes them to enqueue via
the new metadata kwarg.  Drain seams already merge metadata into the
rendered reminder dict (Commit 2), so the SSE event for a watch fire
now carries the structured fields without further plumbing.

* turnstone/core/watch.py — new build_watch_reminder() helper, _poll_watch
  switches from format_watch_message + dispatch(str) to build_watch_reminder
  + dispatch(dict).  set_dispatch_fn / get_dispatch_fn / restore_fn
  signatures widen from Callable[[str, str], None] to
  Callable[[dict[str, Any], str], None].
* turnstone/core/session.py — dispatch closure builds the metadata dict
  via {k: reminder[k] for k in ("watch_name", "command", ...) if k in reminder}
  and passes it to nudge_queue.enqueue.
* tests/test_watch.py — new TestBuildWatchReminder class pinning the
  builder shape; existing dispatch_fn_registry / restore_fn tests
  updated to dict shape.
* tests/test_watch_dispatch.py — every dispatch(...) call updated to
  pass a structured reminder dict via _reminder() helper; new
  TestMetadataPropagation class pins the metadata-on-enqueue contract.
* tests/test_watch_integration.py — _dispatch_result calls updated to
  dict shape.

Plan reference: docs/design/watch-card-ux.md §4 Step 7 + Step 8 watch-test
subset (Commit 3).

(cherry picked from commit 13db19905a)
2026-05-07 17:35:22 -07:00
Patrick Buckley da5321eb88 refactor(metacog): widen NudgeQueue._Entry with optional metadata field
Producers (today only watch_triggered) can now attach a metadata dict
to a queued nudge so the rendered reminder dict on the user/tool side
carries fields beyond {type, text}.  Wire shape stays additive: the
SSE event picks up the optional fields when present, and producers
without metadata leave it None.

* _Entry grows from 4 fields to 5 — metadata: dict[str, Any] | None.
* enqueue accepts metadata=... as a kwarg.
* drain returns list[tuple[str, str, dict | None]] (was 2-tuples).
* pending stays narrow at (type, text) for legacy callers; new
  pending_with_metadata projects the third slot for tests that need
  to assert producer-specific fields.
* Three drain consumers in session.py — _collect_advisories,
  _attach_pending_user_reminders, deliver_wake_nudge_from_queue —
  unpack the new 3-tuple shape and merge metadata into each
  reminder dict.
* on_user_reminder / on_tool_reminder protocol signatures widen
  from list[dict[str, str]] to list[dict[str, Any]] across
  ChatSession.UI, SessionUIBase, CLI, eval harness.

Plan reference: docs/design/watch-card-ux.md §4 Step 6 + Step 8 _Entry
subset (Commit 2).

(cherry picked from commit 30b7e4dd24)
2026-05-07 17:35:22 -07:00
Patrick Buckley baa2214f96 feat(storage): persist _source + _reminders side-channels on conversations
Adds two TEXT-NULL columns to the conversations table so multi-tab /
multi-device replay sees the same metacognitive bubble shape the
originating tab saw live.  Until now, reminders lived only on the
in-memory ChatSession.messages dict, and the wake-driven empty user
turn was not persisted at all (skip at session.py:2685-2686) — a
second tab connecting via /history saw the assistant turn with no
preceding wake context, and missed every other tab's reminder
bubbles besides.

Single Alembic revision 050 (head was 049) adds:
  * conversations._source — today only "system_nudge" for wake rows
  * conversations._reminders — JSON-encoded reminder list

Both backends (sqlite + postgresql) thread the columns through
save_message / save_messages_bulk / load_messages.  reconstruct_messages
unpacks the row tuple as 9 elements (was 7), JSON-decoding _reminders
on the user AND tool branches with the same contextlib.suppress guard
the existing provider_data / tool_calls decode uses.  Tool-row
reminders ride the same column so tool_error / repeat replay shape
matches user-channel parity.

session.py:2685-2686 wake-row persist skip is dropped; _append_user_turn
JSON-encodes user_msg["_reminders"] and passes both source + reminders
to save_message.  The tool-message save site at session.py:3014-3020
mirrors with metacog_reminders.

Plan reference: docs/design/watch-card-ux.md §4 Steps 1-5 (Commit 1).

(cherry picked from commit f64c3e7b10)
2026-05-07 17:35:22 -07:00
Patrick Buckley 3b60a69e4f fix(console): atomic coord-subsystem commit + offload startup teardown
Address Copilot review feedback on PR #487:

1. **Atomic commit invariant**: ``_bootstrap_coord_subsystem`` previously
   stamped ``coord_mgr`` ~50 lines before the final ``coord_registry``
   commit, and started threads + subscriptions in between.  A concurrent
   dashboard request running through ``_require_coord_mgr`` during the
   runtime-bootstrap window could observe ``coord_mgr`` set with
   ``coord_registry`` still ``None`` and surface the misleading
   "Restart the console after adding a model definition" 503.

   Refactored to two phases: (a) build everything as locals, (b) start
   side-effects (StateWriter / observer / nudge watcher / child fan-out
   / cleanup thread), then atomic commit at the end with ``coord_mgr``
   stamped LAST.  The build-phase ``try/except`` rolls back any started
   side-effects from local handles before re-raising — no daemon thread
   or subscription leaks across retries, and ``app.state`` is never
   stamped on a partial failure.

2. **Class-attr cleanup symmetry**: ``_teardown_partial_coord_subsystem``
   now also clears ``ConsoleCoordinatorUI._coord_mgr`` /
   ``_collector`` / ``_console_metrics`` to match the lifespan shutdown
   path (server.py ~line 4629).  A failed bootstrap (or test teardown
   reuse) no longer leaks process-global pointers at a half-built
   subsystem.

3. **Lifespan startup offload**: the lifespan startup error path used
   to call ``_teardown_partial_coord_subsystem`` synchronously, which
   in turn calls ``StateWriter.shutdown(timeout=2.0)`` — a thread-join
   + sync DB writes that could block the event loop for up to 2s
   while the console is still coming up.  Wrapped the whole
   load-and-bootstrap in ``asyncio.to_thread`` via the new
   ``_load_and_bootstrap_coord_subsystem`` synchronous helper, so all
   blocking work (including any rollback) runs on a worker thread.
   Mirrors the pattern the regular lifespan shutdown (line ~4620) and
   the runtime CRUD-triggered path already use.

Tests:
- ``test_bootstrap_atomic_commit_no_partial_visibility``: a polling
  thread in tight loop watches ``coord_mgr`` / ``coord_registry``
  during a real bootstrap and asserts no observation has ``coord_mgr``
  set with ``coord_registry`` still ``None``.
- ``test_real_bootstrap_rolls_back_partial_state_on_side_effect_failure``:
  monkeypatches ``install_idle_nudge_watcher`` to raise mid-build,
  asserts ``app.state`` shows the clean fresh-install state and the
  builder-failure error string surfaces ``RuntimeError`` (not the
  stale "no models" boot-time message).

(cherry picked from commit c6b4dc26be)
2026-05-07 17:35:22 -07:00
Patrick Buckley 5d1213d3dc fix(console): bootstrap coord subsystem on first model add
A freshly-installed console with no model rows in the DB at boot
caught the ``ValueError`` from ``load_model_registry()`` in the
lifespan and skipped the entire coord subsystem build, leaving
``coord_mgr`` ``None``.  ``_refresh_coord_registry`` then bailed
out at ``existing is None`` rather than building the subsystem on
first model add — operators had to restart the console after
configuring their first model in the admin panel for the
"Coordinator subsystem not initialized" banner to clear.

Extract the lifespan's coord build into a reusable
``_bootstrap_coord_subsystem`` and add ``_maybe_bootstrap_coord_subsystem``
that runs as an ``asyncio.to_thread`` follow-on after every admin
model-CRUD endpoint (create/update/delete/reload).  The helper:

- fast-paths to a no-op when ``coord_mgr`` is already set;
- guards concurrent first-install attempts with
  ``_COORD_BOOTSTRAP_LOCK`` + double-checked re-test inside the lock;
- pre-computes config-derived integers BEFORE any thread starts so
  ``int(config_store.get(...))`` failures don't strand a started
  ``StateWriter`` daemon;
- stamps ``coord_state_writer`` to ``app.state`` immediately after
  ``.start()`` so the new ``_teardown_partial_coord_subsystem`` can
  shut it down on a partial failure (no thread leaks across retries);
- atomically commits ``coord_registry`` + clears
  ``coord_registry_error`` as the final step so callers can rely on
  the invariant ``coord_registry`` is set iff ``coord_mgr`` is set;
- replaces the stale boot-time "no model definitions" message with
  a builder-failure-specific diagnosis (carrying ``type(exc).__name__``)
  on construction failure so the dashboard's 503 banner reflects the
  actual cause.

Both the lifespan path and the runtime-bootstrap path now route
through the same helper and the same teardown on failure.

Tests: 12 new tests covering the helper-level wiring (idempotent
fast-path, missing-prereq parametrised over ``config_store`` /
``collector`` / ``console_metrics``, no-rows error recording, builder
failure error replacement, partial-state teardown), the endpoint
integration, the deterministic concurrent-call lock test (uses an
instrumented lock wrapper that signals when a second acquirer arrives,
so the test fails fast on slow CI rather than depending on a
wall-clock sleep), and a real-builder end-to-end case constructing a
working ``SessionManager`` against a real ``ConfigStore`` + real
``ClusterCollector``.

(cherry picked from commit 3143965e00)
2026-05-07 17:35:22 -07:00
Patrick Buckley 9ae2b376c7 fix(mcp): apply Phase 7b PR #485 review feedback
Two of five Copilot comments on PR #485 were valid; this commit applies
both. The other three (one duplicate of comment 1, plus the INFO-logging
and `_pending`-naming nits) get rationale on-thread and resolution.

1. emit_oauth_failure_audit action now derived from `code` (#485 bug-1)

The Phase 7b refactor generalized `emit_insufficient_scope_audit` →
`emit_oauth_failure_audit`, routing both `mcp_insufficient_scope` AND
generic-403 (`mcp_*_forbidden`) through the same helper. The audit
`action` field stayed hardcoded as
`"mcp_server.oauth.insufficient_scope_emitted"`, mislabeling generic
forbidden events under the insufficient_scope bucket — downstream
alerting / analytics filtering on `action` would silently fold both
categories together.

The action is now selected from `code`:
  * `mcp_insufficient_scope` →
    `mcp_server.oauth.insufficient_scope_emitted` (preserves existing
    alerting consumers)
  * `mcp_tool_call_forbidden` / `mcp_resource_read_forbidden` /
    `mcp_prompt_get_forbidden` →
    `mcp_server.oauth.forbidden_emitted` (new, distinct label)

Detail row continues to carry both `code` and `kind` so operators get
sub-bucket distinction within either action.

2. Resource-listener docstrings cite RFC §3.2 (#485 doc-1)

Per the codebase convention established in Phase 7b round-1 q-1
(`_rebuild_user_prompt_map` corrected §3.2 → §3.3 because prompts are
§3.3 in the MCP spec), resource-related docstrings should cite §3.2.
The three resource-listener docstrings were citing §3.3, and the
"Mirrors `_notify_listeners` for tools (RFC §3.3)" parenthetical in
both `_notify_resource_listeners` and `_notify_prompt_listeners` read
as "tools are at §3.3" — confusing twice over. All four sites now
carry the correct catalog-kind citation explicitly:
  * resource-listener docstrings → "RFC §3.2 (resources)"
  * prompt-listener docstrings → "RFC §3.3 (prompts)"

Tests / lint:
  * 119 passed on 3.13 + 3.11 (targeted MCP OAuth pool tests)
  * ruff + mypy clean on both files

(cherry picked from commit 12cc052bca)
2026-05-07 17:35:22 -07:00
Patrick Buckley b368bdeecc feat(mcp): per-user resource + prompt pool dispatch (Phase 7b)
Extends the Phase 7 per-(user, server) ClientSession pool to cover
RFC §3.2 (resources/read) and §3.3 (prompts/get) on the same shape
already proven for tools/call. Pool discovery is capability-gated so
servers without resources/ or prompts/ stay free of extra round-trips.

API additions / widenings (MCPClientManager):
- ``read_resource_sync(uri, *, user_id=None, timeout=120)`` —
  per-user-first dispatch; falls through to the byte-identical static
  path when ``user_id`` is None or the URI doesn't resolve to an
  ``oauth_user`` pool entry.
- ``get_prompt_sync(prefixed_name, arguments=None, *, user_id=None,
  timeout=30)`` — same dispatch shape; structured-error responses
  surface via ``RuntimeError`` so the agent-loop's ``except Exception``
  block renders the JSON without polluting the prompt-protocol return
  shape.
- ``get_resources(user_id=None)`` / ``get_prompts(user_id=None)`` —
  per-user merged catalogs (admin/global call still passes None).
- ``add_{resource,prompt}_listener`` /
  ``remove_{resource,prompt}_listener`` —  ``user_id`` keyword scopes
  the listener so a pool-only catalog change for one user does not
  wake another user's session.
- ``resource_count_for_user(user_id=None)`` /
  ``prompt_count_for_user(user_id=None)`` — method-form variants used
  by ChatSession's ``read_resource`` / ``use_prompt`` tool gating; the
  legacy ``resource_count`` / ``prompt_count`` properties remain
  static-only for admin paths.
- ``_dispatch_pool_resource`` / ``_dispatch_pool_prompt`` async coros
  — mirror ``_dispatch_pool`` for the new SDK calls; share the
  carrier-race-and-cancel core via ``_dispatch_pool_with_entry_call``.
- ``_handle_auth_403`` extended with ``kind=Literal["tool",
  "resource", "prompt"]`` so the per-operation ``mcp_*_forbidden``
  code surfaces (kind="tool" remains the default for back-compat).
- Pool notification handler now refreshes resources / prompts on
  ``ResourceListChangedNotification`` / ``PromptListChangedNotification``
  via ``_refresh_pool_server_resources`` / ``_refresh_pool_server_prompts``.

ChatSession (``turnstone/core/session.py``) call-site updates:
- 12 sites threaded the session-bound ``user_id`` through
  ``add_*_listener`` / ``remove_*_listener``, ``get_resources`` /
  ``get_prompts``, gating, ``read_resource_sync`` /
  ``get_prompt_sync``, and ``is_mcp_prompt`` so the per-user merged
  catalog drives both the visible-tool set and dispatch.
- ``/mcp`` slash command now lists this user's pool resources and
  prompts alongside tools (Phase 7 already scoped tools).

Scope decisions:
- Per-user-first URI ordering (decision 0.1): the dispatcher attempts
  the user's pool catalog first, falling back to the static catalog
  only when no pool entry resolves the URI / prefixed name. Pool-only
  users never see the static catalog leak into their resolution.
- Method-form ``*_count_for_user`` (vs property) keeps the legacy
  ``resource_count`` / ``prompt_count`` properties intact for admin
  endpoints whose contract is "static catalog size only".
- Shared ``_dispatch_pool_with_entry_call`` helper accepts an
  ``sdk_call: Callable[[ClientSession], Awaitable[Any]]`` closure,
  keeping the entry-locked carrier-race / classification / retry
  plumbing single-source instead of a 3x copy across tool / resource
  / prompt paths.

R6 (anyio uniformity): every pool-side list / read / get path uses
``async with asyncio.timeout(...)`` — ``asyncio.wait_for`` is
forbidden in those paths because it wraps the inner awaitable in a
fresh task and surfaces ``CancelledError`` from inside
``streamablehttp_client``'s anyio TaskGroup on Python 3.11
(per ``feedback_asyncio_timeout_vs_wait_for.md``).

Tests:
- ``test_mcp_pool_auth_resource_integration.py`` — 9 real-transport
  resource tests (FastMCP upstream + ``BehaviorMiddleware``):
  401-refresh-retry success, persistent 401 -> consent_required,
  403+insufficient_scope, 403 generic -> mcp_resource_read_forbidden,
  breaker-isolation under repeated auth failures, missing-token,
  decrypt-failure, http:// URL guard, unknown-URI ValueError.
- ``test_mcp_pool_auth_prompt_integration.py`` — 9 mirror tests for
  the prompt path; structured-error responses verified via
  ``RuntimeError`` payload shape.
- ``test_mcp_user_catalog.py`` — extended unit coverage for per-user
  resource / prompt rebuild + collision policy + symmetric eviction.
- ``test_sessions.py::TestMCPToolGating`` — pool-only-user canary
  asserts ``read_resource`` / ``use_prompt`` stay visible when the
  static catalog is empty but the user has pool entries.

Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: ``_exec_use_prompt`` was hardcoding ``"MCP prompt error: failed
  to invoke prompt"`` — discarding the structured-error JSON that
  ``_dispatch_pool_prompt_sync`` raises via ``RuntimeError``. Now uses
  ``f"MCP prompt error: {e}"`` mirroring ``_exec_mcp_tool``; pool-prompt
  consent_required / insufficient_scope / forbidden errors now reach
  the LLM as intended.
- bug-2 + bug-3: resource template discovery was uncapped —
  ``_cap_server_resources`` covered ``res_result.resources`` but the
  separate ``tmpl_result.resourceTemplates`` loop appended every
  template a server returned. Added ``_MAX_RESOURCE_TEMPLATES_PER_SERVER``
  (1000) + ``_cap_server_resource_templates`` helper, applied at both
  the initial discovery site (``_connect_one_pool``) and the refresh
  site (``_refresh_pool_server_resources``). Mirrors the existing
  ``_MAX_TOOLS_PER_SERVER`` / ``_MAX_PROMPTS_PER_SERVER`` defensive
  ceilings.
- sec-1 + sec-2: ``emit_insufficient_scope_audit`` generalized to
  ``emit_oauth_failure_audit(kind, code, ...)``, called from both the
  insufficient_scope branch AND the previously-silent generic 403
  branch. Audit detail now records ``{"kind": kind, "code": code,
  "scopes_required": [...]}`` so operators can distinguish tool-call
  vs resource-read vs prompt-get 403s in audit logs and so cross-
  tenant probing on the generic 403 path leaves a trail. The Phase 7
  inherited gap (``mcp_tool_call_forbidden`` had the same silence) is
  closed in the same refactor.
- perf-1: pool resource discovery now uses ``asyncio.gather(
  list_resources, list_resource_templates)`` inside the existing
  ``async with asyncio.timeout(...)`` budget — disjoint catalogs, no
  ordering dependency. Typical-case 2-RTT cold-connect resource block
  collapses to 1-RTT. Same change applied at ``_refresh_pool_server_resources``.
- q-1: ``_rebuild_user_prompt_map`` docstring corrected RFC §3.2 →
  §3.3 (resources are §3.2; prompts are §3.3).
- q-2: ``_refresh_pool_server_prompts`` docstring now carries the
  R6 / mcp-loop note that the resource sibling already had — both
  refresh paths now declare the asyncio.timeout invariant explicitly.
- q-5: added the ``_user_resource_map`` / DB-mismatch guard to
  ``read_resource_sync`` for parity with ``get_prompt_sync``. A stale
  per-user map entry with no matching oauth_user row now raises a
  specific ValueError instead of silently falling through to a
  generic ``Unknown MCP resource``.
- q-6: ``_dispatch_pool_with_entry`` (now a single-caller wrapper
  after the ``_dispatch_pool_with_entry_call`` extraction) gains a
  one-line docstring explaining why the wrapper is preserved
  (tool-decode localization + stack-trace identity for debugging).
- q-7: added 1 resource + 1 prompt end-to-end integration test that
  drive REAL discovery + dispatch in the same connect (no
  ``_seed_pool_*_map`` shortcuts), mirroring the tool path's
  ``test_integration_pool_reuse_401_refresh_and_retry_succeeds``.
  The seeded-map tests stay (faster, focused on dispatch); the new
  e2e tests cover the connect-discover-dispatch composition that
  caught Phase 6's carrier-on-entry bug.

Pre-push round-1 review fixes (3-finder review on the final state —
the lesson from Phase 7 round-3's q-1 regression: round-2 catches
what the round-1 apply pass missed):
- q-1 (MAJOR): the bug-1 sibling that round-1 missed —
  ``_exec_read_resource`` was hardcoding ``"MCP resource error: failed
  to read resource"`` while ``_exec_use_prompt`` (post-bug-1) preserved
  the structured-error JSON via ``f"... error: {e}"``. The round-1
  apply pass patched the prompt side but not the resource side. q-5's
  per-user-map / DB-mismatch ValueError was being swallowed at the
  agent loop boundary, defeating the operator-diagnostic intent. Now
  ``_exec_read_resource`` mirrors ``_exec_mcp_tool`` and ``_exec_use_prompt``.
- q-6 (nit): defensive-cap comment block at module-level cited
  "(RFC §3.2)" while covering both resource and prompt list paths;
  prompts are §3.3. Now reads "(RFC §3.2 for resources, §3.3 for
  prompts)" matching the convention the q-1 apply established.
- q-5 (rejected with better justification): the reviewer flagged
  ``_dispatch_pool_with_entry`` as a single-caller wrapper that should
  be inlined. After examination — the autouse fixture
  ``tests/test_mcp_pool_auth_introspection.py::_install_capture_intercept``
  monkeypatches this method to stash ``entry.auth_capture`` for the
  fake call_tool stubs in dispatcher-asserting tests. Inlining would
  redirect the patch to ``_dispatch_pool_with_entry_call`` (different
  kwargs shape) and require re-validating every test that depends on
  the interception. The wrapper IS load-bearing; q-6 docstring updated
  to cite the test-fixture rationale instead of the thin "stack-trace
  identity" claim.

Deferred to follow-up (documented rationale):
- perf-2: single-pass partition for system-message resource list
  (concrete vs templates). Sub-microsecond at expected scale;
  opportunistic-only.
- q-2 (pre-push): ~200 lines of fixture infrastructure
  (``BehaviorMiddleware``, ``_build_server``, ``_seed_oauth_server``,
  ``running_loop_mgr``, etc.) duplicated across three pool-integration
  test files. Real maintenance cost, but a 200-line conftest extraction
  is a focused refactor that earns its own commit / PR. Tracking as
  follow-up rather than balloon Phase 7b's diff further.
- q-3 / q-4 (refactor): extract shared dispatcher / scheduler
  helpers to compress three near-identical 90-line bodies (round-1
  q-3 was the same root cause; the pre-push q-3/q-4 reviewer
  reaffirmed it concretely). Three named methods preserve readability
  for the codebase's hottest correctness path; follow-up if
  duplication grows further or if a per-path divergence ships.
- q-4 (round-1, distinct from pre-push q-4): split pool concerns
  into ``mcp_pool.py``. Out-of-scope per finder; future refactor as
  the file approaches the navigation/merge-conflict threshold.

3.13: 5590 passed (5541 baseline -> +49 net; pre-review +47, q-7
e2e tests added +2). Existing audit-detail tests updated in-place
to expect the new ``kind`` and ``code`` fields.
3.11: 5590 passed (parity gate per ``feedback_pytest_env_parity.md``).

(cherry picked from commit 124615cce0)
2026-05-07 17:35:22 -07:00
Patrick Buckley d767aca784 fix(metacog): atomic cap-and-drop helper for soft-cap producers
Closes PR #484 review findings (Copilot): the soft-cap pattern in
``ChatSession.set_watch_runner``'s dispatch closure was a non-atomic
two-call pair (``count_by_type`` then ``drop_oldest_by_type``) with
two separate lock acquisitions.  A concurrent drain on the worker
thread (``USER_DRAIN`` / ``TOOL_DRAIN`` consuming ``"watch_triggered"``
entries via the ``"any"`` channel) could slip between the two calls,
making the drop a no-op.  The dispatch closure also discarded
``drop_oldest_by_type``'s return value and unconditionally logged
``dropped_oldest=True``, so a no-op drop got reported as a successful
drop.

* New ``NudgeQueue.cap_at_or_drop_oldest(nudge_type, max_depth,
  channel=None) -> bool`` does the count+drop in a single critical
  section.  Returns the actual outcome.

* Dispatch closure (``session.py:1410-1416``) now calls the helper and
  uses its return value to gate the WARNING log line, so the log is
  accurate when a drop did NOT happen.

* ``drop_oldest_by_type``'s docstring no longer overstates the
  per-call lock as covering a count+drop pair — it points readers
  to ``cap_at_or_drop_oldest`` for that contract.

7 new tests in ``TestCapAtOrDropOldest`` cover: below-cap no-op,
at-cap drop-oldest, above-cap drop-only-one (per-call), channel
filter, other-type isolation, ``max_depth <= 0`` defensive no-op,
no-match.

5708 non-live tests pass; ruff + mypy clean.

The github-code-quality bot finding ("Statement has no effect" on
``_protocol.py:939``'s ``...`` body) is a false positive — every
Protocol method in ``_protocol.py`` uses ``...`` as its body, which
is the canonical Python Protocol pattern.  Replacing with ``pass``
would diverge from the file's existing style.  No code change.

(cherry picked from commit c757c22f55)
2026-05-07 17:35:22 -07:00
Patrick Buckley 12bc580dee fix(metacog): factor sanitiser regex tail + trim docstrings + drop tombstone
Closes round-2 review findings q-3, q-4, q-5, q-7.

* **q-4:** ``_NAME_CONTROL_CHARS`` and ``_PAYLOAD_CONTROL_CHARS`` shared
  7 lines of Unicode-steering character classes (zero-width / bidi /
  separators / BOM / tag chars above BMP).  Factored into a single
  ``_CONTROL_CHARS_TAIL`` constant; each regex now differs only in its
  leading ASCII range.  Future bidi or zero-width additions edit one
  place.

  Side effect: this corrects a latent bug where ``_NAME_CONTROL_CHARS``
  had two literal ASCII spaces in place of U+2028 / U+2029 (line and
  paragraph separators) — visible as ``r"  "`` in source but rendered
  as the actual codepoints in ``_PAYLOAD_CONTROL_CHARS``.  After the
  factoring both regexes correctly include U+2028 / U+2029, closing
  the gap that would have let a workstream name with embedded line
  separators forge a sibling bullet (the same vector ``\n`` was
  blocked for in the original bug-1 fix).

  Switched to ``\u`` escapes for readability (and to keep future Edit
  tool runs against this block reliable).

* **q-3:** Tombstone clause "standing in for the deleted
  ``_watch_pending`` maxsize bound" survived in
  ``ChatSession.set_watch_runner``'s docstring after the apply-pass
  trim cleaned the inline soft-cap comment.  Dropped.

* **q-5:** ``test_newline_in_name_does_not_forge_extra_bullet`` carried
  five WHAT-narration comments restating what the immediately-following
  asserts already say.  Dropped — the docstring carries the security
  invariant; the assertions speak for themselves.

* **q-7:** ``patch_session_storage`` had a 14-line docstring including
  fallback-guidance and self-justification ("accumulated 7 near-duplicate
  sites").  Trimmed to a 3-line contract.

(cherry picked from commit 39e0f930c1)
2026-05-07 17:35:22 -07:00
Patrick Buckley aa1446364b test(metacog): drop redundant valid_until test + tighten concurrency bound + cover is_watch_active
Closes round-2 review findings q-1, q-2, q-6.

* **q-1:** ``test_valid_until_drops_when_watch_missing`` collapsed to the
  same code path as ``test_valid_until_drops_when_watch_inactive`` after
  the apply-pass switched the predicate from ``get_watch[active]`` to
  ``is_watch_active`` (both stubbed via ``patch_session_storage(active=False)``).
  The "missing" case has no distinguishable branch at the dispatch
  layer, so dropping it removes a tautological duplicate.  The
  missing-row mapping moves to the storage layer (q-2 below) where it
  IS distinguishable.

* **q-2:** ``is_watch_active`` was a new public storage primitive with
  zero direct backend coverage — only via-session-via-stub coverage.
  New ``TestIsWatchActive`` in ``tests/test_watch_storage.py`` covers
  active row → True, inactive row → False, missing row → False.
  Pinned at the storage boundary so future backend changes fail loudly
  there instead of in the dispatch tests.

* **q-6:** Concurrency test had ``n_threads = 2`` alongside two literal
  Thread objects and a tautological ``assert len(threads) == n_threads``.
  Threads are now built from a labels tuple, so ``len(threads)`` drives
  the slack bound; the redundant assertion is gone.

(cherry picked from commit 751ed9c85f)
2026-05-07 17:35:22 -07:00
Patrick Buckley 21507dc02e fix(metacog): tighten concurrency bound + lift storage-patch helper
Closes review findings bug-4 and q-6.

bug-4 — the watch dispatch concurrency test bounded depth at
``_WATCH_QUEUE_SOFT_CAP + 2 * per_thread`` (= 250) which is
tautologically true: two threads × 100 fires can append at most 200
entries above the cap, so the bound asserted nothing more than what
``depth <= 2 * per_thread`` already says.  Tighten to
``_WATCH_QUEUE_SOFT_CAP + N_THREADS`` (= 52): the count-then-drop window
admits at most one slip per concurrent thread.

q-6 — 7 near-duplicate ``monkeypatch.setattr(session_mod, "get_storage",
lambda: _StubStorage())`` sites across ``test_watch_dispatch.py`` +
``test_watch_integration.py`` (4 different stub shapes, mostly trivial
variations on the active flag).  Lift a ``patch_session_storage``
helper into the existing ``tests/_helpers.py`` with kwargs for the
common cases (``active``, ``raise_on_is_active``), returns the call list
so call-shape assertions still work.  Tests collapse from ~10-line
inline-class blocks to one-line helper calls.

(cherry picked from commit 20c4dfaca6)
2026-05-07 17:35:22 -07:00
Patrick Buckley a0ed4b9897 fix(metacog): drop watch_id rebind + trim soft-cap inline comment
Closes review findings q-2 and q-5.

q-2 — ``bound_watch_id = watch_id`` rebind was unnecessary.  ``_dispatch``
is constructed fresh per fire (not in a loop), so ``_still_active``
closes over the function parameter directly without any
loop-variable-capture risk.  Drop the rebind.

q-5 — the inline soft-cap comment restated rationale already covered by
the ``_WATCH_QUEUE_SOFT_CAP`` block-comment at module scope and dragged
in a tombstone reference to the deleted ``_watch_pending`` path.  Trim
to one line stating only the WHY (drop-oldest because latest output is
most useful).  Leave the ``set_watch_runner`` docstring's operational
detail at lines 1356-1378 alone — trimming further risks losing the
``valid_until`` predicate semantics.

(cherry picked from commit 28d9bb4802)
2026-05-07 17:35:22 -07:00
Patrick Buckley ab8ee0d759 test(metacog): integration coverage for _watch_restore_fn closure
Closes review finding q-4.

The closure built inside ``server.py``'s ``_watch_restore_fn`` is the
new contract surface introduced by the switchover — it constructs a
fresh ChatSession, calls ``session.resume(ws_id)`` to adopt the
original ws_id, re-registers the dispatch closure via
``set_watch_runner``, and returns ``WatchRunner.get_dispatch_fn`` for
the runner to invoke directly.  No automated coverage exists today;
a future refactor (e.g. swapping ``manager.create + session.resume``
for ``manager.open``) could silently break the watch-restore pipeline.

Adds ``test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session``
to ``tests/test_watch_integration.py`` — drives the full restore path:
persists a kickoff message for the original ws_id, fires
``_dispatch_result`` against a runner with no registered dispatch fn,
asserts the restore_fn ran exactly once, the rehydrated session is a
distinct object that adopted the original ws_id, and the watch payload
landed on the rehydrated session's NudgeQueue (not on the original).

(cherry picked from commit ed1eaee216)
2026-05-07 17:35:22 -07:00
Patrick Buckley d34f6cd0b1 fix(metacog): is_watch_active storage primitive for hot-path valid_until
Closes review finding perf-1.

The watch dispatch closure's ``valid_until`` predicate fires once per
watch entry at every drain seam — on the chat-loop hot path.  It only
needs the ``active`` flag, but ``storage.get_watch`` runs a full-row
``SELECT *`` and marshals the result into a dict.  At the typical drain
depth (cap-50 + a busy chat loop) that's ~50 throwaway dict allocations
per drain pass for one boolean.

Adds ``StorageProtocol.is_watch_active(watch_id) -> bool`` plus
SQLite + Postgres implementations doing a single-column
``SELECT active FROM watches WHERE watch_id = ?`` (returns False on
missing row).  ``_still_active`` in ``ChatSession.set_watch_runner``
now calls that instead of indexing into the full row.

Test stubs that mocked ``get_watch`` for the predicate are converted
to mock ``is_watch_active`` directly.  Bulk variant deferred — single-row
fix is sufficient at typical drain depths.

(cherry picked from commit 3b495eba15)
2026-05-07 17:35:22 -07:00
Patrick Buckley b219c47ba8 fix(metacog): NudgeQueue.count_by_type primitive + channel-aligned soft cap
Closes review findings perf-2, q-3, bug-3.

The watch dispatch closure's soft-cap pre-check materialised the whole
queue snapshot via ``pending(channel="any")`` only to throw away the
text and count the type — wasteful at typical drain depths (cap-50 +
mixed producers means a 50-tuple allocation per fire just to read a
length).  The other half of the cap pair (``drop_oldest_by_type``)
walked the *whole* queue regardless of channel, so a future producer
that enqueued ``"watch_triggered"`` on a different channel could be
dropped by the watch cap, and vice versa — silently surprising once
that producer existed.

Adds ``NudgeQueue.count_by_type(nudge_type, channel=None) -> int`` that
walks ``_items`` once under the queue lock without materialising
tuples; extends ``drop_oldest_by_type`` to take an optional ``channel``
filter so both halves can agree on the entry set being capped.  The
watch dispatch closure now passes ``channel="any"`` to both —
consistent with where the closure enqueues — so a future channel split
can't bleed across producers.

Adds ``TestCountByType`` mirroring the existing ``TestDropOldestByType``
shape, plus a ``test_drop_oldest_by_type_channel_filter`` case pinning
the new optional argument's behaviour.

(cherry picked from commit e5e6e13307)
2026-05-07 17:35:21 -07:00
Patrick Buckley d770a811a8 fix(metacog): drop test_watch_live.py — defer R9 to operator-driven verification
Closes review finding q-1.

The live-marker scaffold in ``tests/test_watch_live.py`` couldn't actually
run as written: the ``live_client`` / ``live_model_id`` fixtures it
referenced live in ``tests/test_server_live.py`` at ``scope="module"``,
not on a shared ``conftest.py``, so the file would have ImportError'd
at collection if anyone ever tried ``pytest -m live`` against it.

Lifting the fixtures into a shared conftest is a larger refactor
than R9 justifies — the deterministic envelope-arrival contract is
already pinned end-to-end by ``test_watch_fires_then_user_send_drains_envelope``
and ``test_three_back_to_back_watch_fires_drain_into_one_turn`` in
``test_watch_integration.py`` (real ChatSession + real WatchRunner +
real chat-loop drain).  The model-quality-of-response leg is genuinely
manual; the plan doc's R9 entry is updated locally to reflect that
deferral.

(cherry picked from commit 68a44cc7e2)
2026-05-07 17:35:21 -07:00
Patrick Buckley 912e9c57b0 fix(metacog): split sanitiser regex — strict for names, permissive for payloads
Closes review finding bug-1.

The shared ``sanitize_payload`` regex preserved TAB/LF/CR so multi-line
watch shell output kept its layout — necessary for the watch path, but a
correctness gap for the idle_children formatter, which renders the
user-controlled ``name`` field as a single bullet item.  A child name
with an embedded ``\n`` would split the bullet across two rendered rows
and let a hostile name forge a fake sibling entry in the listing.

Splits the regex in two: ``_NAME_CONTROL_CHARS`` strips TAB/LF/CR
(used by the new ``sanitize_name`` helper for single-line name fields),
``_PAYLOAD_CONTROL_CHARS`` keeps the existing permissive shape (used by
``sanitize_payload`` for multi-line watch payloads).
``format_idle_children_nudge`` now calls ``sanitize_name``.

Adds ``test_newline_in_name_does_not_forge_extra_bullet`` — feeds a
hostile name with embedded ``\n`` + bullet-shaped continuation, asserts
the rendered listing still has exactly N bullet rows for N children
(no forged sibling), and the hostile newline got flattened to an inline
space.  Adds a ``TestSanitizeName`` class mirroring the existing
``TestSanitizePayload`` shape for the new strict variant.

(cherry picked from commit e596650a5c)
2026-05-07 17:35:21 -07:00
Patrick Buckley d7c6053441 fix(metacog): drop misleading _watch_restore_fn comment
The deleted comment claimed the closure may be registered "under the
rehydrated workstream's id, which may differ from the original ws_id we
restored against" — but ``ChatSession.resume(ws_id, fork=False)`` adopts
the parameter as the session's id at session.py:1682, so they match
exactly post-resume.  The lookup works because the ids are equal, not
because they may differ.

The accessor name ``get_dispatch_fn`` is self-explanatory; no replacement
comment is needed (per the project's "default to no comments" rule).

(cherry picked from commit d2028aa4f7)
2026-05-07 17:35:21 -07:00
Patrick Buckley e7a17a20b0 test(metacog): watch switchover boundary integration + live scaffold
Adds two boundary-crossing integration tests and one live-marker
scaffold for the watch switchover landed in the previous commits:

tests/test_watch_integration.py — drives a real ChatSession + real
WatchRunner end-to-end (LLM stubbed) through the unified pull-model
chat-loop drain seam.  Pins:

- test_watch_fires_then_user_send_drains_envelope: a synchronous
  WatchRunner.dispatch fire enqueues "watch_triggered" on "any";
  session.send drains the entry into the user message's _reminders
  side-channel — confirms the envelope splice path.
- test_three_back_to_back_watch_fires_drain_into_one_turn: pins the
  intentional behavioural delta from the plan section 3.4 / risk
  register R3 — N back-to-back fires now produce ONE assistant turn
  with N _reminders entries, not N successive turns.

tests/test_watch_live.py (new file, single test, marked @pytest.mark.live):
risk register R9 verification recipe — confirm a real LLM handles a
<system-reminder>-framed watch payload sensibly.  Collects under the
regular -m "not live" run; the user runs it on demand against an
Anthropic-backed config.

Implements watch-switchover plan section 5.2 (integration) and step 11
(live scaffold).

(cherry picked from commit 17c62f7ef3)
2026-05-07 17:35:21 -07:00
Patrick Buckley 931a1eca9d test(metacog): NudgeQueue-based dispatch tests for watch closure
Replaces the deleted tests/test_watch_dispatch.py with a focused
14-test suite exercising the closure that ChatSession.set_watch_runner
now constructs (per the previous commit's switchover).  Each test
pins one assertion:

- enqueue shape: ("watch_triggered", text, "any") on the per-session
  NudgeQueue; not on user / tool channels
- producer-side sanitisation strips control / bidi / zero-width chars
  and angle-bracket tag breakers; preserves TAB/LF/CR so multi-line
  shell output keeps its layout (R8); empty-after-strip → no enqueue
- soft-cap drop-oldest at _WATCH_QUEUE_SOFT_CAP with a queue_full
  WARNING log; non-watch entries on the same queue are not collateral
  damage
- valid_until predicate drops on inactive / missing / storage-raises;
  delivers when active (counter-test)
- concurrent enqueues across two threads stay bounded under the
  3-acquisition count-then-drop window

Implements watch-switchover plan section 5.1 / step 9.  No production
changes — pure test rewrite.

(cherry picked from commit 7ca00b564c)
2026-05-07 17:35:21 -07:00
Patrick Buckley 048285a423 feat(metacog): switchover — watches enqueue onto NudgeQueue not _watch_pending
Replaces the bespoke _make_watch_dispatch / _watch_pending /
_dispatch_pending_watch / _MAX_WATCH_CHAIN machinery with a single
NudgeQueue.enqueue("watch_triggered", ...) call inside
ChatSession.set_watch_runner.  Watch results now drain at the same
<system-reminder> envelope seams as every other metacog nudge
(USER_DRAIN, TOOL_DRAIN, IdleNudgeWatcher IDLE wake) — no separate
worker-spawn, no recursive watch chain, no per-session queue.Queue.

The dispatch closure built inside set_watch_runner carries:
- producer-side sanitize_payload over the whole formatted message
  before enqueue, so steering-vector / control-char shell output
  can't tamper with the envelope at interpolation time
- a soft cap of 50 entries on per-session "watch_triggered" depth
  via the new NudgeQueue.drop_oldest_by_type, replacing the prior
  _watch_pending maxsize=20 + _MAX_WATCH_CHAIN=5 bounds; drop policy
  is drop-oldest (latest output most useful), logged at WARNING
- a valid_until predicate that re-checks
  storage.get_watch(watch_id)["active"] at drain time so a cancelled
  watch's last splat doesn't ride out a future wake

Behavioural delta documented in the plan section 3.4: N back-to-back
watch fires now drain into ONE assistant turn responding to all N
(via the envelope splice) instead of N separate send turns.  This is
intentional — fewer model invocations for noisy watches, and uniform
with the rest of the metacog pull-model surface introduced by #482.

Implements watch-switchover plan steps 5-8.  Server-side simplifications
let the previously-load-bearing _make_watch_dispatch (47 lines), its
session_worker.send import, and the chat-loop _dispatch_pending_watch
seam at the no-tools IDLE branch all disappear.  The obsolete
tests/test_watch_dispatch.py and the wake-tag test in test_session.py
(both pinning contracts that no longer exist) are removed; the
NudgeQueue-based replacement plus an integration test land in the
following commit.

(cherry picked from commit 94ed79d488)
2026-05-07 17:35:21 -07:00
Patrick Buckley 481347eb17 refactor(metacog): widen WatchRunner dispatch_fn signature to (msg, watch_id)
Widens the per-workstream dispatch fn signature from ``(message,)``
to ``(message, watch_id)``.  The runner now passes the originating
``watch_id`` through ``_dispatch_result`` so dispatch closures can
capture per-watch metadata at fire time — the upcoming switchover
needs this for the ``valid_until`` predicate that re-checks
``storage.get_watch(watch_id)["active"]`` before a stale entry rides
out a wake.

Also adds ``WatchRunner.get_dispatch_fn(ws_id)`` as the public
accessor used by the server-side restore path to retrieve the
closure that ``set_watch_runner`` constructed during workstream
rehydrate (avoiding private-attr access into ``_dispatch_fns``).

Implements watch-switchover plan step 4 plus risk register R4.
The pre-existing single-arg callers (``_make_watch_dispatch`` and
``set_watch_runner``'s ``dispatch_fn=`` fallback) get replaced
in the next commit; their mypy types are ``Any`` today so the
type mismatch isn't caught at this step.

(cherry picked from commit 195ff985cc)
2026-05-07 17:35:21 -07:00
Patrick Buckley 31a554a4bd refactor(metacog): shared sanitize_payload + watch_triggered nudge type
Renames _sanitize_child_name to sanitize_payload and widens it to be
the shared producer-side sanitiser for both idle_children and the
incoming watch_triggered nudges.  The regex now skips TAB / LF / CR
so multi-line shell output rendered into a watch payload keeps its
line structure when sanitised as a whole formatted message — the
pre-switchover code path collapsed multi-line output to one line.

Adds the watch_triggered entry to _NUDGE_MAP alongside idle_children
so ``_NUDGE_MAP``-as-registry consumers (should_nudge gating, future
audit / UI tagging) recognise the type.  Body is empty — payload
comes from the producer (the watch dispatch closure), same shape as
idle_children.

Implements watch-switchover plan section 3.2 plus risk register R8
(TAB/LF/CR exclusion) and step 3 (_NUDGE_MAP registration).

(cherry picked from commit 78ae7ae6b5)
2026-05-07 17:35:21 -07:00
Patrick Buckley af2c0ae13a feat(metacog): NudgeQueue.drop_oldest_by_type helper for soft-cap producers
Adds an atomic drop-oldest-by-type operation to NudgeQueue used by
producers that need a per-type soft cap on their own queue depth.
The watch dispatcher (next commit in this stack) is the first user:
when "watch_triggered" saturates, the dispatch closure drops its
oldest entry under the queue lock so the count snapshot and drop
can't interleave with a concurrent enqueue from the same producer.

Implements watch-switchover plan section 3.1 — the producer-side soft
cap takes the place of the deleted _watch_pending maxsize=20 bound.
Other producers (idle_children, advisories) have natural rate limiters
already, so the helper is opt-in per producer rather than a global cap
in enqueue itself.

(cherry picked from commit 74f1958e47)
2026-05-07 17:35:21 -07:00
Patrick Buckley 0808dc0af0 fix(mcp): apply Phase 7 PR review feedback
Three Copilot findings on PR #483 (commit dad98c0); one rejected as a
false positive.

- mcp_client.py:1189 — pool notification handler's exception path
  used ``log.warning(..., exc_info=True)`` which serializes the
  chained ``httpx.Request.headers`` carrying ``Authorization: Bearer
  <token>`` into Sentry / faulthandler frame captures. Same threat
  model as the round-1 sec-1 dispatch-path fix, applied to a site
  the original review missed. Now logs structured fields only
  (server, user, exc type) without ``exc_info``.

- mcp_client.py:1202 — ``_connect_one_pool``'s handshake step used
  ``asyncio.wait_for(session.initialize(), ...)``, the same Python
  3.11 + anyio cross-task-cancel-scope anti-pattern that the
  Phase 7 round-3 q-1 fix removed from the discovery step (and that
  f6a3b66 originally addressed for ``_safe_close_stack``). Pre-
  existing Phase 5 code, but the same latent bug class — a 401
  during initialize() under 3.11 would surface ``RuntimeError:
  Attempted to exit cancel scope in a different task`` as the
  SDK's TaskGroup unwinds. Switched to ``async with asyncio.timeout(...)``
  matching the discovery step's pattern.

- mcp_client.py:1522 — renamed loop tuple-unpack variable
  ``_server_name`` → ``server_name`` in ``_rebuild_user_tool_map``.
  The leading underscore conventionally signals "intentionally
  unused", but the variable is read at the assignment a few lines
  below. Two other ``_server_name`` unpacks in this file (1410,
  3111) genuinely don't use the value and keep the underscore.

Rejected as false positive:
- test_mcp_user_catalog.py:58 (github-code-quality bot, "Statement
  has no effect"): ``await task`` inside ``contextlib.suppress(
  BaseException)`` is the standard pattern for cleanly draining a
  cancelled task. The bot's static analysis treats ``await`` of a
  result that's discarded as a no-op statement, but ``await`` here
  triggers cancellation propagation and waits for the task to
  finish — load-bearing in the fixture's teardown. No change.

Verified on Python 3.11 (``/tmp/venv311``) and 3.13 (``.venv``):
ruff + mypy clean, full test suite green.

(cherry picked from commit 62909d402c)
2026-05-07 17:35:21 -07:00
Patrick Buckley cfc8a6c8c0 feat(mcp): per-user catalog scoping (Phase 7 — tools)
Light up production reachability of pool dispatch (RFC §3, invariant 8)
by widening the public catalog API to optionally take a ``user_id``:

- ``MCPClientManager.get_tools(user_id=None)`` returns the merged
  static + per-user pool view when ``user_id`` is supplied; the default
  preserves the legacy global-only contract.
- ``is_mcp_tool(name, *, user_id=None)`` extends the lookup to the
  per-user ``_user_tool_map``. Pool tools become reachable from
  ``ChatSession._prepare_tool`` only when the session-bound user_id
  flows through — flipping invariant 8 from "must hold" to "satisfied".
- Listener identity becomes ``(user_id, callback)``. Static-path
  changes fire ALL listeners (admin + every user); pool-entry
  changes fire only matching-user + admin (``None``) listeners.
  RFC §3.3.
- Pool sessions discover their tool list on first connect
  (``_connect_one_pool`` → ``await session.list_tools()``); the
  notification closure binds to ``(user_id, server_name)`` so
  push-driven ``list_changed`` updates target the correct user's
  catalog. R6 verified empirically: ``list_tools()`` 401 propagates
  through anyio TaskGroup unwinding, no hang — plain ``await`` is
  fine, no carrier-race shape needed for discovery.
- ``_evict_session`` drops ``entry.tools`` and rebuilds the user's
  index so an evicted-then-reconnected session doesn't carry
  stale catalog state.
- ``web_search.resolve_web_search_client`` refuses
  ``auth_type=oauth_user`` backends (per-node web search can't
  carry per-user tokens).

Resources / prompts pool dispatch deferred to Phase 7b — invariant 8
is satisfied by the tool path alone, and the resource/prompt path
needs sibling ``_dispatch_pool_resource_sync`` /
``_dispatch_pool_prompt_sync`` helpers each with their own
carrier-race plumbing (~400 LOC). Phase 7b will follow the patterns
established here.

CLI sessions default ``user_id=""`` and so cannot use oauth_user
MCP servers — documented limitation; users must use the web UI.

Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: get_tools(user_id) was iterating _user_pool_entries from sync
  threads while the mcp-loop concurrently mutated it (RuntimeError:
  dictionary changed size during iteration). Now reads from a sibling
  _user_tools dict updated atomically by _rebuild_user_tool_map.
- bug-2: _close_pool_entry_if_idle (LRU/TTL eviction) skipped the
  catalog cleanup that _evict_session does — stale tools persisted
  in _user_tool_map and ChatSession's tool list never rebuilt. Now
  mirrors _evict_session.
- perf-1: _last_pool_notification_refresh debounce dict was never
  pruned in either eviction path. Now popped alongside the entry.
- perf-3: web_search resolver was issuing a sync SQL query per LLM
  turn to gate oauth_user backends. Now reads from the cached
  in-memory config.
- sec-1: bearer token could leak into exc_info-rendered tracebacks
  via Sentry/faulthandler. log.debug now uses structured fields,
  not exc_info.
- sec-2: tools-per-server response now capped at 1000 (defensive,
  mirrors _MAX_ERROR_LEN / _MAX_INSUFFICIENT_SCOPE_REPORTED).
- Test cleanup: dropped two listener fan-out tests duplicating
  test_mcp_client.py coverage; renamed test_pool_session_notification_handler
  to match its actual scope (_refresh_pool_server_tools); removed
  stale comments referencing /tmp/r6-spike*.py scratchpads and a
  misleading "copy-on-write" comment.

Round-2 pre-push review fixes (focused single-pass review applied):
- round2-1: bug-2's catalog-cleanup block in _close_pool_entry_if_idle
  had no integration test (exactly the failure mode flagged in
  feedback_tests_through_boundaries.md). Added
  test_close_pool_entry_if_idle_clears_catalog_and_fires_listener
  driving the LRU/TTL eviction path through real streamablehttp_client +
  MockTransport. Negative-test verified: reverting the
  _rebuild_user_tool_map / _notify_user_tool_listeners calls makes
  the new test fail.
- round2-3: documented the _oauth_user_server_names cache invariant
  in add_server_sync / remove_server_sync docstrings. Cache is
  reconcile_sync's sole owner — direct callers leave it stale, but
  _db_servers_to_config strips oauth_user rows so production paths
  are unaffected. Static→oauth_user transitions correctly leave the
  name in the cache because remove_server_sync drops the static
  connection, not the cache identity.
- round2-6: strengthened test_rebuild_user_tool_map_populates and
  test_rebuild_user_tool_map_drops_empty_user to assert on the
  _user_tools sibling cache (bug-1 fix). Without this, a future
  revert dropping the sibling write would still pass the unit
  tests because get_tools coverage lives in separate tests.

Round-3 full-stack review fixes (multi-stage review on the final
state caught what the layered apply passes missed):
- q-1 REGRESSION: pool tool-discovery used asyncio.wait_for around
  session.list_tools(), the exact pattern the f6a3b66 fix (and
  feedback_asyncio_timeout_vs_wait_for.md) put in place to avoid.
  Python 3.11's asyncio.wait_for wraps the inner coroutine in a
  fresh task → cross-task scope-exit when the SDK's anyio TaskGroup
  unwinds on a 401. Switched to `async with asyncio.timeout(...):`
  pattern used by _safe_close_stack.
- sec-2: TOCTOU in _connect_one_pool — entry.tools was published
  (via _rebuild_user_tool_map + listener fan-out) BEFORE entry.session
  was assigned. A sync-thread reader could observe a tool whose
  backing entry has session=None. Defence-in-depth — dispatch
  re-fetches its own token and lazy-reconnects on session=None — but
  reordering catches the race at the source. entry.session now
  publishes BEFORE catalog visibility.
- bug-1: _close_pool_entry_if_idle's _user_pool_locks.pop ran
  unconditionally after the try/finally, but the early-return
  branches (entry None on re-check, in_flight > 0 under lock) skip
  it via Python's return-through-finally semantics. The lock was
  never popped on those paths. Now gated behind an `evicted` flag
  set only on the success path; in_flight > 0 leaves the lock for
  the active dispatcher to reuse, entry-None races leave the lock
  for re-allocation by _ensure_pool_entry. Comment now describes
  the actual semantics, not the original promise.
- bug-2: softened the _rebuild_user_tool_map docstring's atomicity
  claim. The two-dict write is technically non-atomic across Python
  statements; in practice the window is sub-microsecond on the
  mcp-loop with no awaits between writes, and the listener fan-out
  fires AFTER both writes complete. Docstring now says "back-to-back
  on the mcp-loop" instead of "atomically alongside".
- q-3: dropped `hasattr(mcp_client, "server_auth_type")` defensive
  check in web_search.py. The method ships in this commit; the
  hasattr created a silent fallthrough that would let a future
  rename silently re-enable oauth_user backends.
- q-4: surfaced the CLI / empty-user_id limitation in a docstring
  comment at ChatSession.__init__'s self._user_id assignment. The
  note previously lived only inside is_mcp_tool's docstring — a
  future maintainer wiring CLI features against MCP pool servers
  wouldn't think to read is_mcp_tool to find the constraint.
- q-2 + q-5: deleted a tautological duplicate test in
  test_mcp_user_catalog.py whose docstring claimed to test
  ChatSession.close but never instantiated a ChatSession (the
  manager-level identity semantics are already covered by
  test_listener_identity_includes_user_id in the same file and by
  test_session_close_removes_listener_with_same_user_id in
  test_mcp_client.py which DOES drive a ChatSession). Reworded a
  misleading "fixture provides only 5s" comment to point at the
  actual `_run_on_loop(..., timeout=5)` site.
- q-6: the `self._user_id or None` collapse repeated at 8 sites
  across session.py. Cached once at __init__ as
  ``self._mcp_user_id`` (since ``_user_id`` is set once and never
  mutated); 8 call sites now read the cached value. The empty-
  string-is-CLI-sentinel invariant is documented at the assignment
  site, not re-asserted at each consumer.

Deferred to follow-up:
- sec-1: a hostile MCP server bound to user-A could craft a
  tool.name containing `__` to synthesize a prefixed-name collision
  in user-A's own catalog. Bounded impact: cross-tenant dispatch is
  prevented by the per-tenant token gate in _dispatch_pool, and
  user-B's get_tools(user_id="B") never includes user-A's pool
  entries. The fix needs policy decisions (reject vs. sanitize)
  and touches _mcp_to_openai which is shared between static and
  pool paths; better discussed in its own follow-up where the
  policy applies uniformly to static-path servers too. The threat
  model already requires user-A to have consented to a malicious
  server, who has many more dangerous vectors than tool-name
  shenanigans.

Test count delta: +31 tests (5435 → 5466, ``-m "not live"``; one
test deleted in round-3 apply per q-2):
- ``tests/test_mcp_client.py`` +20 (per-user catalog state, listener
  identity, session thread-through)
- ``tests/test_mcp_user_catalog.py`` +9 NEW (integration tests
  driving real ``streamablehttp_client`` + ``httpx.MockTransport`` per
  invariant 14: discovery on connect, user isolation, eviction +
  reconnect, LRU/TTL eviction (round2-1), R6 401-propagation
  regression, static byte-identical canonical regression; review
  passes dropped duplicate listener fan-out tests from earlier
  drafts whose coverage lived in test_mcp_client.py)
- ``tests/test_web_search.py`` +2 (oauth_user backend rejection +
  static backend acceptance regression; updated to use the new
  ``server_auth_type`` in-memory accessor)

(cherry picked from commit a8b34bfe54)
2026-05-07 17:35:21 -07:00
Patrick Buckley 266e3536aa fix(metacog): bot-review fixes — watcher gate + two stale docstrings
Three confirmed findings from the PR #482 bot review pass.

* **Copilot (idle_nudge_watcher.py)**: ``IdleNudgeWatcher`` was gating
  wake dispatch on ``len(_nudge_queue) == 0`` (any channel), but
  ``deliver_wake_nudge_from_queue`` only drains ``USER_DRAIN``.  A
  ``"tool"``-channel entry queued by ``_queue_tool_advisory`` would
  pass the gate, spawn a wake daemon, and immediately no-op at the
  drain guard — repeating on every IDLE event for as long as the
  tool entry sat unconsumed.  No correctness bug (the no-op return
  prevents bad state) but a wasted thread spawn per IDLE.  Fixed by
  gating on ``has_pending(USER_DRAIN)``; tool-only queues no longer
  trigger the wake path.

* **Copilot (coordinator_idle_observer.py)**: docstring referenced
  the old module path ``turnstone.core.metacognition.IdleNudgeWatcher``;
  the class moved to ``turnstone.core.idle_nudge_watcher`` in q-3 of
  the apply-pass.

* **Copilot (nudge_queue.py)**: ``has_pending`` docstring cited
  ``ChatSession.deliver_wake_nudge_from_queue`` as its caller, but
  that method calls ``drain(USER_DRAIN)`` directly — no production
  caller used ``has_pending`` until this commit.  Updated to point
  at the now-actual caller (``IdleNudgeWatcher``).

* **github-code-quality (test_nudge_queue.py)**: false positive on
  ``test_channel_is_required`` — the no-channel ``q.enqueue("a", "1")``
  call is wrapped in ``pytest.raises(TypeError)`` to verify the
  validation contract.  No code change.

5571 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 0fbf31e713)
2026-05-07 17:35:21 -07:00
Patrick Buckley 42bf9aecaf fix(metacog): apply-pass fixes from pre-push full-stack review
Round-2 review caught 11 confirmed findings on the 3-commit metacog stack;
this commit applies them.

* **bug-1 (major)**: Wake source tag was leaking onto real user messages
  flushed during a wake send.  ``_append_user_turn`` and ``send`` now
  take an explicit ``from_wake: bool`` parameter — only the wake's
  synthesized first turn passes True, so ``_flush_queued_messages``'s
  real user input no longer inherits the audit tag.  Regression test
  pins the contract.

* **perf-1 (major)**: ``CoordinatorIdleObserver._maybe_enqueue`` was
  issuing list_workstreams + visible_memory_count storage queries
  before the cheap cooldown gate could short-circuit.  New
  ``_cooldown_allows`` read-only peek runs first; storage queries only
  fire when cooldown actually allows the nudge.

* **q-1 (major)**: Added the missing coord-side integration test that
  exercises ``CoordinatorIdleObserver`` + ``IdleNudgeWatcher`` together
  in the production install order against a real ``SessionManager``,
  protecting the subscription-order contract from silent regression.

* **perf-2/3 (minor)**: Cap check moved above ``_last_assistant_used_wait``;
  ``_fire_counts`` restructured as ``dict[str, dict[str, int]]`` keyed by
  ws_id so the leave-IDLE existence check is O(1).

* **perf-4 (minor)**: ``NudgeQueue.drain`` fast-paths the all-match
  case (the common one for chat-loop drain seams) by swapping
  ``self._items`` directly instead of allocating a fresh ``kept``
  deque + per-entry append.

* **perf-5 (minor)**: Wake's synthesized empty user turn no longer
  writes a content-empty row to the conversations table — the
  ``_source`` audit tag isn't column-backed and the side-channel
  reminder is stripped before persist, so the row would carry nothing.

* **q-3 (minor)**: Split ``IdleNudgeWatcher`` + ``install_*`` /
  ``shutdown_*`` helpers out of ``metacognition.py`` into the new
  ``turnstone/core/idle_nudge_watcher.py``; metacog stays a
  static-template module.

* **sec-1 (nit)**: Widened ``_sanitize_child_name``'s control-char
  regex to cover Unicode bidi-overrides, zero-width chars,
  line/paragraph separators, BOM, and tag chars.

* **q-4/q-5 (nits)**: Docstring referenced the wrong peek primitive
  (``has_pending`` → ``len()``); ``_last_assistant_used_wait``'s
  ``session`` parameter now typed ``ChatSession``.

5571 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 3f106f98b2)
2026-05-07 17:35:21 -07:00
Patrick Buckley 191775dd7e feat(metacog): coord idle-children nudge — observer + valid_until predicates
Adds the first concrete consumer of the wake trigger: when a coordinator
goes IDLE while interactive children are still running, a
``CoordinatorIdleObserver`` enqueues an ``idle_children`` nudge that the
``IdleNudgeWatcher`` then dispatches as a synthetic empty-user-turn
``send``.  The model receives a system-reminder body listing the active
children (capped at 6 inline + 32 in the suggested ``wait_for_workstream``
call) and a nudge to block on them rather than reply prematurely.

Observer gates (in order): coord-only filter, skip if last assistant
turn used ``wait_for_workstream``, per-(ws, nudge_type) hard cap (3)
that resets only on non-wake leave-IDLE, active-children query,
``should_nudge`` cooldown.  Console lifespan registers the observer
BEFORE the watcher so subscriber-fire order has the observer
enqueueing first on the same IDLE event.

Adds an opt-in ``valid_until`` predicate on ``NudgeQueue.enqueue``
(R9 from the design risk register) — drain re-checks the predicate
outside the queue lock; falsy / raising drops the entry without
delivering it.  ``deliver_wake_nudge_from_queue`` now drains inline
before synthesizing the empty user turn so a stale predicate-drop
doesn't leave the wake send with empty content; ``_attach_pending_user_reminders``
consumes the pre-drained reminders via ``_wake_drained_reminders``.

The observer's ``valid_until`` uses ``count_workstreams_by_state``
(boolean check, no row fetch) instead of full ``list_workstreams``,
keeping the chat-loop user-attach path off the heavy query.

User-controlled child workstream names are sanitized
(``_sanitize_child_name``) before interpolation so a name like
``</thinking>...`` can't steer the model's reasoning channels through
the rendered body — the wire-boundary ``escape_wrapper_tags`` only
covers ``<system-reminder>`` / ``<tool_output>`` envelopes.

(cherry picked from commit 908e67fe4f)
2026-05-07 17:35:21 -07:00
Patrick Buckley c41fd2be2e feat(metacog): wake trigger — IdleNudgeWatcher + ChatSession.deliver_wake_nudge_from_queue
Adds the third metacog channel: an out-of-band wake that converts a
workstream's IDLE transition into a synthetic empty-user-turn ``send``
when the session has any-channel nudges queued.  The ``IdleNudgeWatcher``
subscribes to ``SessionManager.subscribe_to_state``; on IDLE it dispatches
via ``session_worker.send`` with a no-op ``enqueue`` callback so a
busy-worker race silently drops without spawning a competing worker.

Wake-source-tag plumbing on ``ChatSession`` short-circuits metacog
detection on the synthetic empty input, suppresses queue producers
during the wake's own tool dispatch, and stamps ``_source = "system_nudge"``
on the synthetic user-message for audit / replay distinction.  The tag
is saved / restored across ``_dispatch_pending_watch`` so watch chains
recursing off the wake are processed as normal user turns rather than
inheriting the wake's guards.

Generic ``install_idle_nudge_watcher`` / ``shutdown_idle_nudge_watchers``
helpers wire the watcher into both the interactive and coord lifespans
via a single ``app.state`` registry so both surfaces share the same
teardown contract.

Foundation for PR 3 (CoordinatorIdleObserver + idle_children formatter)
and PR 4 (watch dispatcher switchover).

(cherry picked from commit f0e7fea549)
2026-05-07 17:35:21 -07:00
Patrick Buckley 1787fb5c11 refactor(metacog): unify advisory channels into pull-model NudgeQueue
Replaces the dual `_pending_user_advisories` / `_pending_tool_advisories`
list pair with a single channel-tagged `NudgeQueue` per session.
Producers tag entries with a channel ("user", "tool", or "any");
consumers drain by channel filter at their existing seams. Foundation
for the wake trigger (PR 2) and coordinator idle-children nudge (PR 3).

Existing nudges (start, correction, completion, denial, resume,
tool_error, repeat) keep their wire shape and drain timing — zero
behavior change. Cancel paths now `clear()` the unified queue.

(cherry picked from commit 94b3720916)
2026-05-07 17:35:21 -07:00
Patrick Buckley 814c42763d fix(mcp): asyncio.timeout (not wait_for) for safe-close-stack on Python 3.11
Python 3.11's ``asyncio.wait_for`` wraps its inner coroutine in a fresh
``asyncio.Task`` via ``ensure_future``. When the inner is
``stack.aclose()`` on an ``AsyncExitStack`` containing
``streamablehttp_client(...)`` (anyio cancel scopes entered in the
calling task), the fresh task's attempt to exit those scopes raises
``RuntimeError('Attempted to exit cancel scope in a different task
than it was entered in')``. Python 3.12+ rewrote ``wait_for`` to use
``asyncio.timeout`` internally — runs in the current task — so 3.13
ran the same code path successfully.

Symptom on 3.11: integration tests where ``session.initialize()``
returns 4xx (e.g., 403 insufficient_scope tests) hit
``_connect_one_pool``'s ``except Exception:`` handler →
``_safe_teardown_on_connect_failure`` → ``_safe_close_stack`` → cross-
task RuntimeError. The ``concurrent.futures._base.CancelledError``
that surfaces in ``future.result(timeout=...)`` is the cascade
fallout from the asyncio loop's exception handler reacting to the
unretrieved-task-exception.

Fix: use ``asyncio.timeout`` instead of ``asyncio.wait_for`` for the
5s aclose bound. Equivalent semantics, current-task execution, works
on 3.11+. The 5s guard against ``aclose()`` hanging on a broken stack
is preserved.

Verified on Python 3.11.14 (full suite 5427 passed) and 3.13.7 (full
suite 5427 passed); all 9 integration tests pass on both.

Pre-existing bug — surfaced only after the marker fix in 5c9850c
let CI's test (3.11) actually run the 4xx tests.

(cherry picked from commit f6a3b66ea4)
2026-05-07 17:35:21 -07:00
Patrick Buckley 242596ced3 fix(mcp): pool-reuse 401 — entry-owned carrier + race-and-cancel
Two pre-existing defects in the Phase 6 pool dispatch path that only
manifest when a pooled session is reused for a second dispatch:

1. The per-dispatch _AuthCapture allocated in _dispatch_pool was wired
   into the httpx response hook only at first connect (via
   _connect_one_pool). On a reused session no fresh connect runs, so
   the hook continues writing to the original-connect's carrier while
   the new dispatch inspects an empty carrier — auth_401/403 silently
   misclassified to "other", refresh-and-retry never fires.

2. Even with the carrier on the entry (so the hook writes to a stable
   reachable object), session.call_tool itself hangs forever on
   upstream 4xx for reused sessions. Trace: SDK's spawned
   handle_request_async raises HTTPStatusError, the outer
   streamablehttp_client TaskGroup cancels post_writer, post_writer's
   finally aclose's read_stream_writer, BaseSession's _receive_loop
   exits and enters its CONNECTION_CLOSED-fanout finally. anyio's
   send_nowait skips waiting receivers with pending_cancellation; the
   dispatch task (created by run_coroutine_threadsafe for the reuse
   case) is NOT in any cancel-scope chain, so the send "delivers" but
   the receiver's Event is set on stale state — receive() never
   wakes. Test 21 doesn't hit this because its 401 happens during
   initialize, in the same task that opens streamablehttp_client, so
   the cancel scope DOES propagate.

Fix:
- Move _AuthCapture ownership to PoolEntryState (and asyncio.Event
  alongside, allocated lazily on the mcp-loop). The hook closes over
  entry.auth_capture at first connect and stays valid across
  dispatches; reset under open_lock before each call_tool.
- Race session.call_tool against the carrier's fired_event in
  _dispatch_pool_with_entry. If the event wins (hook captured 4xx
  before SDK propagated), cancel call_tool and raise an internal
  _CarrierAuthSignal — _classify_failure resolves to auth_401/403
  via the carrier's status, the dispatcher evicts the broken
  session, and the cross-task retry handshake reconnects on a fresh
  bearer.

Adds tests/test_mcp_pool_auth_integration.py::test_integration_pool_reuse_401_refresh_and_retry_succeeds
which drives the reuse path through real upstream + real SDK and is
the structural gate against this class regressing. Negative-tested
twice: revert PoolEntryState.auth_capture → test fails (carrier
empty); revert the race → test times out (SDK hang).

Also drops the @pytest.mark.asyncio decorator (replaced with
@pytest.mark.anyio) on four tests in test_mcp_pool_auth_introspection.py.
The project depends on anyio's pytest plugin (anyio is in deps);
pytest-asyncio is NOT a project dep and CI's test (3.13) failed on
those four. Local pytest happened to pick it up via system Python.

Found via Copilot review on PR #481.

(cherry picked from commit 97086fc617)
2026-05-07 17:35:21 -07:00
Patrick Buckley bde0913442 feat(mcp): SDK 401/403 introspection via httpx response hook
Phase 6 of OAuth-MCP. Recovers upstream 401/403 from MCP servers via a
capturing httpx_client_factory: an async response hook records 4xx
status + WWW-Authenticate header into a per-dispatch carrier before
the SDK's post_writer swallows the underlying httpx.HTTPStatusError.

Splits _classify_failure into auth_401 (refresh-and-retry once) vs
auth_403 (parse insufficient_scope, emit mcp_insufficient_scope with
parsed scope set). The 401 retry runs on a fresh asyncio.Task via
run_coroutine_threadsafe in _dispatch_pool_sync, escaping the anyio
cancel-scope state of the prior dispatch's TaskGroup.

WWW-Authenticate parsing extracted to a new mcp_http_parsers module
with an RFC 7235 challenge tokenizer (replaces hand-rolled substring
scanners). Two-layer defense against multi-Bearer-challenge injection:
the hook uses get_list("www-authenticate")[0] to drop attacker's
second challenge, the parser truncates at challenge boundary as
belt-and-braces. Scope set capped at 32 entries before hitting the
audit row or the LLM-visible structured-error JSON.

Auth failures (401/403) never trip the per-server circuit breaker
(server-only breaker invariant). Static path remains byte-identical.
_PgRefreshLock untouched. Pool dispatch still reachable from the
agent loop only via Phase 7 catalog scoping; Phase 6 behaviour is
testable via direct call_tool_sync.

5557 tests pass. 33 tokenizer unit tests in tests/test_mcp_http_parsers
cover the RFC 7235 grammar + the scope/error wrappers + the 4 KB input
cap. 7 integration tests in tests/test_mcp_pool_auth_integration drive
real upstream 401/403 through streamablehttp_client + a FastMCP
subprocess fixture — the structural exit gate that makes
HTTPStatusError-injection-only unit tests insufficient.

(cherry picked from commit db9260d8c4)
2026-05-07 17:35:21 -07:00
Patrick Buckley 570b198f1b fix(man): accept canonical name(section) page notation
Models often emit page references in the standard man-page form
(``printf(3)``, ``open(2)``, ``perlfunc(3pm)``) rather than splitting
them into ``page`` + ``section`` args. The page-name sanitizer was
rejecting the parens as invalid input, killing the call. Parse the
section out of the page string before sanitization (explicit
``section`` arg still wins) and widen the section validator to accept
multi-letter suffixes like ``3pm`` / ``3perl`` that already appear on
real systems.

(cherry picked from commit 39a6b7b447)
2026-05-07 17:35:21 -07:00
Patrick Buckley 96d935f1f7 fix(mcp): cancellation-safe orphan-lock drain + lock-reorder + test integrity
Phase 5 PR #479 review fix-up. Three review rounds (bot + two internal
multi-stage /review) caught:

- _PgRefreshLock now allocates a per-instance ThreadPoolExecutor instead of
  a module-global single-worker one. The global shape preserved psycopg2
  thread-affinity but serialized every advisory-lock acquire on the node
  behind one thread, even for unrelated (user, server) keys.
- get_user_access_token_classified flips to `async with lock, pg_lock:` so
  concurrent same-key callers serialize on the in-process asyncio.Lock
  before allocating the pg_lock's per-instance executor + spin loop. N
  concurrent same-key callers collapse to one executor allocation.
- _drain_orphan_pg_lock no longer re-awaits the cancelled asyncio Future
  from `__aenter__`. It receives the underlying concurrent.futures.Future
  and re-wraps it via asyncio.wrap_future, getting an independent asyncio
  Future tied to the worker outcome. This way cancellation of the awaiter
  doesn't poison the drain's wait, and the drain genuinely waits for the
  worker to settle before deciding whether to call cm.__exit__.
- Module-level _pg_refresh_drain_tasks set holds strong refs to in-flight
  drains (asyncio's task set is weak — fire-and-forget tasks could be GC'd
  mid-cleanup; RUF006 hazard).
- Drain narrows except clauses to Exception so a drain-task cancellation
  records as cancelled instead of being silently logged as 'completed
  normally with no acquire'.

Test integrity (was a major finding in round 2 — old generator-based cm
let the test pass via GC finalization timing rather than drain logic):

- New _ObservableLockCm class-based context manager whose __exit__ is a real
  observable method (records call args + thread). Distinguishable from
  GeneratorExit thrown by GC of a generator-based cm.
- Strong external ref to the cm via created_cms list — keeps cm alive past
  the test's awaits, so a no-op drain genuinely fails the assertion rather
  than papering over via GC timing.
- Deterministic drain wait via _pg_refresh_drain_tasks gather — no
  fixed-duration sleeps.
- _run_cancel_scenario helper drops the duplicated setup between the two
  cancellation tests.

Negative-test verified: replacing _drain_orphan_pg_lock body with `return`
makes test_pg_refresh_lock_cancellation_releases_on_same_thread fail with
'drain did NOT call cm.__exit__ — orphan Postgres lock + open transaction'.

Other fixes: protocol docstring corrected to describe pg_try_advisory_xact_lock
spin + retry (was claiming pg_advisory_xact_lock blocking acquire);
get_user_access_token_classified docstring rewritten for new lock order;
narrow `except BaseException` -> `except Exception` in
test_mcp_user_pool.py concurrent-dispatch helper.

882 tests pass (MCP + auth + storage). ruff + mypy clean.

(cherry picked from commit 3eb9d22ad5)
2026-05-07 17:35:21 -07:00
Patrick Buckley 1a1043c4df feat(mcp): per-(user, server) ClientSession pool with OAuth dispatch
Phase 5 of OAuth-MCP — adds a per-(user, MCP-server) ClientSession
pool to MCPClientManager alongside the existing static-server path,
gated entirely on the per-server `auth_type='oauth_user'` config.

Pool architecture:
- `_user_pool_entries: dict[(user_id, server_name), PoolEntryState]`
  with lazy connect on first dispatch, per-key asyncio.Lock allocated
  on the mcp-loop, idle eviction coroutine (default 600s TTL, LRU cap
  200), and an `in_flight` counter as the eviction interlock so live
  calls can never be torn down mid-flight.
- `_dispatch_pool` runs the token-state machine: missing token →
  `mcp_consent_required`; key-rotation decrypt failure →
  `mcp_token_undecryptable_key_unknown` with NO consent prompt and NO
  auto-delete; expired token → silent refresh under per-(user, server)
  advisory lock; refresh failure → revoke + consent.
- `_classify_failure` separates transport (trips breaker) from auth
  401/403 (does NOT trip breaker — server-only invariant) from
  protocol (no breaker change).
- `entry.open_lock` held only across connect-or-reuse and released
  before the `await session.call_tool` so concurrent calls from one
  user against one server overlap (validated by Spike 1 scenario 2).

Auth-class failures are fail-soft in Phase 5: any 401/403 surfaced by
the SDK propagates to the agent as a tool error and the next dispatch
reconnects on a fresh refresh. Real introspection of upstream 401/403
is a Phase 6 concern — the MCP SDK's `streamable_http` post_writer
swallows `httpx.HTTPStatusError` upstream, so detecting status from
the response chain requires `McpError(CONNECTION_CLOSED)` payload
parsing or a custom httpx middleware around `streamablehttp_client`.
The mid-flight 401 refresh-retry path and the `mcp_insufficient_scope`
structured error for 403 step-up land together in Phase 6, gated by
an integration test that drives a real upstream 401/403 (the unit-
test injection of `HTTPStatusError` is what masked the production gap
on the first apply-findings pass — the integration test is the
structural gate so the gap can't reopen). RFC §1.5 steps 4-5 and the
phase table in §Implementation phases reflect this scope split.

Multi-node refresh contention:
- New `StorageBackend.acquire_advisory_lock_sync` Protocol method.
  SQLite returns nullcontext (single-node, in-process asyncio.Lock
  is sufficient). Postgres uses `pg_try_advisory_xact_lock` with
  retry on a fresh per-attempt connection, so waiters don't pin pool
  connections during the AS roundtrip. Inner try/except + nested
  finally ensures conn is always returned to the pool, even when
  begin / execute / yield / commit raises mid-body.
- Lock ordering: pg_advisory outer, asyncio.Lock inner. Re-read after
  lock collapses cluster-wide contention to one HTTP roundtrip per
  (user, server) per refresh window.
- `_PgRefreshLock` enter/exit pinned to a single-worker
  ThreadPoolExecutor so SQLAlchemy connection state stays
  thread-affine across cancellations.

Token storage refactor:
- `get_user_access_token_classified` returns a tagged TokenLookupResult
  (Token / MissingToken / DecryptFailure / RefreshFailed) so the
  dispatcher maps each state to the right user-facing error.
- `get_user_access_token` is now a thin wrapper around the classified
  variant; the previous duplicated state machine is gone.

Security:
- Pool dispatch + admin endpoints reject `http://` URLs for
  `auth_type='oauth_user'` servers (only exact loopback hostnames are
  exempt — `*.localhost` is intentionally NOT honored because RFC 6761
  localhost-zone resolution is configuration-dependent and could route
  bearers to non-loopback IPs via custom resolvers / hosts file /
  Docker overlays). Validated at three layers:
  `_dispatch_pool` (structured `mcp_oauth_url_insecure` error),
  `_connect_one_pool` (defensive ValueError), and
  `admin_create_mcp_server` / `admin_update_mcp_server` (400 before
  storage write).
- Admin URL change on an oauth_user row purges per-user OAuth tokens
  bound to the old URL: bearers are bound (via OAuth resource /
  audience) to the URL active at consent time, so silently rebinding
  them to a new URL is a token-binding violation. Re-consent forces
  fresh issuance for the new resource.
- Encryption-key fingerprints stay in audit logs only; no longer
  surfaced in agent-facing error payloads.

User_id thread-through:
- `MCPClientManager.call_tool_sync(..., user_id=None)` (additive;
  default None preserves the static path byte-identically).
- `ChatSession._exec_mcp_tool` passes `self._user_id or None`.
- `set_app_state(app_state)` setter wires OAuth state at lifespan
  startup, called from both turnstone-server and turnstone-console.

Performance:
- LRU cap eviction iterates `_user_pool_entries` (not
  `_user_pool_last_used`) so pre-dispatch entries are eligible.
- Eviction batch closes via `asyncio.gather` instead of serial await.
- `_resolve_pool_target` returns the resolved server row to
  `_dispatch_pool` to eliminate the second DB lookup.
- Production reachability of pool dispatch is gated on Phase 7
  (catalog scoping) wiring pool tools into `_tool_map`; until then
  pool dispatch is reachable only via direct `call_tool_sync` with a
  prefixed name (the path the new pool tests exercise).

Hardening parity preserved:
- Static path (auth_type ∈ {none, static}) byte-identical; PR #296
  hardening (SDK #2147 mitigations, anyio cancel-scope, stale-session-
  and-stack guard, server-only circuit breaker) intact.
- `test_reconnect_preserves_static_state_identity` unchanged + green.
- `MCPTokenStore.get_user_token` does not auto-delete on
  MCPTokenDecryptError (key-rotation safety).
- Notification debounce stays manager-level.
- Connect-failure cleanup factored into
  `_safe_teardown_on_connect_failure` shared by both connect paths.

Tests: 5475 → 5493 (+18). New file `tests/test_mcp_user_pool.py`
plus additions to test_mcp_oauth_refresh.py, test_mcp_admin_api.py,
and test_mcp_client.py covering: pool data structures, lazy connect,
eviction TTL + LRU + lock interlock, dispatch state machine (token
states), failure classification, http-rejection at dispatch and
admin layers, URL-change-purges-tokens (sec), concurrent dispatch on
one (user, server), pg_advisory lock parity, and user_id threading.

Phase exit criterion (synthetic load test 50 users × 3 servers × LRU
30 × 1000 calls × 200 evictions) deferred to a post-Phase-5 fitness
spike that runs against a staging deployment with real FDs and real
network behaviour, not a CI mock — same shape as Spike 1's
pre-Phase-0 SDK validation.

Out-of-scope for Phase 5 (Phase 6+): SDK-level 401 refresh-retry +
403 `mcp_insufficient_scope` (Phase 6), per-user catalog scoping
(Phase 7), consent UX SSE event + dashboard renderer (Phase 8),
admin UI status indicators (Phase 9).

(cherry picked from commit 4db7d9c6cf)
2026-05-07 17:35:21 -07:00
Patrick Buckley 55aab54774 test(mcp): SDK 1.27 concurrency spike for per-(user, server) pool
Spike artifact validating MCP SDK behavior before Phase 5 builds the
per-(user, MCP-server) ClientSession pool. Three scenarios, all pass:

1. N=20 concurrent ClientSession instances against the same URL — no
   FD blow-up, no shared transport state, each session's tools/list
   returns independently.

2. Two concurrent tools/call on a shared ClientSession with
   interleaving payloads — request_id demux works under contention.

3. Per-session Authorization header isolation across 5 sessions —
   httpx connection pooling does not cross headers between sessions,
   so per-session bearer tokens reach the server unmixed.

Outcome gates the Phase 5 architecture (lazy dict[(user_id,
server_name), ClientSession] + per-key asyncio.Lock + LRU eviction).
Had any scenario failed, the fallback was per-call header injection
(Alternative F in the OAuth-MCP RFC).

Spike-only — not collected by pytest. Run manually:

  uv run python tests/spike_sdk_concurrency.py

(cherry picked from commit e695a98c54)
2026-05-07 17:35:20 -07:00
Patrick Buckley 0f8c8b38a3 fix(mcp): pin OAuth return_url + sanitise read-scope status
Addresses ten findings on the Phase 4 OAuth-MCP commit: four from the
PR #478 review surface, plus six surfaced by a follow-up multi-stage
review of the first round of fixes. Two of the latter were genuine
security regressions in the very code that claimed to close those
holes.

Security
--------

- _validate_return_url now pins return_url same-origin against the
  configured oidc_config.redirect_base instead of request.url. Behind
  a permissive front proxy that did not normalise Host, an attacker
  could spoof Host and provide a matching absolute return_url to mint
  an open redirect off /api/mcp/oauth/start. Same fix pattern as
  PR #476 OIDC.
- Reject return_url values containing literal backslashes or starting
  with `//` up front. urlparse leaves backslashes inside `path`, so a
  value like `/\evil.example/foo` slipped through the path-only branch
  and became the protocol-relative `//evil.example/foo` after WHATWG-
  conformant browsers normalised the backslash — re-introducing the
  open redirect the same-origin pin was meant to close.
- internal_mcp_status (read-scoped) projects through a new
  _strip_server_status_for_read helper that drops the verbose `error`
  text and replaces it with a coarse `has_error` boolean. The error
  string is built as `f"{type(exc).__name__}: {exc}"` and so carries
  stdio binary paths (FileNotFoundError) or internal MCP URLs
  (httpx.ConnectError) — equivalent to leaking command/url, which
  this same patch deliberately strips. Approve-scoped refresh and
  reconnect callers continue to receive the full `error` text via
  the existing _strip_server_status helper.
- internal_mcp_status now returns the projected (sanitised) entries
  for every server in mcp_mgr.get_all_server_status() instead of
  emitting the un-sanitised dict that included `command` (stdio argv)
  and `url` (remote MCP endpoint). Sibling refresh/reconnect endpoints
  already used _public_server_status to strip these.
- internal_mcp_status docstring documents the trust boundary — server
  enumeration to read scope is intentional so dashboards can render
  per-server indicators; verbose error detail and command/url remain
  approve-scoped.

Correctness / UX
----------------

- _validate_return_url comparison normalises (scheme, host, port)
  before equality. Lowercases hostname and collapses the scheme's
  default port, so `https://App.Example.COM/x` and
  `https://app.example.com:443/x` are recognised as same-origin
  with `redirect_base = https://app.example.com` instead of being
  silently downgraded to the `/` fallback.
- mcp_crypto startup-gate error message now names both
  `mcp_token_encryption_keys` (rotation list) and
  `mcp_token_encryption_key` (single) so an operator using rotation
  isn't misled into thinking only the singular form is valid.

Cleanup
-------

- Delete the unused _KNOWN_TRUSTED_ENDPOINT_HOSTS legacy re-export
  shim in oidc.py (zero callers — a no-op that survived the Phase 4
  oauth_ssrf extraction). Sphinx :data: docstring reference at
  validate_discovered_endpoint updated to point at
  turnstone.core.oauth_ssrf.KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS
  directly. The Google multi-origin allowlist is unaffected — it
  lives at the canonical name and is read from oauth_ssrf.py:164.
- test_mcp_oauth_handlers TestValidateReturnUrl imports
  _validate_return_url at module level instead of repeating the
  import inside each test method.
- test_server_lifespan_mcp_crypto replaces a fragile
  `messages.count("mcp_token_encryption_key") >= 2` substring trick
  with `re.search(r"mcp_token_encryption_key(?!s)", messages)` —
  asserts the singular form directly via negative lookahead.

Tests
-----

5448 pass (+13 vs the prior tip):

- TestValidateReturnUrl gains backslash-bypass, protocol-relative,
  default-port, uppercase-host, and explicit-port-mismatch cases
  alongside the original same-origin / cross-origin / scheme-
  mismatch / path-only cases.
- TestInternalMcpStatusEndpoint asserts the `error` text never
  reaches the read-scope wire (binary-path FileNotFoundError no
  longer appears anywhere in the rendered response) and that the
  coarse `has_error` boolean lights up correctly on the failed
  server.
- TestInternalMcpStatusEndpoint also pins the no-mcp-client path to
  `{"servers": {}}`.
- _routes_with_internal extended to include the
  /api/_internal/mcp-status route so the new tests can exercise it
  through TestClient.
- Existing test_startup_aborts_with_oauth_user_row_and_no_key
  strengthened to require both singular and plural key names appear
  in the error log.

(cherry picked from commit 62bbc332af)
2026-05-07 17:35:20 -07:00
Patrick Buckley b0f7029ff1 feat(mcp): per-(user, server) OAuth 2.1 + PKCE flow
Lands the OAuth flow that uses the token-at-rest store from the prior
commit: discovery (RFC 9728 PRM + RFC 8414 AS metadata with operator-
override precedence), PKCE S256 (mandatory — refuse AS without it),
RFC 8707 resource indicator on every authorize and token request,
RFC 7591 minimal one-shot dynamic client registration, authorization-
code exchange, refresh-token grant with re-read-after-acquire single-
flight lock, and the /v1/api/mcp/oauth/{start,callback} endpoints
mounted on both server and console.

Refactored:
- validate_url_no_ssrf, validate_discovered_endpoint, is_localhost,
  effective_port, sanitize_log_text moved out of oidc.py into a shared
  oauth_ssrf module; oidc.py re-exports for compatibility. The shared
  helpers also expose async wrappers (validate_url_no_ssrf_async,
  validate_discovered_endpoint_async) so OAuth-MCP discovery — invoked
  from async handlers — does not block the event loop on the
  synchronous socket.getaddrinfo call.
- MCPTokenStore.get_oauth_client_secret reader path added (the prior
  commit was write-only)
- Storage protocol gains create/pop/cleanup_*_mcp_oauth_pending_state
  and get_mcp_oauth_client_secret_ct (mirror OIDC pending-state
  pattern: SQLite BEGIN IMMEDIATE select-then-delete, Postgres atomic
  DELETE...RETURNING)

Refresh-grant correctness:
- When the AS omits refresh_token (RFC 6749 §6 — MAY rotate), the
  existing refresh value is preserved at the OAuth-flow layer rather
  than cleared, so production ASes (Google, Auth0 default, Okta) don't
  force re-consent every hour
- expires_in accepts int, float, str-with-decimal — earlier int-coerce
  through str() failed on float and silently dropped expiry tracking
- The refresh-grant `resource=` parameter (RFC 8707) is the canonical
  MCP server URL, not the audience. Audience and resource are distinct
  concepts; using audience as resource would mismatch the AS RS
  allowlist.

Audience handling:
- _validate_token_audience accepts str or tuple; the callback resolves
  accepted_audiences = {server_url, oauth_audience} and validates
  against the set, so Auth0-style ASes that honor `audience=` (not
  RFC 8707 `resource=`) issue tokens that pass audience-bound
  validation
- build_authorize_url emits both `resource=` (RFC 8707) and
  `audience=` (Auth0-style) per server config; comment documents which
  AS implementations need which form

Security hardening:
- redirect_uri pinned to oidc_config.redirect_base instead of the
  request Host header — closes the same Host-header injection PR #476
  fixed for OIDC. Both /start and /callback return 503 with operator-
  actionable hint when redirect_base is unset
- DCR registration runs under per-server asyncio.Lock with re-fetch
  inside the lock, so concurrent /start callers don't both register
  and overwrite each other's client_id (the second user's code is no
  longer rejected on callback)
- /callback error branch pops the pending state row before redirecting
  so a leaked state can't be replayed against a separately-obtained
  code in the 60s cleanup window
- WWW-Authenticate Bearer parser handles RFC 7235 quoted-string
  escapes (\" and \\) instead of the naive [^"]+ regex
- AS-controlled response bodies and error_description query params go
  through sanitize_log_text before reaching exception messages or
  audit details. AS error responses are parsed for the standard
  RFC 6749 fields (error, error_description, error_uri), each
  capped at 80 chars and run through redact_credentials to defend
  against ASes that echo the request body back into their error
  payload.
- oauth_as_issuer_cached is re-validated against the SSRF guard on
  read; on rejection the column is cleared and PRM rediscovery runs
- DCR / token-endpoint / refresh-endpoint response bodies cap at 64
  KiB (PRM/AS metadata cap stays at 256 KiB) so a hostile or
  malfunctioning AS can't exhaust client memory.
- oauth_client_secret operator input capped at 1024 chars at the
  admin-form boundary; longer plaintext rejected with 400.
- /start and /callback responses stamp `X-Frame-Options: DENY` so the
  redirected pages can't be framed by attacker sites.
- delete_user cascades to mcp_user_tokens and mcp_oauth_pending so
  user deletion no longer leaves dangling per-user OAuth state.
- Renaming or deleting an oauth_user MCP server purges per-user
  tokens and pending OAuth state for the previous server name
  (delete_mcp_oauth_rows_by_server_name). The OAuth tables key on the
  mutable server_name; without this purge, a future server with the
  same name (and an attacker-controlled URL) would silently rebind
  prior user tokens. A future schema migration will replace the
  server_name key with a server_id FK + ON DELETE CASCADE.
- get_user_access_token catches MCPTokenDecryptError (raised when no
  installed key can decrypt the row, e.g. after key rotation) and
  falls through to None so dispatch surfaces a re-consent rather than
  crashing.
- oauth_user MCP server rows are skipped in the static auto-connect
  path. Auto-connecting them at startup with empty headers fails the
  AS check and trips the circuit breaker; per-user tokens come online
  lazily once the user has consented.

Audit (mcp_server.oauth.* prefix):
- consent_started, consent_completed, consent_failed, token_refreshed,
  token_revoked, dcr_registered. _audit_event is async and wraps
  record_audit in asyncio.to_thread so the audit write doesn't block
  the event loop. resource_id on the audit row is the immutable
  server_id (PK UUID) so admin-driven server renames don't break
  event correlation; server_name is exposed in detail for cross-
  reference. dcr_registered detail.has_secret reflects whether the
  DCR-issued secret was actually persisted (the prior code reported
  has_secret=true even on persistence failure).
- _admin_mcp_action audits the immutable server_id, not the mutable
  server_name (which is what the column is — the table's PK was
  always server_id).
- All OAuth-flow log keys use the mcp_server.oauth.* prefix to match
  the audit-action taxonomy.

Lifespan close-order in turnstone.server and turnstone.console.server
is reversed (LIFO) — mcp_oauth → mcp_crypto → oidc — to match init
order.

Deferred until the upcoming per-user pool integration:
- Multi-node refresh-lock contention via pg_advisory_lock
- DCR re-register on token-endpoint 401 (the dispatch path surfaces
  those 401s)
- TTL-LRU caching of decrypted plaintext access tokens
- DNS-rebinding hardening (httpx Transport pin) — documented as
  limitation in oauth_ssrf module docstring

Tests: 7 new test files / ~85 new tests covering discovery precedence
+ PRM quoted-string parsing, PKCE round-trip, SSRF helper extraction,
authorize/callback handlers including 503-on-no-redirect-base + DCR
concurrency + JWT audience polymorphism + callback-error-pops-pending,
refresh single-flight lock, refresh resource-vs-audience regression,
decrypt-error fallthrough, _db_servers_to_config skipping oauth_user,
pending-state CRUD round-trip.

(cherry picked from commit 29c42c1427)
2026-05-07 17:35:20 -07:00
Patrick Buckley a4c335d7bf feat(mcp): token-at-rest encryption layer for OAuth-MCP
Phase 3 of docs/design/oauth-mcp.md. Adds the Fernet/MultiFernet wrapper,
[security] config loader with rotation support, MCPTokenStore CRUD facade,
typed MCPTokenDecryptError that maps to the RFC's mcp_token_undecryptable_
key_unknown class, and a startup gate that fails loud when auth_type=
'oauth_user' rows exist without a configured encryption key.

Crypto module (turnstone/core/mcp_crypto.py):
- MCPTokenCipher wraps cryptography.fernet.Fernet + MultiFernet for
  rotation; encrypt with first key, decrypt by trying each in order
- load_mcp_token_cipher_config reads [security] mcp_token_encryption_keys
  (plural list) or mcp_token_encryption_key (singular), validates each
  key is base64-decodable to exactly 32 bytes
- MCPTokenCipherConfig is repr=False with custom __repr__ that redacts
  raw key bytes (defense in depth against accidental log/traceback leak)
- _key_fingerprint produces an 8-hex-char SHA-256 prefix for audit
  attribution without exposing the key
- MCPTokenStore handles encrypt-on-write / decrypt-on-read for
  mcp_user_tokens and mcp_servers.oauth_client_secret_ct
- get_user_token MUST NOT auto-delete the row on MCPTokenDecryptError
  (test_get_user_token_with_wrong_key_raises_decrypt_error verifies
  the row stays intact across a key-mismatch read)
- initialize_mcp_crypto_state / close_mcp_crypto_state lifespan helpers
  shared between server and console

Storage protocol (5 new ciphertext-only methods):
- set_mcp_oauth_client_secret_ct (dedicated writer; deliberately NOT
  added to MCP_SERVER_MUTABLE so generic update_mcp_server cannot write
  the secret column)
- create_mcp_user_token, get_mcp_user_token,
  update_mcp_user_token_after_refresh, delete_mcp_user_token

Server + console lifespans (turnstone/server.py + console/server.py):
- after OIDC init, count auth_type='oauth_user' rows; if any exist and
  no encryption key is configured, log an actionable error and
  raise SystemExit(1)
- without oauth_user rows, missing key is fine (lazy validation; admin
  flip without restart returns 503 from the admin handler)
- app.state.mcp_token_cipher / .mcp_token_store populated when key
  configured; None otherwise

Admin handlers:
- _require_token_store_for_oauth_secret pre-mutation gate validates
  token_store availability and oauth_client_secret type BEFORE
  storage.create_mcp_server / update_mcp_server runs, so a 503 from a
  missing key never leaves an orphan row or partial-update state
- _apply_oauth_client_secret encapsulates the encrypt + audit write
  used after the storage mutation; rolled out across both create and
  update handlers
- 503 message references both mcp_token_encryption_key (singular) and
  mcp_token_encryption_keys (plural for rotation)
- non-string oauth_client_secret payloads (false / 0 / lists / dicts)
  are rejected with 400 instead of being str()-coerced
- when auth_type transitions away from oauth_user, the encrypted
  secret column is cleared in the same admin call (with audit), so
  flipping back doesn't silently resurrect a stale credential

Audit events (mcp_server.oauth.* per audit.py taxonomy; RFC's
mcp.oauth.* renamed for consistency):
- mcp_server.oauth.client_secret_set fired from admin handlers with
  cleared:bool and key_fingerprint
- mcp_server.oauth.token_decrypt_failure fired from MCPTokenStore
  .get_user_token when no installed key can decrypt; carries
  key_fingerprints_attempted

Tests: 35 new tests across test_mcp_crypto, test_mcp_token_store,
test_server_lifespan_mcp_crypto, plus 6 admin-API tests covering the
no-orphan-row, no-partial-update, secret-clear-on-transition, and
non-string-secret-rejection invariants. Suite at 5337 (Phase 3 added
~50 tests including the rebase-imported skill suite).

cryptography>=42 promoted from transitive (lacme[tls]) to direct dep
since the encryption layer is now core, not optional.

Phase 4 (OAuth flow) wires the actual callers; Phase 3 adds only the
crypto layer and is exercised entirely by tests.

(cherry picked from commit 7f132e7230)
2026-05-07 17:35:20 -07:00
Patrick Buckley 21663d1567 feat(mcp): oauth schema + minimum admin form
Adds the data model and admin UI surface required by the OAuth-MCP flow.
Phase 2 of the per-user delegation initiative.

Schema:
- migration 049 creates mcp_user_tokens (PK user_id, server_name) and
  mcp_oauth_pending (PK state, indexed by created_at)
- eight new columns on mcp_servers: auth_type ('none' / 'static' /
  'oauth_user', NOT NULL DEFAULT 'static') plus six oauth_* config
  fields and oauth_as_issuer_cached
- post-upgrade UPDATE normalises auth_type to 'none' for streamable-http
  rows whose headers are NULL/empty/'{}'; stdio rows are left at the
  'static' default (auth_type is HTTP-auth-only)
- _schema.py kept in lockstep with the migration so metadata.create_all
  and alembic upgrade produce identical shapes
- mcp_user_tokens / mcp_oauth_pending TypedDicts in _protocol.py for
  Phase 3/4 use (no CRUD methods yet)

Storage / API:
- create_mcp_server gains the eight kwargs across protocol + sqlite +
  postgresql
- MCP_SERVER_MUTABLE picks up auth_type and the six text oauth_* fields;
  oauth_client_secret_ct is intentionally NOT in the whitelist — Phase 3
  will own ciphertext writes via a dedicated method
- McpServerInfo + Create/Update Pydantic schemas extended; oauth_client_secret
  accepted as plaintext input but discarded (Phase 3 wires encryption)

Admin handlers:
- _parse_auth_type validates against {'none', 'static', 'oauth_user'} and
  rejects empty / unknown values; shared between create and update
- when auth_type changes away from 'oauth_user', the oauth_* config
  columns are explicitly nulled in the same UPDATE so the row stays
  consistent
- _clean_oauth_text caps text fields at 512 chars (URLs at 2048) to bound
  admin write surface
- _mask_mcp_secrets now masks oauth_client_secret_ct to '***' regardless
  of reveal=true (write-only field)
- audit detail dict redacts oauth_client_secret if present

Frontend:
- new "Multitenant Authorization" fieldset on the MCP-server modal with
  three radio buttons (None / Shared / Per-user OAuth 2.1)
- conditional OAuth subform: AS URL, registration mode (preregistered /
  dcr; cimd is future), client ID, client secret, scopes, audience
- secret input is autocomplete=off and never round-trips on edit
- audience auto-populates from the MCP server URL on blur
- headers textarea hidden and submitted as {} when auth_type is 'none' or
  'oauth_user' so flipping the radio cleans up server-side state

Tests: storage round-trip for the new columns, oauth_pending table smoke,
migration 049 upgrade/downgrade with stdio-vs-http normalisation, four
admin-API tests for auth_type validation and oauth_*-clear-on-flip-away.
Suite passes 5284 (matched pre-Phase-2 baseline 5267 + 17 new).

Stacks on Phase 0; no behavioural change for existing rows.

(cherry picked from commit d675b237a3)
2026-05-07 17:35:20 -07:00
Patrick Buckley c823156af5 refactor(mcp): consolidate per-server state into StaticServerState dataclass
Phase 0 of the OAuth-MCP RFC: prepare MCPClientManager for the per-(user,
server) session pool that lands in Phase 5, without changing static-path
behavior.

Two changes:

1. Hardening helpers _pre_close_streams and _tcp_probe rename their first
   parameter from `name` to `key`.  Type stays `str` for now; widening to
   `str | tuple[str, str]` happens in Phase 5 when callers actually pass
   tuples.  _safe_close_stack takes the stack directly and is unchanged.

2. The eleven parallel name-keyed dicts (_sessions, _per_server_stacks,
   _per_server_tools, _per_server_resources, _per_server_prompts,
   _supports_list_changed, _supports_resources, _supports_resource_list_changed,
   _supports_prompts, _supports_prompt_list_changed, _server_streams) are
   consolidated into _static_servers: dict[str, StaticServerState].  Server-
   level state (circuit breaker, notification debounce, last-error,
   db-managed, merged catalog maps, listener lists) stays on the manager,
   unchanged.

PoolEntryState is defined for Phase 5 use but no code instantiates it.  The
typed map declarations (dict[str, StaticServerState] vs dict[tuple[str, str],
PoolEntryState]) make accidental cross-keying lookups easier to catch.

PR #296 hardening preserved exactly:
- pre-close-streams atomic take-and-clear before stack teardown
- stale-session-and-stack guard at _connect_one top: both state.session and
  state.stack checked, cleared independently, entry preserved (not popped)
- transport-error session-eviction in dispatch sets state.session=None only,
  leaving stack/streams for the next connect-time guard sweep
- _safe_close_stack CancelledError suppression unchanged
- TCP probe before streamablehttp_client unchanged
- future.cancel() after TimeoutError in all sync bridges unchanged
- notification debounce stays manager-level (not migrated into the dataclass)

Refresh helpers (_refresh_server_tools/_resources/_prompts) snapshot
state.session into a local immediately after the None guard so concurrent
transport-error eviction during await cannot null the session reference
mid-call.

Tests: shared _seed_static_state helper in tests/conftest.py replaces eleven
direct dict mutations; new test_reconnect_preserves_static_state_identity
guards the entry-preservation invariant.  Pass count rises 5266 → 5267.

(cherry picked from commit be0950bb98)
2026-05-07 17:35:20 -07:00
Patrick Buckley bace928477 refactor(mcp): remove periodic refresh, add manual refresh/reconnect controls
Deletes the _periodic_refresh task and its supporting state
(_refresh_task, _refresh_failures, _refresh_backoff_until,
_REFRESH_BACKOFF_BASE/MAX, _DEFAULT_REFRESH_INTERVAL, refresh_interval
kwarg) from MCPClientManager. Push notifications and operator-driven
manual refresh now cover all catalog-update needs; the long-running
4-hour timer was dead complexity that obscured the per-user pool
work to come.

Catalog freshness on auto-reconnect is preserved by scheduling an
unblocking _refresh_server task on the mcp-loop after _connect_one
succeeds; the calling thread returns immediately so half-open
recovery latency does not double. Adds MCPClientManager.reconnect_sync
(clears the circuit, closes any existing session, calls _connect_one,
clears stale catalog on failure).

Wires a new pair of operator endpoints —
POST /v1/api/admin/mcp-servers/{name}/refresh and
/v1/api/admin/mcp-servers/{name}/reconnect — that fan out to all
nodes through the existing _internal route family, with per-row
"Refresh" and "Reconnect" buttons in the MCP Servers admin tab.
The new node-internal paths /api/_internal/mcp-{refresh,reconnect}/
are gated to the approve scope to prevent direct unprivileged
reconnects bypassing the console's admin.mcp gate. Internal
endpoints return generic error messages and a filtered status
payload (no command/url) to keep transport details admin-gated.

Drops the [mcp] refresh_interval setting, the
--mcp-refresh-interval CLI flag, and the matching config-mapping
entry; updates docs/architecture.md, docs/tools.md,
docs/settings.md, and the three PlantUML diagrams that referenced
the periodic loop.

Tradeoffs (intentional):
- Idle nodes will not auto-rejoin a recovered MCP server until
  traffic arrives or an operator clicks Reconnect. The previous
  background reconnection loop is gone by design — push
  notifications + operator controls replace it.
- Console fan-out blocks on the slowest node (existing pattern);
  not changed here.

This is Phase 1 of the OAuth-MCP series — feature subtraction
ahead of per-user state.

(cherry picked from commit eb2a119da9)
2026-05-07 17:35:20 -07:00
Patrick Buckley d16c911750 feat(skills): paste SKILL.md to auto-fill the Create Skill modal (#477)
* feat(skills): paste SKILL.md to auto-fill the Create Skill modal

When a user pastes an Anthropic-style SKILL.md (YAML frontmatter +
markdown body) into the Create Skill content textarea, the frontend
sniffs the leading ``---``, posts the raw text to a new backend parse
endpoint, and populates name / description / tags / author / version /
license / compatibility / allowed_tools from the parsed fields.  The
textarea is left with the body only (frontmatter stripped), and a toast
reports how many fields were set vs. kept (already-typed values are
preserved).

Backend
- ``POST /v1/api/admin/skills/parse`` (admin.skills permission) wraps
  the existing ``turnstone.core.skill_parser.parse_skill_md`` so admin
  imports and external installs share one parser.  ``ParseSkillRequest``
  / ``ParseSkillResponse`` schemas added; OpenAPI spec + sync/async
  console SDK methods updated.
- Hardening: 32 KiB cap on ``raw`` (Pydantic ``max_length`` + handler
  enforcement); ``Content-Length`` pre-check returns 413 before any body
  buffering; parse offloaded via ``asyncio.to_thread`` so deeply-nested
  YAML cannot stall the event loop.

Frontend (turnstone/console/static)
- New paste handler with optimistic paint (raw text shown immediately,
  textarea disabled + ``aria-busy`` flipped, hint switches to
  "Parsing...") so the round-trip is visible on slow networks.
- ``AbortController`` + generation guard (``_ctmPasteController``) so a
  fresh paste or modal close cancels a stale fetch — the previous
  handler's callbacks see the controller has been replaced and bail
  before touching the DOM.
- Non-destructive overwrite: ``_setSkillFormField`` returns "filled" /
  "skipped" / "absent" and refuses to clobber non-empty values.  Toast
  reports counts.
- Bumps ``#toast`` z-index above modal overlays (was 200 vs. modal 600
  — toasts fired while a modal was open were invisible).  Console-wide
  fix exposed by this being the first feature to fire toasts mid-modal.

HTML / CSS
- New ``.skill-paste-hint`` line above the textarea announcing the
  affordance, sized to match surrounding ``.label-hint`` text.
- ``aria-describedby`` ties the hint to the textarea; ``aria-live=
  "polite"`` announces the busy-state transition to screen readers.
- "Skill Content" heading hint reworded "system message — ..." →
  "available: ..." and the variables row label "Variables" → "Used"
  to disambiguate available vs. in-use template variables.

Tests
- 11 new cases in ``tests/test_skill_parse_api.py``: happy paths
  (full / minimal / nested-metadata / unquoted-colon recovery),
  malformed YAML 400, missing/blank/missing-name 400, RBAC 403, raw
  body 32 KiB cap (Content-Length pre-check), chunked-encoding bypass
  forces the application-layer cap.  Test pins ``raw_frontmatter``
  omission so a future ``dataclasses.asdict`` refactor can't silently
  leak the full YAML dict back to clients.

Validation
- 5146 / 5146 ``pytest -k "not live"`` pass.
- ``ruff`` + ``mypy`` clean on changed sources.
- ``node -c`` clean on governance.js.
- Two-stage code review (full pipeline + bug+quality re-review of the
  fix patches) applied; all confirmed findings addressed.

* fix(skills): Copilot PR #477 review fixes (cumulative bug-1, bug-2, q-1)

bug-1 (server.py): Content-Length pre-check was clamped to 32 KiB —
the same number as the per-string char cap on ``raw``.  A legitimate
``raw`` of exactly 32 KiB produces a JSON body well above 32 KiB once
the ``{"raw":"..."}`` wrapper and any escaping is added, so valid
near-max requests were 413'd.  New constant
``_PARSE_SKILL_MAX_BODY_BYTES = _PARSE_SKILL_MAX_CHARS * 4`` admits the
wrapper + multibyte expansion while still refusing obviously oversized
payloads early; the per-string ``len(raw)`` check stays authoritative.

bug-2 (governance.js): hideCreateTemplateModal aborted the inflight
paste controller and nulled the global, but the handler's ``.catch``
and ``.finally`` guard each DOM mutation behind ``_isCurrent()`` —
both bail when the controller has been nulled, leaving the textarea
``disabled`` + ``aria-busy`` and the hint stuck on "Parsing…".
Reopening the modal landed on a poisoned state.  The second-pass
review's q-2 cleanup that dropped the show-side defensive reset
missed this scenario — the verifier's reachability argument confused
"controller is null" with "UI state is reset"; the two are
independent.  Hide now resets the paste-induced visible state
alongside the abort.

q-1 (console_spec.py): error_codes for the parse endpoint listed only
400; handler also returns 413 for oversized bodies.  Added 413; kept
403 implicit per the convention sibling admin endpoints follow.

Test fixup: bumped the Content-Length test payload to 200 KB so it
clearly exceeds the new 128 KB pre-check threshold; otherwise it was
falling through to the per-string check and duplicating
test_oversized_raw_chunked_returns_413's coverage.

(cherry picked from commit 0a8083e6d5)
2026-05-07 17:35:20 -07:00
Patrick Buckley b8fadad94f fix(oidc): close transient client on disable paths + correct docstring
PR #476 review feedback (Copilot, oidc.py:584,616):

1. initialize_oidc_state's docstring claimed "on any failure
   enabled is False" but the JWKS-prefetch failure branch
   intentionally keeps enabled=True so the callback's lazy-fetch
   retry can recover from a transient IdP issue at startup.
   Docstring rewritten to spell out the three post-conditions:
   disable, JWKS-failure-keeps-enabled, success.

2. The long-lived httpx.AsyncClient was created up front, then
   three disable branches (discovery exception, discovery-returned-
   disabled, missing redirect_base) returned without closing it,
   leaving sockets held until shutdown.

   Restructured: discovery now uses a transient AsyncClient inside
   a context manager (closed at exit). The long-lived client is
   only created after the disable checks pass. The JWKS-failure
   branch still legitimately keeps the client open because the
   lazy-retry path needs it.

   The pre-existing single-client-passthrough test was replaced
   with three more specific tests: long-lived client only goes to
   fetch_jwks (not discover_oidc); discovery-exception path leaves
   http_client=None; missing-redirect_base path leaves
   http_client=None.

(cherry picked from commit b2153d907f)
2026-05-07 17:35:20 -07:00
Patrick Buckley 63aecdf2fa chore(oidc): consolidate test OIDCConfig helper + fix exceptions banner (cumulative q-4, q-5)
q-4: tests/test_oidc.py's _make_config and tests/test_oidc_handlers.py's
_make_oidc_config built the same OIDCConfig with sensible defaults but
had drifted — only the handlers helper set redirect_base. After b3
made redirect_base operationally required, every test_oidc.py test
that exercised redirect_base had to override it explicitly. A future
test could omit redirect_base and silently exercise the wrong
production path.

Moves make_oidc_test_config to tests/conftest.py with the more
complete handler-version defaults (including redirect_base). Both
test files import it under their existing local alias
(_make_config / _make_oidc_config) so the 60+ call sites in
test_oidc.py and the handler tests don't have to change.

q-5: section banner '# Exception' (singular) at oidc.py:79 became
inconsistent after b5 (callback robustness) added OIDCKeyNotFoundError.
Renamed to '# Exceptions'.

(cherry picked from commit 5d4a50d2cd)
2026-05-07 17:35:20 -07:00
Patrick Buckley cbe8940b30 perf(auth): migrate handle_auth_status to count_users (cumulative q-3)
The OIDC perf batch added storage.count_users() and migrated the two
OIDC handlers (handle_oidc_authorize, handle_oidc_callback) but missed
handle_auth_status — which still ran storage.list_users() then
len(users) > 0 for the same has-any-users gate.

count_users() is one COUNT(*) round-trip vs list_users() rehydrating
every row dict. Wrapped in asyncio.to_thread to match the OIDC handler
pattern; the async handler no longer blocks the event loop on storage
I/O for what's effectively an existence probe.

(cherry picked from commit 7c6bc22d02)
2026-05-07 17:35:20 -07:00
Patrick Buckley 1dcd1e2ec4 fix(oidc): serialise role-mapping concurrency + skip no-op write lock (cumulative bug-2, perf-1)
bug-2 (Postgres) — replace_oidc_roles read existing rows under default
READ COMMITTED with no row lock. Two concurrent OIDC callbacks for the
same user_id (racing token refreshes with differing claim sets) could
both observe the same baseline and produce a final role state matching
neither caller's intent. Adds .with_for_update() to the SELECT so the
existing rows for this user are locked for the duration of the
transaction.

The lock is per-user_id, not table-wide; unrelated user writes are
unaffected. Empty result sets acquire no locks, so a brand-new user
with no rows yet still allows two callers to proceed and merge via
ON CONFLICT DO NOTHING — that's a permissive race that self-heals on
the next reconciliation cycle, documented in code.

perf-1 (SQLite) — replace_oidc_roles took the SQLite global write
lock unconditionally via BEGIN IMMEDIATE before reading. Steady-state
re-logins (claims unchanged, no INSERT/DELETE needed) paid the lock
cost for nothing and serialised against unrelated writers.

Replaces with a double-check pattern: phase 1 reads under the default
deferred transaction (no write lock), computes the diff, and returns
(set(), set()) on no-op. Phase 2, only when mutation is needed,
commits the read txn, escalates to BEGIN IMMEDIATE, RE-READS, and
re-computes the diff under the lock before writing. The returned
(added, removed) reflects what was actually written, so caller logging
in apply_role_mapping stays truthful even when concurrent writers
shifted state between the two reads.

The OR IGNORE on insert is now defense-in-depth (the lock makes it
unnecessary) but kept as a safety net.

(cherry picked from commit d5087ef3b9)
2026-05-07 17:35:20 -07:00
Patrick Buckley 32e29ff255 docs(oidc): document TRUSTED_ENDPOINT_HOSTS + fix three-vs-four required drift (cumulative q-1, q-2)
The 8-commit OIDC stack added TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS
(operator allow-list for cross-host IdP discovery endpoints) and
promoted TURNSTONE_OIDC_REDIRECT_BASE to required, but the docs drifted
in two places:

q-1 — Troubleshooting > "OIDC not configured" still listed three
required env vars. An operator hitting the missing-redirect-base
startup error landed on a debugging entry that didn't mention the
variable they were missing. Fixed; added a separate troubleshooting
entry naming the exact log message produced by initialize_oidc_state
when redirect_base is unset.

q-2 — TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS was undocumented entirely.
Added a row to the env-var table and a new "Cross-host endpoints"
section explaining when the knob is needed (Google is the canonical
multi-origin IdP, but it's auto-handled; the env var is for any other
IdP whose discovery doc legitimately references hosts beyond the
issuer's origin). Added a troubleshooting entry pointing at the new
section.

(cherry picked from commit 3cf87628d2)
2026-05-07 17:35:20 -07:00
Patrick Buckley 3e2fe0bc9d fix(oidc): self-heal stranded user when role mapping fails post-create (cumulative bug-1)
If apply_role_mapping raised after create_oidc_user committed (transient
storage failure, race with role deletion, etc.), provision_oidc_user's
inline safety-net was skipped — and on retry the existing-identity
branch never reached the safety-net code, leaving the user permanently
stranded with zero roles.

Extracts _ensure_default_role(storage, user_id, desired_role_ids=None)
helper. Calls it on BOTH the new-user and existing-identity paths so a
user stranded by a transient failure recovers on next login.
desired_role_ids is a hint that lets the helper skip list_user_roles
when claim-driven mapping populated at least one role; the new-user
path was already paying that query, the existing-identity path now
pays it only when claim mapping returned an empty desired set.

Documents the admin-strip behavior in the helper docstring: stripping
all roles from an OIDC user no longer locks them out, since the next
login will re-grant builtin-viewer (assigned_by='oidc-default'). The
documented way to deny an OIDC user is to unlink their OIDC identity
via the admin endpoint, not to strip roles. The pre-fix behavior
(stripped user actually locked out) was the bug.

The 'oidc-default' vs 'oidc' assigned_by distinction is preserved:
apply_role_mapping's revocation lane only touches 'oidc' rows, so the
safety-net role survives every subsequent login regardless of claims.

Six new tests cover both paths, the hint short-circuit, the
list_user_roles fallback, the missing-builtin-viewer no-op, and the
self-heal regression case for already-stranded users.

(cherry picked from commit 1c41212f15)
2026-05-07 17:35:20 -07:00
Patrick Buckley 5f5eee4aab test(oidc): close coverage gaps + tighten fetch_jwks shape check (q-5, q-8)
q-5: _derive_username's UUID-retry tier (oidc.py:923-933) was untested.
  After perf-6 collapsed tier-2 to a single find_existing_usernames call,
  the only remaining tail was the 3-attempt UUID-retry loop and the final
  raise. New TestDeriveUsername class covers:
  - falls into UUID retry when all 10 suffix candidates are taken
  - UUID retry succeeds on the second attempt after one collision
  - UUID retry exhausted -> raises OIDCError

q-8: filled the unit-level coverage holes the multi-stage review flagged:
  - test_validate_id_token_retry_after_kid_rotation — direct unit test of
    the OIDCKeyNotFoundError path with real RS256 keys + JWKS rotation
    (previously only exercised end-to-end through the handler).
  - test_callback_uses_pending_audience_not_handler_audience — pins down
    the bug-3 fix by decoding the issued JWT cookie and asserting aud
    matches the audience stored at /authorize time, not the handler param.
  - test_apply_role_mapping_int_claim / _dict_claim — exercises the
    else: values = [str(claim_value)] branch for non-string non-list
    claim shapes.
  - TestFetchJWKS — non-200 status, non-dict body, dict-missing-keys,
    keys-not-list, transport network error.
  - TestExchangeCode network/4xx/5xx error tests (the non-dict-body case
    already shipped in batch 5).

Also a small production hardening that fell out of writing the
TestFetchJWKS::test_fetch_jwks_non_dict_body_raises test: fetch_jwks now
guards isinstance(result, dict) before result.get("keys"), matching the
shape-check pattern that discover_oidc and exchange_code already use.
A list/null body now surfaces as OIDCError("...not a JSON object") rather
than AttributeError leaking up to the lifespan.

(cherry picked from commit 5c11ab985f)
2026-05-07 17:35:20 -07:00
Patrick Buckley 6d532ed776 refactor(oidc): quality cleanup (bug-3, q-1/3/4/6/7/9/10/11/12/13)
Eleven small maintenance fixes; no behavior change beyond bug-3.

bug-3: pending.get('audience', audience) couldn't fall back because
  pop_oidc_pending_state always returns a dict with the audience key
  set verbatim from a non-null TEXT column. Replaced with
  pending.get('audience') or audience to cover the empty-string case
  defensively. Comment explains the security rationale.

q-1: extract _env_or_cfg_str / _env_or_cfg_bool helpers in oidc.py;
  load_oidc_config's six near-identical env-or-config blocks collapse
  to one-liners. role_map / trusted_endpoint_hosts / redirect_base
  retain bespoke parsing.

q-3: discover_oidc narrows except (httpx.HTTPError, ValueError, KeyError)
  with exc_info=True.

q-4: OIDC_STATE_TTL_SECONDS = 300 constant in oidc.py; auth.py imports
  and passes it explicitly. Storage signatures keep the literal default
  (storage layer doesn't know OIDC TTL semantics).

q-6: hoist runtime imports (OIDCError, OIDCKeyNotFoundError, exchange_code,
  fetch_jwks, provision_oidc_user, validate_id_token, build_authorize_url,
  generate_pkce_verifier) to module scope in auth.py. The genuine cycle
  is only oidc._derive_username -> auth.is_valid_username, kept
  function-scoped. test_oidc_handlers.py mock targets repointed to
  turnstone.core.auth.X to match the new binding.

q-7: comment + docs explain the 'oidc' vs 'oidc-default' assigned_by
  marker distinction.

q-9: OIDCIdentity / OIDCPendingState TypedDicts in storage protocol.
  Implementations construct via TypedDict syntax so mypy structurally
  verifies all required fields.

q-10: fetch_jwks narrows except (httpx.HTTPError, ValueError); docstring
  matches.

q-11: rename generate_pkce_pair -> generate_pkce_verifier; return only
  the verifier (build_authorize_url already recomputes the challenge).

q-12: extract _buildOidcRow helper in admin.js so future field additions
  go in one place.

q-13: OIDCConfig docstring lists startup-config vs discovery-derived
  field groups.
(cherry picked from commit bae4adca12)
2026-05-07 17:35:20 -07:00
Patrick Buckley c3d9cdae82 perf(oidc): batch perf hardening (perf-1..8)
Eight independent perf wins on the OIDC hot path:

perf-1: list_users() full-scan setup-gate replaced with new count_users()
  on both authorize and callback. Saves a full users-table fetch per login.

perf-2: handle_oidc_callback's sync DB chain wrapped in asyncio.to_thread
  for cleanup, pop_oidc_pending_state, count_users, and provision_oidc_user.
  handle_oidc_authorize gets the same treatment for count_users and
  create_oidc_pending_state. Event loop no longer blocks for the full
  callback duration on Postgres deployments.

perf-3: apply_role_mapping N+1 collapsed via new replace_oidc_roles
  storage method. One transaction handles the diff + insert + delete
  instead of 2N+1 commits per login. Returns (added, removed) so the
  caller can still emit per-role audit logs.

  The diff respects the documented invariant "manually-assigned roles
  are never touched" — desired_role_ids is filtered against rows where
  assigned_by != 'oidc' before computing added/removed. This prevents a
  PK conflict (Postgres lockout) or silent OR-IGNORE no-op (SQLite lying
  return) when admin-ui or oidc-default already holds the same role_id.

perf-4: provision_oidc_user no longer re-queries list_user_roles after
  apply_role_mapping. The new-user builtin-viewer fallback is gated on
  desired_role_ids being empty, which is information apply_role_mapping
  already returned.

perf-5: JWKS refetch dedup via asyncio.Lock on app.state. Both lazy-fetch
  (cold-start recovery) and rotation paths share the same lock with a
  double-check pattern: re-resolve kid against the current cache before
  issuing a new GET. N concurrent callbacks during rotation now produce
  at most 1 fetch.

perf-6: _derive_username's 9-suffix loop collapsed via new
  find_existing_usernames(candidates) -> set query. Worst case drops
  from 13 sequential queries to 1 + up-to-3 UUID-retry queries.

perf-7: cleanup_expired_oidc_states gated to once-per-60s per process
  via app.state.oidc_last_cleanup_monotonic. The pop already deletes
  the consumed row; the bulk cleanup is only relevant for abandoned
  authorize flows, so frequency was overkill.

perf-8: Long-lived httpx.AsyncClient stashed on app.state.oidc_http_client
  by initialize_oidc_state. discover_oidc/fetch_jwks/exchange_code accept
  an optional client= kwarg; when set, skip the per-call AsyncClient
  context-manager. New close_oidc_state lifespan teardown closes it.
  Tests pass client=None to keep the transient-client legacy path.

New storage methods (sqlite + postgresql):
- count_users() -> int
- find_existing_usernames(candidates) -> set[str]
- replace_oidc_roles(user_id, desired) -> (added, removed)

(cherry picked from commit 39a647f39c)
2026-05-07 17:35:20 -07:00
Patrick Buckley 366d316941 fix(oidc): callback robustness — typed exceptions, shape checks, log sanitize, JS race (bug-4, bug-5, bug-6, sec-4)
Four small hardening fixes on the OIDC callback hot path:

bug-4: JWKS rotation retry was matching the substring 'not found in JWKS'
  inside an OIDCError message. A future rephrasing would silently break
  key rotation. Adds OIDCKeyNotFoundError(OIDCError); validate_id_token
  raises the subclass at the kid-not-found site; handle_oidc_callback
  catches it explicitly. Other 'not found' errors in validate_id_token
  remain as plain OIDCError.

bug-5: tokens['id_token'] raised KeyError if the IdP returned 200 without
  id_token. exchange_code now rejects non-dict response bodies; the
  callback validates id_token shape (must be non-empty str) before
  passing to validate_id_token. Both raise OIDCError, surfaced as the
  standard 'Authentication failed' redirect.

bug-6: shared_static/auth.js — the OIDC error display raced showLogin's
  /v1/api/auth/status fetch via a 300ms setTimeout. showLogin now takes
  an optional oidcError parameter and paints it after _switchMode clears
  the error, in both the success and catch branches of the fetch.

sec-4: oidc.py exchange_code's non-200 OIDCError interpolated up to 500
  bytes of attacker-controlled IdP body, which then went to log.warning
  via 'OIDC callback failed: %s'. CRLF in resp.text could forge log
  lines. New _sanitize_log_text helper escapes control chars via
  unicode_escape and caps at the rendered length.
(cherry picked from commit 0af3adae1d)
2026-05-07 17:35:20 -07:00
Patrick Buckley 3e87f4262e fix(oidc): atomic user + identity provisioning to prevent orphan rows (bug-1)
provision_oidc_user previously called create_user (INSERT OR IGNORE
on SQLite — silent no-op on UNIQUE conflict), then create_oidc_identity
(also INSERT OR IGNORE), then apply_role_mapping which writes user_role
rows for the supposedly-new user_id. On a username TOCTOU race or
concurrent (issuer, sub) double-create, both inserts no-opped but
user_role rows were already written — leaving orphan rows pointing
at a user_id that doesn't exist.

PostgreSQL's create_user raised IntegrityError instead of silently
no-opping so it produced a misleading 'Authentication failed' error
without orphans, but the user-facing UX was equally poor.

Adds StorageConflictError to the storage protocol and create_oidc_user
that does both inserts in one transaction. Username collision and
(issuer, subject) collision both raise StorageConflictError, mapped
to OIDCError by provision_oidc_user. Crucially the new code does not
silently bind a colliding-username new identity to the existing user
— that would be an account-takeover vector. It raises.

SQLite uses BEGIN IMMEDIATE inside the try block so lock-contention
errors surface as StorageConflictError instead of leaking the raw
sqlalchemy OperationalError.

PostgreSQL relies on SQLAlchemy 2.x begin-on-demand semantics; the
explicit conn.commit()/rollback() in the catch block is the only
materialization path. Discrimination on PG uses
exc.orig.diag.constraint_name with message-substring fallback.

(cherry picked from commit 11618bb1d7)
2026-05-07 17:35:20 -07:00
Patrick Buckley f50b559792 fix(oidc): require TURNSTONE_OIDC_REDIRECT_BASE; drop Host-header fallback (sec-2)
_build_oidc_redirect_uri previously fell back to the request Host
header when redirect_base was unset. With a permissive reverse proxy
or direct backend access, a spoofed Host minted an authorize URL
pointing to attacker-controlled host — combined with a permissive
IdP redirect_uri allowlist this enables auth-code interception.

There is no production scenario where a Host-derived redirect_uri is
correct, so this fails closed:

- initialize_oidc_state checks redirect_base after discovery succeeds
  and disables OIDC (with an explicit error log naming the env var)
  if it's empty. Runs before fetch_jwks so a misconfigured deploy
  doesn't make a wasted JWKS call.
- _build_oidc_redirect_uri simplifies to f"{redirect_base}/v1/api/auth/oidc/callback".
  request parameter dropped; both call sites (handle_oidc_authorize,
  handle_oidc_callback) updated.
- docs/oidc.md promotes TURNSTONE_OIDC_REDIRECT_BASE from "Recommended"
  to "Required" with the security rationale.

(cherry picked from commit 52aba17740)
2026-05-07 17:35:20 -07:00
Patrick Buckley c6b3c0bc5f refactor(oidc): unify server+console lifespan via initialize_oidc_state (q-2, bug-2)
The OIDC discovery + JWKS prefetch block was duplicated byte-for-byte
between turnstone/server.py and turnstone/console/server.py. The bare
except branch in that block also left app.state.oidc_config unchanged
on unexpected exceptions — leaving the runtime with enabled=True and
empty endpoints, producing malformed authorize URLs.

Extracts initialize_oidc_state(app_state) into turnstone/core/oidc.py
which guarantees a coherent post-condition on every code path:
- discovery exception -> oidc_config replaced with enabled=False, jwks_data=None
- discovery returns enabled=False -> jwks_data=None
- JWKS prefetch fails -> jwks_data=None but enabled=True preserved (the
  callback's lazy-fetch retry path remains the recovery)
- success -> oidc_config + jwks_data both populated

Also hardens discover_oidc against non-dict discovery responses
(list/null/string/int) — previously these raised AttributeError out
of doc.get and propagated past the lifespan's bare except.

server.py and console/server.py lifespan blocks collapse to a single
await initialize_oidc_state(app.state) call.

(cherry picked from commit 6f9e140a41)
2026-05-07 17:35:20 -07:00
Patrick Buckley cefb74a226 fix(oidc): SSRF + plaintext credential exfil via discovery doc (sec-1, sec-3)
OIDC discovery-document endpoints (token_endpoint, jwks_uri,
userinfo_endpoint) were stored verbatim in OIDCConfig and later passed
to httpx without revalidation. Only the issuer URL was checked. A
hostile or compromised IdP could return token_endpoint pointing to an
internal IP (169.254.169.254, 10.0.0.0/8, etc.) and Turnstone would
POST the client_secret there.

Extracts the existing scheme/userinfo/SSRF check into
_validate_url_no_ssrf, adds validate_discovered_endpoint that runs the
same checks plus an issuer-binding check, and wires it into
discover_oidc for authorization_endpoint, token_endpoint, jwks_uri,
and userinfo_endpoint (when present).

Issuer binding accepts:
- Same (scheme, hostname, effective port) as the issuer.
- A hostname in _KNOWN_TRUSTED_ENDPOINT_HOSTS for the issuer (Google's
  multi-origin discovery is in the allow-map by default).
- A hostname in OIDCConfig.trusted_endpoint_hosts, settable via
  TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS env var or config.toml, for
  IdPs not in the static map.

Effective port comparison treats https://host and https://host:443 as
the same origin (urllib.parse.urlparse leaves the explicit form's port
as 443 and the implicit form's as None).

24 new tests cover the validator, the Google known-hosts path, the
operator allow-list, default-port equivalence, foreign-host
rejection, private-IP rejection, embedded credentials, and DNS
rotation between issuer check and endpoint use.

(cherry picked from commit 0df7dc026b)
2026-05-07 17:35:20 -07:00
Patrick Buckley 273d547f4e chore: bump version to 1.5.7 2026-05-04 03:04:09 -07:00
Patrick Buckley bbb404c363 feat(console): inline node picker replaces back-to-console banner (#475)
* feat(console): inline node picker replaces back-to-console banner

Drops the 32px banner the console proxy used to inject above proxied
server-UI pages and replaces it with an inline node-id pill in the
existing #ui-header.  Click the pill to open a dropdown that lists
healthy nodes (health dot, ws count, reachable/degraded/unreachable
text) plus a top-row link back to the console.

Reuses the .ws-tab-dropdown shell from ui/static/style.css for
animation, shadow, theme override, and item layout, so the picker
visually matches the workstream-tab chevron menu it sits next to.
Keyboard nav (ArrowDown/Up/Home/End/Tab/Escape) mirrors the chevron
menu's handler with cross-reference comments at both sites.
Lazy-fetches /v1/api/cluster/nodes against the console origin
(bypassing the prefix shim) on first open.

Reclaims 32px of vertical space, consolidates three separate
"you're on node X via console" indicators into one, and turns the
wayfinding chrome into a real cluster-nav primitive.

* fix(console): address Copilot review on node picker

- Request /v1/api/cluster/nodes?limit=1000 (collector's hard cap)
  instead of relying on the default 100 — clusters with more than
  100 nodes were silently dropping rows from the picker.
- Hand off focus to the first menu item after the async fetch
  resolves: openMenu()'s deferred focus hook ran while only the
  skeleton was in the DOM, so first-open keyboard users were
  stranded on the trigger until they pressed an arrow key.
- Tab now closes the menu without preventDefault, so focus moves
  to the next focusable element on the first press (ARIA APG menu
  pattern).  Escape still preventDefault + returns to the pill.
- Cap pill max-width at 240px and ellipsize the id span; node ids
  are accepted up to 256 chars upstream and could otherwise push
  the title and right-side controls off the appbar.  Pill carries
  a title attribute so the full id is still legible on hover.
2026-05-04 02:59:35 -07:00
Patrick Buckley 072113f7ca fix(session): properly inject queued user messages mid-loop (#474)
* fix(session): properly inject queued user messages mid-loop

Two queued-user-message bugs in ``ChatSession.send()``.

**Mid-tool-call: ``Unexpected role 'tool' after role 'user'`` on Mistral.**
The ``supports_tool_advisories`` capability flag (default False for
unknown openai-compatible models) routed cap-off providers down a
short-circuit branch in ``_collect_advisories`` that called
``_flush_queued_messages`` directly. That appended a ``user`` turn
between ``assistant(tool_calls)`` and ``tool``, which mistral-common's
``_validate_message_order`` rejects with a 400.

Drop the flag. All providers now run the unified path: queued user
messages become ``UserInterjection`` advisories that ride inside the
tool result envelope via ``wrap_tool_result``, splicing
``<system-reminder>`` text into the tool message's content. Role
sequence stays ``assistant → tool``. Live-confirmed on Mistral
medium and Qwen3 — both correctly distinguish system-reminder from
tool stdout in their reasoning.

**Mid-stream: queued message orphaned until next user send.**
After a no-tool assistant turn, ``_flush_queued_messages`` would
append the queued user message to history and the loop would
``break``, leaving the message at the tail of history with no
model response. Visible as "two sends to get one reply".

``_flush_queued_messages`` now returns ``bool``. The no-tool branch
``continue``s on drain instead of ``break``ing, so the model gets a
turn over the extended history.

Tests:
- ``test_collect_advisories_drains_text_queued_messages_to_persistent``
  pins the unified-path drain (text-only queue → ``UserInterjection``,
  no separate user turn appended to ``self.messages``).
- ``test_send_continues_when_messages_queued_during_streaming`` pins
  the loop-continue behavior (fails with 1 stream call pre-fix,
  passes with 2 post-fix).

* fix(session,ui): reject queued attachments + paperclip busy state

Copilot pointed out that the attachment-bearing branch in
``_collect_advisories`` had the same role-ordering bug as the
text-only path that 802658f fixed: an attachment-bearing queued
item would still call ``_append_user_turn`` mid-tool-call,
injecting ``user`` between ``assistant(tool_calls)`` and ``tool``.

Pragmatic fix: don't allow attachments to be queued at all.

**Backend.** ``ChatSession.queue_message`` raises a new
``AttachmentsNotQueueableError`` when called with non-empty
``attachment_ids``. The interactive ``/send`` route catches it,
releases reservations via the existing ``_release_reservation_on_fail``
hook, and surfaces ``status: "attachments_busy"`` to the caller
with the IDs in ``dropped_attachment_ids``. The coord adapter
mirrors the cleanup (releases the soft-locked reservation taken
for ``_send_id``) so the create-with-attachments path can't leak.

Now that the queue can never carry attachments, the per-item
``att_ids`` slot is gone:

- Queue tuple slimmed ``(cleaned, priority, att_ids)`` →
  ``(cleaned, priority)``.
- ``_flush_queued_messages`` collapses to a single combined-text
  user turn (no attachment branch).
- ``_collect_advisories`` queue-drain pushes ``UserInterjection``
  advisories only (no ``attachment_items`` list).
- ``dequeue_message`` no longer unreserves (queue can't reserve).
- ``_resolve_attachment_ids`` had no remaining production callers
  and is deleted along with the tests that exercised it in
  isolation.

**Frontend.** ``Composer.setBusy`` disables the paperclip whenever
busy (regardless of ``queueWhileBusy``) — text still queues,
attachments don't. ``chat.css`` gains a ``.composer-attach:disabled``
rule (mirrors the existing ``.composer-send:disabled`` treatment)
so the affordance actually looks unclickable instead of falling
through to the UA default. ``title`` and ``aria-label`` are kept in
sync for AT users (WCAG 4.1.2).

Both interactive and coordinator UIs handle the new
``attachments_busy`` response with a chat-surface error bubble:

> Attachments can't be sent while the assistant is working.
> Send a text-only message now, or wait and resend with attachments.

Chips stay in the composer so the user can retry once idle.

**Tests.** Replaced the now-impossible ``TestQueuedWithAttachments``
class with a rejection-coverage class. Rewrote the
``_queue_with_attachment`` route-test fixture to reserve directly
via ``reserve_attachments`` (the queue path no longer reaches the
reserved state). Added a route-level test for the new
``attachments_busy`` contract.
2026-05-04 02:59:35 -07:00
Patrick Buckley 7f1b0acf7a Bound search tool output against pathological inputs (#473)
* Bound search tool output against pathological inputs

Replaces the per-line truncation with a fully bounded pipeline so the
search tool can no longer overflow the LLM context — or OOM the parent —
on minified bundles, multi-GB JSONL records, or huge result sets.

Backend:
- Prefer ripgrep when on PATH; grep is the fallback. Detection is
  cached via functools.cache.
- ripgrep flags do most of the bounding natively: --max-columns 1024
  + --max-columns-preview, --max-filesize 10M, --max-count 100,
  --no-config, --no-messages, plus negative globs for the same
  noisy directories grep has been excluding.
- ripgrep added to the Dockerfile.

Streaming subprocess (_search_capture):
- subprocess.Popen with a streaming, byte-capped stdout read (4 MB).
  Defends against single-line files (training data, minified bundles)
  that would have OOM'd the previous subprocess.run capture.
- threading.Timer watchdog enforces tool_timeout even when the
  pipe read is blocked in the kernel — proc.wait(timeout=…) alone
  was insufficient because the read sat ahead of it.
- Stderr drained in a daemon thread to avoid pipe-deadlock when the
  child writes to stderr while we're still reading stdout. Cap on
  captured stderr keeps a hostile child from growing the buffer.

Tier-based formatter (_format_search_results):
- Tier 1: full path:line:content output, stream-emitted with a
  running-cost short-circuit so we never materialize past the budget.
- Tier 2: K samples per file with overflow notes; K is computed
  analytically from budget / file_count / avg-line-length so we hit
  the right ladder rung in a single pass.
- Tier 3: per-file counts only, also budget-bounded with a tail line
  reporting the omitted files. Sorted by descending count.
- Total output budget (32 KB) is well under tool_truncation, so the
  head+tail _truncate_output strategy never silently drops middle
  files in a search result.

Argument injection fix:
- The ripgrep arg list was missing the `--` separator that the grep
  branch already had. With auto_approve on the search tool, that was
  exploitable: path='--pre=COMMAND' would have made ripgrep run the
  script as a per-file preprocessor and surface its stdout. Added
  `--` and a regression test.

State-machine cleanup in _exec_search:
- rc < 0 (signal-killed by something other than us) now surfaces a
  dedicated 'killed by signal N' message instead of being parsed as
  success.
- capped + zero parsed records (e.g. one multi-MB line with no \n)
  now returns a dedicated byte-cap message instead of the malformed-
  output message that previously masked the real cause.
- _report_tool_result descriptions now match the returned payload
  (no more 'no matches' tag on a 'malformed' payload).

Defence-in-depth on env scrub:
- RIPGREP_CONFIG_PATH, GIT_CONFIG, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM
  added to _EXPLICIT_SCRUB. We pass --no-config on the rg CLI today,
  but if a future caller forgets the flag, an attacker who can set
  one of these env vars could plant a config containing --pre=… and
  recreate the same RCE shape.

Tests:
- TestSearchLineTruncation rewritten to mock _search_capture instead
  of subprocess.run (the previous tests passed ChatSession kwargs
  that no longer satisfy the constructor).
- TestSearchBackendSelection covers rg/grep detection and arg
  construction, including the --pre flag-injection regression.
- TestSearchOutputBudget exercises Tier 1/2/3 directly.
- TestSearchCaptureStreaming spawns real Python subprocess writers
  to exercise the byte-cap trim, mega-line-no-newline edge case, the
  watchdog timeout when the child writes nothing, and the stderr
  drain under load.
- test_env_scrub picks up the new tool-config keys.

* Address Copilot review on #473

- Budget the Tier 2/3 header up front so the formatter's emission stays
  strictly within _SEARCH_OUTPUT_BUDGET. Previously the fit checks only
  counted body bytes, letting the final string overflow by ~120 chars
  (header + separator) and triggering _truncate_output's head+tail
  dropout — exactly the shape this code was trying to avoid.
- Restore the (5, 3, 1) ladder in Tier 2: the analytical K from perf-2
  is kept as a starting estimate, but if that K's actual emission
  doesn't fit (the estimate ignores the header and overweights shared-
  path compression) we step down through the ladder before falling
  through to Tier 3. The previous one-shot K could collapse to counts-
  only when 3/file or 1/file would have fit.
- Only normalise rc to 0 in the capped-output path when rc < 0 (our
  SIGKILL). There's a narrow race where the child can exit naturally
  between our read and our kill; preserving a non-negative rc means
  rg's rc=2 ('matches found but some files had errors') no longer
  silently turns into a clean success when the byte cap also fires.
- Clarify _MAX_SEARCH_LINE_LENGTH doc: the cap applies to the content
  portion (after path:lineno:), not the whole emitted line.
- Add explanatory comments on the two intentional `except Exception:
  pass` blocks in _search_capture (stderr drain, pipe close in the
  cleanup finally) so static analysis and future readers can see the
  silence is deliberate.
- Tighten the budget tests: now assert strict `<= _SEARCH_OUTPUT_BUDGET`
  instead of the +512-char slack that was masking the header overflow.
- New regression tests:
  - Tier 2 ladder step-down (K=5 over budget, K=3 fits, no Tier 3 fall-through)
  - capped + rc=2 surfaces stderr instead of being normalised to success
  - capped + rc<0 (our SIGKILL) flows through as a partial-result success

* chore(search): post-review cleanup

Follow-up to the Copilot-review fixes in 39d2aa2 — these are all small
quality items (no behaviour change, no new tests).

- q-1: collapse the Tier 2 candidates filter to a single expression.
  Drops the redundant inner ``max(estimated_k, 1)`` and the unreachable
  ``if not candidates`` branch (the ladder ends in 1 and ``estimated_k``
  is already floored at 1, so the comprehension always yields ≥ ``[1]``).
  ``or [...]`` is kept as defence against future ladder changes.
- q-2: update _format_search_results docstring to match the new ladder
  semantics (analytical seed → step down through (5, 3, 1) from the
  highest rung ≤ the estimate). The previous wording suggested every
  Tier 2 attempt started at 5.
- q-3: combine the two ``from turnstone.core.session import ...``
  statements in test_tier2_steps_down_ladder_before_falling_to_tier3
  into a single top-of-function import (matches the surrounding tests).
- q-4: shorten the explanatory comments on the two best-effort cleanup
  paths in _search_capture to one line each. Both sites now read with
  the same shape ("# best-effort: pipe may be torn down by ...").
- q-5: trim the _MAX_SEARCH_LINE_LENGTH comment from 7 lines back to 3.
  Keeps the load-bearing semantic (cap is on the content portion only)
  and the pathological-line defence; drops the paths-aren't-bounded
  parenthetical, which was background reading rather than WHY.
2026-05-04 02:59:35 -07:00
renovate[bot] 6904bd8f39 chore(deps): lock file maintenance 2026-05-04 02:59:35 -07:00
renovate[bot] 4d677d1ebf chore(deps): update github actions 2026-05-04 02:59:35 -07:00
Patrick Buckley 35f462a46d chore: bump version to 1.5.6 2026-05-03 13:44:50 -07:00
Patrick Buckley ec74334e74 feat(providers): api_surface toggle + mistral medium reasoning fix (#469)
* feat(providers): api_surface toggle + mistral medium reasoning fix

Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions.  The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).

Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
  and thread it through `create_provider` / `model_registry.get_provider`.
  `openai-compatible` defaults to Chat Completions; operators can flip
  individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
  on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
  `chat_template_kwargs`.  Operators running gpt-oss-style local
  templates that consume `reasoning_effort` from the chat template now
  opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
  server-side at create/update time; pre-filled by Detect via the
  profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
  api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
  for capability and server_compat resolution; previously the fallback
  passed `alias=None`, which silently dropped per-model caps on the
  agent path.

Tests: 5117 passed (-m "not live"); ruff + mypy clean.

* fix(providers): don't auto-suggest Responses for Mistral medium

vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls.  Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.

Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile.  Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.

* fix(providers): address Copilot review on PR #469

- providers/__init__.py: drop the redundant *_responses_provider /
  *_chat_provider names; have create_provider use _openai_provider and
  _openai_compat_provider directly so they're not flagged as unused
  globals.
- console/server.py: tighten _validate_api_surface to a strict equality
  match against the canonical {"chat", "responses"} set.  The previous
  strip().lower() membership check accepted ' Responses '/'CHAT' but
  stored the raw string verbatim, which then failed to round-trip
  through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
  type, api_surface, extra_body) on provider == "openai-compatible" at
  save time so toggling provider away can't leave a stale hidden surface
  selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
  longer flags the call as a wrong-name keyword (the point of the test
  is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
  for the api_surface validation on both create and update — covers the
  bogus-value rejection, non-canonical-string rejection, and the happy
  path persisting through to the refreshed registry.
2026-05-03 13:40:30 -07:00
Patrick Buckley 2fd0c29a92 fix(memory): query-aware candidate selection + OR-of-terms search (#468)
* fix(memory): query-aware candidate selection + OR-of-terms search

The system-message memory composition path used a recency-ordered
candidate set (`_list_visible_memories(limit=fetch_limit)`).  On
deployments with more than `fetch_limit` (default 50) visible
memories, BM25 only ever ranked the 50 most-recently-touched memories
— a relevant memory written months ago was silently invisible
regardless of how well it matched the recent context.  Multi-word
search at the SQL layer used AND-of-terms, killing recall on any
multi-word query without an exact field overlap.

## Functional changes

- `_init_system_messages` (`turnstone/core/session.py`): extract
  recent context first, then `_search_visible_memories(context)` to
  pull query-aware candidates.  Search hits below `fetch_limit` union
  with the recency list (deduped by memory_id) so the BM25 candidate
  pool is always a SUPERSET of the prior recency-only pool — even on
  noisy queries where the cap fills with stopwords, the recency-50
  the original bug surfaced still reaches BM25.  Empty context skips
  search entirely.  Candidate-selection logic extracted into
  `_select_memory_candidates`.

- `search_structured_memories` (PostgreSQL + SQLite): per-term
  clauses join with OR instead of AND.  A row matches if ANY term
  matches ANY of name/description/content.  Downstream BM25 narrows
  back down by relevance.

## Perf hardening

- Collapse the 1-3 fanned scope queries into a single SQL.  New
  backend methods `list_visible_structured_memories` /
  `search_visible_structured_memories` union the visibility scopes
  into one WHERE OR-group, so a composition rebuild now hits the DB
  at most twice (search + recency) instead of up to six times.

- Cap and normalize search terms.  Composition can hand a multi-KB
  pasted message to ILIKE-based search; without a cap, every distinct
  token would emit one unindexable predicate per scope-fanned query.
  `normalize_search_terms` (`storage/_utils.py`) de-dupes
  case-insensitively, drops <2-char tokens, and hard-caps at 16.

- Per-turn search cache.  `_init_system_messages` fires from many
  call sites within one turn (state transitions, MCP refresh, tool
  results) and the recent-context query is identical across them.
  Session-instance cache keyed by (query, mem_type, limit) absorbs
  the duplicates; invalidated in `_append_user_turn` and after
  memory save/delete tool actions.

- Stable secondary sort by `memory_id`.  `updated` is second-precision
  and `touch_structured_memories` can land a batch on identical
  timestamps; without a tie-breaker SQL returns rows in
  implementation-defined order, BM25 input shuffles, and the
  LLM-side prompt cache misses across calls.  All four backend ORDER
  BYs now break ties on `memory_id ASC`.

## Quality cleanups

- Coalesce `memory.search.term_count` + `memory.search.zero_results`
  into a single `memory.search` log carrying both `term_count` and
  `result_count`.
- New `memory.composition` log: source / candidates / injected.
- Promote a shared `make_chat_session` factory to `tests/_helpers.py`.
- Rename SQL builder local `extra` -> `scope_filters` for clarity.
- Add docstrings on `search_structured_memories` so the AND->OR flip
  survives future readers.

## Tests

Adds 20 tests across `tests/test_structured_memory.py`,
`tests/test_structured_memory_storage.py`, and
`tests/test_memory_relevance.py`: recency-ceiling regression,
empty-query fallback, sparse-match union, recency-preserved-when-
search-returns-noise (locks in the pool-superset invariant),
OR-of-terms on both backends, scope filtering preserved,
search-facade multi-word behavior, term-cap normalization, the new
visible-scope helpers (list + search + empty-scopes guard),
coord-scope composition isolation, end-to-end
`memory(action='search')` tool execution, per-turn cache hit +
invalidation, and stable ordering under tied `updated` timestamps.

Memory test sweep: 102/102.  Broader regression
(session, storage, coordinator, load_skill): 411/411.

* fix(memory): address Copilot review on PR #468

Three follow-ups from Copilot's inline review:

1. SUPERSET invariant violation (Copilot, session.py:5510).
   `(search_hits + extra)[:fetch_limit]` capped the union back down to
   fetch_limit, evicting the recency tail when search added distinct
   hits.  Recency tail is exactly where ancient-but-recently-touched
   memories live — the recall this PR is supposed to improve — so
   tail eviction recreated the bug for the narrow case where a query
   term fell off the 16-cap and the matching memory sat in
   recency[40-49].  Drop the cap; both halves are already SQL-capped
   at fetch_limit, so the union is at most 2 × fetch_limit (~100 with
   defaults).  BM25 over 100 candidates in pure Python is sub-ms;
   irrelevant recency fillers get score=0 and don't pollute ranking.
   Updates the docstring to actually be honest about the invariant.
   Adds `test_recency_tail_preserved_when_search_adds_distinct_hits`
   that locks the behavior in: 5 search hits + 10 recency = 15-item
   pool, every recency item present, source="union".

2. Unbounded `query.split()` in normalize_search_terms (Copilot,
   _utils.py:74).  `str.split()` allocates the full token list before
   the cap-after-16 break, so a 100KB pasted query did MB of throwaway
   work even though only 16 tokens entered SQL.  Switch to
   `re.finditer(r'\S+', query)` — streaming iterator, stops scanning
   at the first 16 normalized terms regardless of input size.

3. Misleading + unbounded log term_count (Copilot, session.py:8571).
   `len(item["query"].split())` had two problems: same unbounded
   split as #2, and the value reported the raw input token count
   rather than the normalized term count that actually hit the SQL
   WHERE clause — misleading metric for an operator trying to
   understand storage-side behavior.  Switch to
   `len(normalize_search_terms(item["query"]))` — accurate count, and
   bounded for free via #2.

Refuted: github-code-quality flagged `...` bodies in the new Protocol
methods as "statement has no effect."  False positive — `...` is the
canonical Protocol body convention, used 213 other times in the same
file.

Memory test sweep: 103/103.  Broader regression: 411/411.
2026-05-03 13:40:30 -07:00
Patrick Buckley 1207d27363 fix(tests): isolate metrics-singleton swaps so they don't leak across files
CI failure on main: test_publish_records_metric_outcome saw an empty
calls list — its monkeypatch was patching a different metrics
instance from the one `_publish_models_metadata` reads.

Two changes:

- test_close_reason_persistence.py: replace the bare
  `srv_mod._metrics = MetricsCollector()` assignment in `_make_app`
  with an autouse `monkeypatch.setattr(srv_mod, "_metrics", ...)`
  fixture so the test's metrics swap auto-restores. Other test
  files (test_auth.py, test_server_attachments_endpoints.py) carry
  the same anti-pattern; left for a follow-up since they're not on
  the critical path here.

- test_server_node_models_metadata.py: switch the publish-helper
  metric test to a string-form `monkeypatch.setattr("turnstone.
  server._metrics", FakeMetrics())` so it replaces whatever binding
  the live module currently holds, regardless of what other tests
  did to it. Robust against future leaks of the same shape.
2026-05-03 13:40:29 -07:00
Patrick Buckley 733c9818d4 feat(coord): expose healthy model aliases per node on list_nodes (#466)
* feat(coord): expose healthy model aliases per node on list_nodes

Surfaces a `model_aliases` field on each `list_nodes` row so a
coordinator can discover which model aliases each cluster node will
accept on `spawn_workstream(model=...)` without an HTTP fan-out.

Each server projects its registry into a `models` entry on
`node_metadata` (`{alias, provider, healthy}` per alias) at lifespan
startup, on every 30s heartbeat tick, and after `internal_model_reload`.
The publish helper short-circuits on a payload-equality cache so a
stable cluster doesn't pay UPSERT churn — exposed via the new
`turnstone_node_models_publish_total{outcome="written|skipped"}`
Prometheus counter so operators can graph cache hit-rate.

Coord client filters the per-alias rows to healthy aliases only and
drops the provider-side model identifier (`cfg.model`) — coords kept
reaching for it when they should pass the local alias.

* fix(coord): address Copilot+CodeQL feedback on list_nodes models work

- internal_model_reload: reuse a single get_storage() local across the
  registry load and the metadata publish (Copilot:3047)
- _collect_node_models_metadata: iterate sorted aliases so two
  structurally identical registries built in different insertion orders
  serialize to the same JSON — directly improves the publish-cache hit
  rate exposed via turnstone_node_models_publish_total (Copilot:3105)
- tests: drop mixed turnstone.server import style flagged by CodeQL —
  hoist _metrics into the from-import block, and use sys.modules in
  the shutdown-race regression test instead of `import as srv`
2026-05-03 13:40:29 -07:00
Patrick Buckley 28a2779c10 fix(core): scope rehydrate fallback to manager, fix resume orphan
Address Copilot feedback on PR #465:

1. The has_alias fallback in both session_factories silently rewrote
   any unknown caller-supplied alias to the default, including on the
   fresh-create path where the create handler maps the factory's
   ValueError to a 503 with operator-friendly text. A typo in
   body.model would now silently start a workstream on the default
   instead of telling the caller their requested model could not be
   resolved. Move the fallback out of the factories: each factory
   raises again on unknown aliases, and SessionManager filters stale
   aliases out of the rehydrate path via a new ``model_validator``
   constructor kwarg (production wiring passes ``registry.has_alias``
   on both interactive and coordinator).

2. ChatSession.resume()'s elif branch flipped self.model to the
   persisted model name even when the alias was unresolvable, leaving
   the session paired with the constructor's default provider/client
   but a removed model name — a broken state whose next API call
   fails. Drop the model copy: keep the constructor's coherent
   default (provider + model + capabilities) and just log the
   unreachable saved values so the missing alias is auditable.

Tests:
- Move stale-alias coverage from the factory level into
  SessionManager (tests/test_session_manager.py): validator drops
  stale aliases before reaching build_session; live aliases pass
  through unchanged.
- tests/test_sessions.py renamed test_resume_restores_model →
  test_resume_keeps_defaults_when_alias_unresolvable to match the new
  contract.
2026-05-03 13:40:29 -07:00
Patrick Buckley 56364b0b5b fix(core): preserve workstream model + config on rehydrate
SessionManager.open() was calling build_session(ws) without a model
arg on the rehydrate path. The session_factory then resolved the
*current* default alias, ChatSession.__init__'s _save_config() (INSERT
OR REPLACE per-key) clobbered the persisted workstream_config with
those defaults, and the subsequent resume() "restored" what was now
the default — silently resetting model_alias, model, temperature,
reasoning_effort, max_tokens, skill, creative_mode, instructions,
token_budget, and notify_on_complete on every reopen and every
service restart, for both interactive and coordinator workstreams.

Three layers:

1. SessionManager.open() now reads workstream_config via
   self._storage.load_workstream_config(ws_id) and threads the saved
   model_alias into build_session(ws, model=saved_alias).

2. ChatSession.__init__ now skips its initial _save_config() when a
   workstream_config row already exists for self._ws_id — protects
   every other persisted knob without having to plumb each one
   through the adapter signature, and catches any future construction
   path that forgets to thread model through build_session.

3. Both session_factories (server.py interactive, console
   session_factory.py coordinator) now treat an unknown caller-
   supplied alias the same as an unset alias: fall back to the
   runtime default rather than raising. Without this, a workstream
   pinned to an alias an operator has since removed from the registry
   would 500 on every reopen — defeating the "best effort restore,
   default if the original is gone" contract this fix is meant to
   deliver. Mirrors _effective_default_alias's existing has_alias
   guard against a stale ConfigStore default.
2026-05-03 13:40:29 -07:00
Patrick Buckley afb5804a7c fix(console): address Copilot feedback on Models → Roles sub-tab
Three changes from PR review:

- Permission gating: hide the Roles sub-tab button when the user
  lacks ``admin.settings``.  The sub-tab loads/saves through
  ``/v1/api/admin/settings``, so an admin with ``admin.models`` but
  no ``admin.settings`` would otherwise see a perpetual 403 loader.
  When Roles is the active sub-tab and the permission check fails,
  snap the panel back to Definitions so the user lands somewhere
  usable.

- Drop the redundant ``/v1/api/admin/model-definitions`` fetch from
  ``loadAdminModelRoles``.  Both entry points (initial Models-tab
  open + ``models_changed`` SSE refresh) flow through
  ``loadAdminModels`` first, which already populates ``_modelDefs``
  + ``_modelDefaultAlias``; ``_saveModelRole`` doesn't touch model
  definitions, so the cached snapshot stays accurate when the save
  chains back here.  Halves the per-render request count and
  removes a wasted round-trip on every cluster-wide model edit.

- Add ``test_models_changed_event.py`` covering the SSE fanout the
  prior commit introduced: each model-definition CRUD endpoint
  emits exactly one ``models_changed``, settings PUT/DELETE only
  emit for keys in ``_MODEL_AFFECTING_SETTING_KEYS`` (parametrised
  over all eight), and unrelated settings (e.g.
  ``session.retention_days``) don't trigger spurious refreshes.
  The expected key set is pinned in the test so a stray addition
  to the allowlist doesn't silently bypass coverage.
2026-05-03 13:40:29 -07:00
Patrick Buckley 4b508a1319 feat(console): add plan_agent + task_agent to Models → Roles
Same shape as the coordinator/judge rows already there: alias dropdown
+ reasoning_effort dropdown sourced from the existing
``model.plan_alias`` / ``model.plan_effort`` and
``model.task_alias`` / ``model.task_effort`` settings.  Adds the four
keys to the SSE ``models_changed`` allowlist so changes from the
Settings API also trigger a live dropdown refresh, and filters them
out of the Settings tab so they only render in one place.
2026-05-03 13:40:29 -07:00
Patrick Buckley 9c2cb185e1 feat(console): consolidate role-model settings + live-refresh dropdowns
Lifts judge and coordinator model assignments out of their respective
admin tabs and into a new Models → Roles sub-tab so role overrides live
next to the model definitions they reference. Forward-looking shape for
the upcoming perception.{audio,image,video} model settings — adding a
new role is one entry in the declarative MODEL_ROLES array.

Also drops the misleading "Coordinator subsystem not configured" home
banner. The session factory already falls back to the registry's
default model when coordinator.model_alias is unset, so the banner was
nagging on fresh installs where the system was actually working. The
related _probeCoordSubsystem / _homeCoordReady plumbing went with it.

Wires SSE-driven live refresh: the console now emits a models_changed
event when a model definition is created/updated/deleted/reloaded, or
when a model-affecting setting (model.default_alias, judge.model,
coordinator.model_alias, coordinator.reasoning_effort) changes.
Connected browsers refetch /v1/api/models on receipt so the home
composer's model dropdown and the Roles sub-tab stay accurate without
a manual reload — fixes the case where editing the underlying model
for an existing alias left the dropdown showing the old model id.

Companion cleanups:
- Renamed .judge-section-* CSS classes to .admin-subtab-* and shared
  them with the Models sub-tab switcher (same a11y attrs, arrow-key
  nav). Old names had no other callers.
- Filtered judge.model out of the Judge Settings sub-tab and
  coordinator.model_alias / coordinator.reasoning_effort out of the
  Settings tab — they live exclusively under Models → Roles now.
- Reworded the _require_coord_mgr 503 messages to point operators at
  the Models tab instead of suggesting they set coordinator.model_alias.
2026-05-03 13:40:29 -07:00
Patrick Buckley 0519b847bd docs(skills): add import-conversation-history SKILL.md
Source-agnostic guide that teaches an agent Turnstone's destination
contracts (workstream + conversations schema, ws_id routing, OpenAI
message shape, tool-call/result pairing, provider_data fidelity blob,
attachment lifecycle) so it can map any external chat export onto them.
Validated against turnstone.core.skill_parser.
2026-05-03 13:40:29 -07:00
Patrick Buckley bbc8b99a9f fix(console): home composer attachments + coord chat user-message pills (#462)
* fix(console): home composer attachments + coord chat user-message pills

Two parity gaps in the console's coordinator surface:

- The embedded creator on the home page accepted only text — the
  paperclip / paste / drop pipeline that the in-coord composer and the
  interactive new-ws modal both expose was missing, so a user couldn't
  attach files at create time. Stage Files in memory (no ws_id yet) and
  ship them multipart on Start; the coord create endpoint already accepts
  multipart via create_supports_attachments=True.

- User messages with attachments rendered as plain text on both live
  send and history replay — no chip cluster like the interactive pane.
  Added appendUserMessageWithAttachments and a structured userAttachments
  list built from _attachments_meta (preferred) or the multipart parts
  themselves, then rendered the same .msg-user-attach pill strip the
  interactive pane uses.

Polish from a designer pass:

- Pill background was --panel-2, equal to the .msg bubble background in
  both themes (border contrast ≈1.4:1, below WCAG 1.4.11). Switched to
  --panel so the pill sits on a different surface than the bubble.
- Capped chip filename width inside the home composer (max-width 200px +
  ellipsis) so a long filename doesn't push the strip past the textarea.
- aria-live="assertive" → "polite" on #home-coord-error; client-side
  validation isn't an interrupt-level event.
- Reserved min-height on .home-composer-error and dropped the
  display: none/block toggling so validation messages no longer reflow
  the active-coordinators list below.

* fix(console): address PR #462 review feedback

- Block home-composer submit when files are staged but the task field is
  empty.  Server's _coord_create_post_install short-circuits on an empty
  initial_message, so the multipart upload would create pending
  attachment rows that never reserve onto a turn — orphaned until the
  GC sweep.  Fail in the browser instead.
- Drop the redundant `part &&` guard in coordinator.js's history-replay
  multipart loop; the earlier `if (!part || ...) continue` already
  filtered.
- Rewrite the home-mount .composer-chip-name CSS comment.  shared/chat.css
  defines .composer-chip{,-size,-remove} but no .composer-chip-name rule
  — the span inherits the parent chip font with no width cap.
- Add smoke-guard string assertions in test_coordinator_page.py for
  appendUserMessageWithAttachments and msg-user-attach so a future
  rename can't silently regress the attachment affordance.
2026-05-03 13:40:29 -07:00
Patrick Buckley bd9f780b21 chore: bump version to 1.5.5 2026-05-01 14:09:38 -07:00
Patrick Buckley 5d14b5f675 fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration (#461)
* fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration

Loading a saved workstream silently dropped tool results and missed
verdict / output-guard / truncation signals on replay. Root cause was
in `Pane.prototype.replayHistory`: an assistant message carrying both
content and tool_calls cleared the `lastToolBlock` anchor before the
following tool-result iteration could attach. The fix reorders content
to render before the tool block (matching live SSE order) and
restructures the tool-result branch to anchor by `data-call-id` so
multi-tool batches render `[hdr A][out A][hdr B][out B]` rather than
bunching outputs at the bottom.

Beyond the bug, replay now reaches near-parity with the live UX:

- Persisted intent verdicts and output_assessments flow through both
  the SSE replay (`_build_history`) and the `/history` REST endpoint
  used by coord. Single shared helper module owns the wire shape.
- Memory/recall calls persist instead of being filtered at storage
  time — full audit trail; UI dims them by default with hover-reveal
  so heavy memory usage doesn't crowd the narrative.
- Truncation indicator surfaces as a sibling pill (consistent across
  interactive + coord) when a tool result hit the 2000-char cap.
- `replayHistory` wraps DOM work in `aria-busy` so screen readers
  don't get a chatty announce-flood on long replays.
- `_build_history`'s storage I/O moves off the event loop via a new
  `events_replay_prepare` async hook for the SSE path; other async
  callers wrap in `asyncio.to_thread`.

Coord parity:

- `/history` REST endpoint decorates tool_calls with verdict +
  output_assessment + truncation flag (was previously raw
  `load_messages` output).
- Coord JS stamps `judge_verdict` / `heuristic_verdict` from
  history-loaded `tc.verdict` so the existing batch render paints
  the persisted pill, seeds the verdict cache to dedupe later live
  SSE events, and emits an inline `.coord-tool-row-warning` chip
  per call instead of a generic chat line.
- Memory/recall dim rule mirrored on `.coord-tool-row[data-tool-name=...]`.

* fix(replay): address PR #461 review feedback + raise tool-result storage cap

Copilot review feedback:

- Sibling-chain dim rule (memory/recall) now adds :focus-within
  alongside :hover for .tool-output / .media-embed / .output-warning
  / .tool-output-truncated — keyboard users tabbing into a faded
  subtree now get full opacity.
- ``cfg.open_post_load`` is now invoked via ``await asyncio.to_thread``
  so its sync ``_build_history`` call (storage I/O for verdict
  indexes + message reconstruction) doesn't block the event loop on
  every workstream open. Mirrors the SSE replay path that's already
  protected via ``events_replay_prepare``.
- Replaced the hardcoded ``2000`` literal in server.py and session.py
  with ``TOOL_RESULT_STORAGE_CAP`` from the shared decoration module
  so the UI truncation-pill detection can't silently desync from the
  storage write side.

While here:

- Raised ``TOOL_RESULT_STORAGE_CAP`` from 2000 → 10000. A 2000-char
  clip routinely cut grep / file-read bodies mid-line, leaving the
  audit trail useless for retrospective debugging. FTS5 + row size
  grow proportionally; the per-tool upper bound is still bounded
  upstream by ``_truncate_output``'s context-budget clamp.
- Updated the user-visible truncation-pill tooltip on both
  interactive and coord to reflect the new cap.
- ``test_decorates_tool_calls_and_marks_truncated`` now references
  the constant instead of a literal so it stays correct on future
  cap changes.
2026-05-01 14:06:08 -07:00
Patrick Buckley 4693fa95f1 chore: bump version to 1.5.4 2026-04-30 23:51:06 -07:00
renovate[bot] c3423d6606 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.8 2026-04-30 23:48:53 -07:00
Patrick Buckley 53f1222c22 refactor(coord): remove priority queue + queue depth indicator + broken CSS
Speculative reliability machinery from the Stage 3 push that turned
out not to address any user-visible bug. The actual fixes (state /
activity disjunction in handleChildState, bulk-fetch race fix in
_fetch_live_block, push approve_request via cluster bus) are what
resolved the wedged-row issues. Manual testing showed the per-tab
SSE listener queue depth never climbed past single digits even when
rows were stuck — overflow was never the cause.

Removed
- ``_CRITICAL_EVENT_TYPES`` + ``_put_with_priority`` helper.
- Per-tab listener queue selective drop (back to plain
  ``contextlib.suppress(queue.Full)`` everywhere).
- ``ClusterCollector._fanout`` reverts to the same.
- WebUI ``_broadcast_intent_verdict`` / ``_broadcast_approval_resolved``
  / ``_broadcast_approve_request`` revert to plain ``put_nowait``.
- ``_queue_stats`` periodic SSE emit + frontend status-bar indicator
  + the supporting CSS rules.
- Broken ``.approval-block`` ``transition: max-height`` /
  ``max-height: 80vh`` / ``overflow: hidden`` rules — the transition
  never fired (nothing toggled max-height) and ``overflow: hidden``
  clipped long verdict reasoning. Layout-shift on auto-expand jumps
  again, which is preferable to clipped content (Copilot review).

Tidied
- ``_CollectorProtocol`` / ``_ManagerProtocol`` method bodies switch
  from ``...`` ellipsis to docstring-only bodies, silencing four
  CodeQL "statement has no effect" warnings without changing the
  Protocol contract.

5024 passed, ruff + mypy clean.
2026-04-30 23:48:53 -07:00
Patrick Buckley 802d87a57f feat(coord): Stage 3 SessionManager Children primitive lift + cluster bus push paths
Lift the Children primitive out of CoordinatorAdapter into universal
SessionManager core primitives, replace the fragile poll + state-event
piggyback paths with first-class cluster bus event types for inline
approval delivery, and clean up the resulting frontend reducer.

Architecture
- New `turnstone/core/children_registry.py` — universal parent → children
  + reverse-lookup primitive with atomic `add_child` (returns parent UI
  for race-free dispatch). Lifted from `CoordinatorAdapter`.
- New `turnstone/core/child_source.py` — `ChildSource` Protocol with
  `SameNodeChildSource` (in-process via SessionManager state observer)
  and `ClusterChildSource` (cross-node via ClusterCollector listener).
- `SessionManager._on_state_change` upgraded to multi-subscriber
  (`subscribe_to_state` / `unsubscribe_from_state`) under a dedicated
  lock; CLI consumer migrated.
- `CoordinatorAdapter` shrunk: 731 → ~640 LOC. Children data lives in
  the registry; fan-out lives in ClusterChildSource. Backward-compat
  property facades dropped; tests updated to use the registry surface.

Cluster bus event vocabulary
- New event types `intent_verdict`, `approval_resolved`,
  `approve_request` flow through both `ClusterCollector._apply_delta`
  (translation from node SSE) and `emit_console_ws_*` (synthesis on
  console pseudo-node).
- `CoordinatorAdapter._dispatch_child_event` re-emits as
  `child_ws_intent_verdict` / `child_ws_approval_resolved` /
  `child_ws_approve_request` on the parent coord's SSE stream.
- New `_broadcast_intent_verdict` / `_broadcast_approval_resolved` /
  `_broadcast_approve_request` no-op hooks on `SessionUIBase`. WebUI
  pushes to the global queue; ConsoleCoordinatorUI pushes to the
  collector. `approve_tools` calls `_broadcast_approve_request` right
  after setting `_pending_approval` so the items reach the coord tree
  immediately, eliminating the bulk-fetch race.

Cleanups
- `pending_approval_detail` piggyback on `ws_state` / `cluster_state`
  removed end-to-end. Bulk fetch + explicit verdict / approve-request
  push are the canonical carriers.
- Browser `_judgePollTick` 90-second poll loop deleted; push path is
  authoritative.
- `urgent` flag on `scheduleLiveFetch` deleted (only caller was 409
  retry; replaced with `invalidateLiveBadge` + standard schedule).
- Console `_fetch_live_block` derives `pending_approval` from a
  disjunction (`activity_state="approval"` OR `state="attention"`
  OR detail present) so the bulk fetch can't return false during the
  state-transition race window.
- Coord-side merge guard in `flushLiveFetches` no longer clobbered:
  `handleChildState` only stamps `sseUpdatedAt` when authoritatively
  clearing detail.
- `child_locality` capability flag removed (was inert dead code).

Reliability
- Selective drop on listener queue overflow: critical event types
  (verdicts, approvals, ws_closed, child_ws_*) evict one oldest item
  to make room rather than dropping themselves on a full queue.
  Best-effort events (state ticks, content tokens, status, activity)
  drop as before. Applied to `SessionUIBase._enqueue`,
  `ClusterCollector._fanout`, and the `WebUI._global_queue` puts in
  the new broadcast hooks.
- `_state_subscribers` snapshot under a dedicated lock so concurrent
  subscribe / unsubscribe during dispatch can't shift the iterator.

UX / a11y
- Loading placeholder in renderChildRow keeps row height stable while
  the bulk fetch is in-flight (sr-friendly aria-label).
- Focus preservation across `_renderChildrenNow` (capture +
  restore by row + marker) and across targeted `_updateChildRow` swaps.
- Layout-shift transition on the approval block max-height; respects
  `prefers-reduced-motion`.
- Sidebar pending count: `(N children · M pending)`.
- Risk pill `aria-label` spells out level + confidence for SR users.
- Per-coord SSE listener queue depth surfaced in the status bar
  (`queue N/500`) with color escalation (warn at >50%, danger at >80%).

Tests
- 305+ test changes across 8 files. New unit tests for
  `ChildrenRegistry`, `ChildSource` (both impls + multi-subscriber
  observer), the new collector emit + apply_delta cases, the dispatch
  cases for new event types, the broadcast hook overrides on both
  WebUI and ConsoleCoordinatorUI, and the focus / placeholder /
  pending-count frontend assertions in `test_coordinator_page.py`.

5024 passed, ruff + mypy clean.
2026-04-30 23:48:53 -07:00
Patrick Buckley 8349d9994d feat(console): multi-select delete UX for Saved Coordinators (#458)
* feat(console): multi-select delete UX for Saved Coordinators

Mirror the per-server "Saved Workstreams" multi-select delete onto the
console's "Saved Coordinators" section.  Coordinator deletes go through
the existing routing proxy at POST /v1/api/route/workstreams/delete
(body-keyed by ws_id, since coordinators live on the node that owns
them) — no backend change required.

Pagination caps the visible page (and therefore the Select-All fan-out)
at 24.  Without it, a Select-All on a busy cluster would pin the
console proxy pool with hundreds of parallel deletes through the
fan-out router.  While in delete mode the saved-coordinators list is
frozen against SSE re-renders so visible cards don't shuffle out from
under the user's selections (drained on cancel / post-delete close).

Refactor: shared logic now lives in turnstone/shared_static/cards.{css,js}.

  * .ws-delete-* CSS moved out of ui/static/style.css into the shared
    sheet alongside .dashboard-card; the existing ui/static modal
    markup picks up class hooks instead of id-scoped rules.
  * createSavedCardsController() owns mode state, checkbox decoration,
    toolbar wiring, focus trap, modal lifecycle, and batch fan-out.
    Both ui/static (Saved Workstreams) and console/static (Saved
    Coordinators) instantiate one controller; ui/static is now ~300
    LOC lighter as a result.
  * Internalises stale-selection prune across SSE re-renders, the
    wsId->item lookup map (was O(selected x N)), and the aria-hidden
    wrap on the toggle button's emoji glyph.

Designer review tightened the affordance:

  * Modal close restores focus to the toggle button (was landing on
    <body>) — WCAG 2.4.3.
  * Modal [role="alert"] gets a red-chip treatment when populated,
    stays invisible at rest via :not(:empty).
  * Pagination consolidated onto the existing .pagination control
    (terse "X / Y" label + arrow-glyph buttons) instead of a parallel
    .coord-pagination treatment.
  * Filled destructive buttons darkened to #dc2626 in dark theme so
    the white label clears WCAG AA contrast (was 3.0:1 on --red).
    Light theme keeps --red unchanged (5.9:1 already passes).
  * Toolbar wraps below 700px viewport — Delete Selected drops to its
    own full-width row underneath count + Cancel + Select All for
    thumb-target separation.
  * .ws-card-check:focus-visible outline + word-break on
    .ws-delete-item for narrow-modal long aliases.

* fix(cards): address Copilot review feedback on PR #458

* closeModal focus restore now falls back to the section toggle button
  (opts.buttonId) when prevFocus is hidden or detached.  The post-delete
  Close path runs cancel() before closeModal(), which puts the bar at
  display:none — so the captured prevFocus (the bar's "Delete Selected"
  button) is no longer focusable and focus would land on <body>,
  defeating the WCAG 2.4.3 fix.  Esc / Cancel paths still land on the
  original focus owner because the bar stays visible in those flows.

* Saved Coordinators onClose drains _savedCoordsRetry before reloading.
  Without it, SSE events that arrived during the delete-mode freeze
  leave the retry flag true, so loadSavedCoordinators's .finally()
  re-fires a second fetch immediately after the first resolves.  Mirrors
  the same idiom in cancelCoordDeleteMode.
2026-04-30 23:48:53 -07:00
Patrick Buckley ac1fd67137 chore: bump version to 1.5.3 2026-04-30 13:34:35 -07:00
Patrick Buckley 1b40ae79f9 fix(storage): address PR #457 review feedback
Three issues from the Copilot review on PR #457:

1. SQLite race in bulk_close_stale_orphans (Copilot): the SELECT-then-
   UPDATE flow doesn't re-apply the eligibility predicates on the
   UPDATE, so a row that gets touch_workstream-bumped (or set_state-
   transitioned) between the two statements would still be flipped
   to closed.  Postgres dodges this via UPDATE...RETURNING (one atomic
   statement); SQLite needs the explicit re-application.  Fix: rebuild
   the WHERE conditions list once, apply on both SELECT and UPDATE,
   then SELECT-back by ``state='closed' AND updated=now`` to get the
   accurate closed-id list.  A row that became fresh between the two
   statements skips the UPDATE entirely.

2. SQLite IN-clause bind-parameter limit (Copilot): default 999 cap
   could be exceeded on a backlog reap (e.g. after a long outage).
   Chunked the candidate id list at 500 — same chunk size
   prune_workstreams (line 453) uses for the same reason.

3. Wall-clock-dependent test asserts (Copilot, two locations): the
   tests asserted ``updated > '2024-01-01T00:00:00'`` which is fragile
   on systems with skewed clocks or pre-2024 dates.  Replaced with
   ``updated != stale_seed`` — captures the same intent (the value
   was bumped) without depending on wall-clock date.

Two ``...``-as-no-op flags from github-code-quality were false
positives — ``...`` is the standard Python idiom for Protocol method
bodies and matches every other method in _protocol.py.  No code change.
2026-04-30 13:34:15 -07:00
Patrick Buckley b078ddccf0 fix(session_manager): scope orphan reaper by services.last_heartbeat
Replaces the ``node_id == self_node_id`` orphan-scoping heuristic from
earlier on this branch with liveness-based scoping using
``services.last_heartbeat``.  The heuristic was wrong for the post-#384
world: PR #384 (refactor: replace hash-ring rebalancer with rendezvous
hashing) deleted the rebalancer that used to keep workstreams.node_id
pointing at a live node.  Without it, ``workstreams.node_id`` is now
stamped at create time and never updated, so in containerized
deployments with dynamic hostnames a dead pod's rows have ``node_id``
matching no surviving service — they'd accumulate forever under the old
heuristic.

services.last_heartbeat is the same primitive the rendezvous router
uses for routing.  Reusing it here keeps reap scoping aligned with
routing: dead pods' rows fall out of the live set after the heartbeat
window and become reapable; alive pods' rows stay protected as long as
they heartbeat.

Mechanics:

- ``bulk_close_stale_orphans`` parameter renamed
  ``node_id: str | None`` → ``live_node_ids: list[str] | None``.  The
  WHERE clause becomes ``(node_id IS NULL OR node_id NOT IN
  live_node_ids)``.  ``None`` skips the filter entirely (single-process
  / tests / operator backfill).  ``[]`` treats every row as
  unprotected.
- ``SessionManager.close_idle`` pass 2 calls
  ``storage.list_services(self._service_type)`` to enumerate live
  peers, passes their service_ids as ``live_node_ids``.  ``_service_type``
  is derived from ``self.kind`` (INTERACTIVE→"server",
  COORDINATOR→"console") via a module-level mapping — no constructor
  param, so production wiring can't miswire the kind/service_type
  pairing.
- list_services failure → pass 2 is skipped this tick (conservative;
  never reap when liveness state is unknown).  Pass 1 still runs.
- ``workstreams.node_id`` with NULL value is always eligible — defends
  against ANSI ``NULL NOT IN (...)`` evaluating to NULL (not TRUE) and
  silently protecting orphans forever.
- Migration 048 simplified to ``(kind, updated)``; the new query's
  ``NOT IN (small list)`` predicate against an unbounded-cardinality
  column doesn't index well, so leading ``node_id`` would just add
  write cost.

Tests cover the live-services protection (own/dead/null cases), the
empty-peers reap-all case, the list_services-failure conservative
fallback, both kind/service_type pairings (interactive→"server",
coordinator→"console"), and the combined live_node_ids +
exclude_ws_ids filter matrix.
2026-04-30 13:34:15 -07:00
Patrick Buckley 4b6c93a0e9 perf(storage): partial composite index for the orphan reaper query
bulk_close_stale_orphans runs every min(300s, idle_timeout/4) on
every server and console process.  Its WHERE shape is:

    WHERE kind = ?
      AND state IN ('idle','thinking','attention','running')
      AND updated < ?
      AND node_id = ?           -- multi-node interactive only

At current scale the existing single-column indexes are sufficient —
idx_workstreams_state prunes to non-closed and the planner filters the
rest sequentially.  At 100k+ rows that filter becomes a tablescan-
shaped cost.

A partial index covering only BULK_CLOSE_STATE_VALUES rows matches the
reaper's query exactly while staying tiny — closed rows (typically
95%+ of the table) and error rows are excluded, so the index is
roughly 5% the size a full multi-column index would be.  Write
amplification only kicks in for transitions touching one of the four
covered states.

Column order (node_id, kind, updated): node_id is the most selective
filter for multi-node interactive (each server prunes to its own
node's rows), kind second so coord-only and interactive-only queries
within a node still get index-only scans, updated last so the range
comparison rides the trailing column.

Postgres uses CREATE INDEX CONCURRENTLY so the build is non-blocking
on a live system; SQLite has no concurrent concept and the table-
level write lock already serializes, so a plain CREATE INDEX is fine.
2026-04-30 13:34:15 -07:00
Patrick Buckley 9d283e951f fix(console): periodic idle cleanup for the coordinator pool
The console's coord SessionManager had no idle thread — close_idle was
never called for coordinator workstreams.  This is the worse half of
the lifecycle leak: the dashboard filters via the in-memory pool, so
DB-only orphan coords were invisible.  At empirical diagnosis,
coord closure was 16% (10 closed / 64 total) vs interactive 63%.

Adds _coord_idle_cleanup_thread mirroring turnstone/server.py's
_idle_cleanup_thread but skipping the rate-limiter / global-queue arms
the console doesn't have.  Started from the lifespan when coord_mgr is
constructed and server.workstream_idle_timeout > 0 (reuses the
existing setting — same cadence works for both kinds).

Initial sweep runs INSIDE the thread before the first sleep, not
synchronously in the lifespan: cold-start orphans are reaped without
blocking Starlette boot.  Important because cold start with many DB
orphans (the precise condition this code targets) is exactly when the
UPDATE is most likely to be slow.

Helper takes an optional stop_event parameter purely for tests —
production callers pass None and the daemon runs for process lifetime.
This avoids the SystemExit-from-stub + module-wide filterwarnings
fragility a previous iteration relied on.

Four tests: initial sweep runs before first sleep, ticks fire each
loop, exceptions don't kill the thread, stop_event exits cleanly.
2026-04-30 13:34:15 -07:00
Patrick Buckley 4e407e7d4f fix(session_manager): close DB-orphan workstreams in close_idle
Real bug: workstream rows accumulate in non-closed states (idle,
thinking, attention, running) when their owning process restarts or
crashes.  Empirical diagnosis on a live deployment found ~60 stuck
coord rows in DB invisible to the in-memory-keyed dashboard, plus
100+ interactive rows older than the 2h timeout (one stuck "thinking"
for 2 weeks — impossible across a process restart).

Root cause: close_idle iterates self._workstreams.values() — only the
loaded subset.  Anything left behind by a prior process incarnation
sits in DB forever because nothing ever re-loads it.

This commit gives close_idle a second pass.

Pass 1 (existing, unchanged): close loaded IDLE rows whose
ws.last_active (monotonic) is past timeout.  IDLE-only so legitimately-
attentive rows (waiting for user response) stay live.

Pass 2 (new): bulk-close DB rows of this manager's kind whose updated
is past the wall-clock cutoff and which aren't currently loaded.
Closes the broader BULK_CLOSE_STATE_VALUES set — any matching row is
by definition not loaded by any process and cannot be in a live
interaction.  Scoped by self._node_id so a sibling node can't reap
rows we own (multi-node interactive correctness).  No emit_closed —
never-loaded rows have no SSE listeners expecting them.

Lock invariant: pass 1 holds self._lock briefly to snapshot victims
and pop them (existing behavior).  Pass 2 holds self._lock briefly to
snapshot the loaded keys, then releases before the DB UPDATE so a slow
reaper query can't block create/get/set_state.

Also fixes a same-process race in open(): the rehydrate path read DB,
released the manager lock, then re-acquired to install — a concurrent
pass 2 between the two acquisitions snapshots loaded keys without the
in-flight ws_id, and could clobber its DB row to closed.  open() now
calls touch_workstream(ws_id) on rehydrate so the row's updated is
fresh against any pass-2 cutoff.  Pure timestamp write is safe against
concurrent close() (close still wins on the state column).

Three new tests cover the DB orphan pass (basic, exclude-loaded, kind
filter) plus node_id scoping (own/foreign rows, None-skips-filter) and
the open() rehydrate touch.
2026-04-30 13:34:15 -07:00
Patrick Buckley 7ab24e500b fix(storage): add bulk_close_stale_orphans + touch_workstream primitives
Two new methods on the StorageBackend Protocol, with implementations on
both Postgres (UPDATE ... RETURNING) and SQLite (SELECT-then-UPDATE in
one transaction).  No callers yet — wiring lands in subsequent commits.

bulk_close_stale_orphans(kind, cutoff, exclude_ws_ids, node_id=None)
flips rows in BULK_CLOSE_STATE_VALUES (idle/thinking/attention/running)
to closed when their updated timestamp is lex-older than cutoff.  The
node_id filter scopes the reap to a single node's partition — required
for multi-node interactive deployments where each node only has
authority over its own workstreams.node_id rows.  Excludes loaded ids
so the in-memory pass owns those.

touch_workstream(ws_id) bumps updated without changing state.  Used by
the open() rehydrate path to defend against the orphan reaper clobbering
a freshly-loaded row whose DB updated is older than the cutoff.  Pure
timestamp write is safe against concurrent close() because close still
wins on the state column.

BULK_CLOSE_STATE_VALUES is centralized in workstream.py so the two
backend implementations and FakeStorage all agree; if a new transient
state is added to WorkstreamState, deciding whether it joins this set
is part of the change rather than an after-the-fact audit across three
files.

Storage tests (run against both backends via the conftest fixture) cover
the kind/state/cutoff/exclude/node_id matrix plus touch_workstream.
2026-04-30 13:34:15 -07:00
Patrick Buckley 5bcbcb73b9 chore: bump version to 1.5.2 2026-04-30 03:15:59 -07:00
Patrick Buckley af6749421a fix(metacog): drop duplicate [repeat: tool()] info line
The themed ``tool_reminder`` bubble below the tool block already
shows the metacog text, and the tool block immediately above it
carries the tool name — so a separate gray ``[repeat: list_workstreams()
called with same arguments]`` info line was just duplicate visual
noise (operator-visible in the screenshot below the bubble).

Drop the ``ui.on_info`` call inside ``_apply_post_execute_advisories``
that emitted the diagnostic line.  Update the docstring to reflect
that the bubble is the canonical signal.  Rename
``test_emit_repeat_ui_line_on_streak_fire`` →
``test_no_legacy_repeat_info_line_on_streak_fire`` and invert the
assertion.
2026-04-30 03:15:22 -07:00
Patrick Buckley 4d6cb77075 fix(cli): add on_user_reminder + on_tool_reminder to TerminalUI
CI typecheck failed because ``WorkstreamTerminalUI(TerminalUI)``
inherits from ``SessionUI`` (the Protocol), and the Protocol's
``on_user_reminder`` / ``on_tool_reminder`` declarations have empty
bodies — mypy treats those as implicitly abstract, so the subclass
became un-instantiable.

Add real implementations on ``TerminalUI`` that render reminders as
``[metacognition · type] text`` lines in yellow.  This also restores
the metacog signal on the CLI surface (the legacy
``[metacognition: nudge injected — …]`` info-line went away with
``_emit_nudge_ping``; without this commit the CLI showed no signal
at all for metacog nudges).  Tool-channel and user-channel render
identically because terminal output is anchored by stdout flow
rather than by DOM anchor — the line lands directly after the
message it advises.
2026-04-30 03:15:22 -07:00
Patrick Buckley 5c225ef39b docs(metacog): align comments with side-channel + tool-channel scope
Address Copilot's review feedback on PR #456 — the docstrings and
inline comments hadn't all caught up with the architectural shift
across the branch:

  - ``_apply_reminders_for_provider`` docstring: "every user message"
    → role-agnostic, since tool messages also carry ``_reminders``
    (tool_error / repeat).
  - ``_mark_reminders_delivered`` docstring: same role-agnostic
    update; explicitly note both channels.
  - ``_append_user_turn`` callsite comment near
    ``_attach_pending_user_reminders``: still described splicing
    ``<system-reminder>`` blocks into user content; updated to
    reflect the side-channel attach + transient-copy splice at the
    provider boundary.
  - ``_build_history`` block comment: was user-message-only; now
    mentions tool messages and both ``user_reminder`` /
    ``tool_reminder`` SSE events.
  - ``_build_history`` propagation comment: same role-agnostic note
    on the per-entry surface.
  - ``app.js`` ``user_reminder`` SSE handler comment: said the
    bubble renders "above" the user message, but
    ``insertAdjacentElement('afterend', el)`` drops it BELOW.
  - ``app.js`` ``replayHistory`` comment: said "insertBefore drops
    the reminder directly above the just-rendered user bubble";
    same fix — bubble lands BELOW.

No behaviour change.
2026-04-30 03:15:22 -07:00
Patrick Buckley f5a843f44a fix(metacog): drop write-success-clear so sequential same-call streaks fire
The repeat-detection block in ``_apply_post_execute_advisories`` had
a leftover "clear streak when a write tool succeeded" branch from
when ``RepeatDetector`` tracked cumulative counts.  With the
consecutive-streak semantics introduced earlier in the branch the
branch became:

  1. Redundant — any different (name, args) signature already resets
     the streak via ``RepeatDetector.record``, so an intervening
     read/write naturally breaks the streak.
  2. Actively wrong — the clear runs ONCE at the top of each
     ``_apply_post_execute_advisories`` call, before the per-result
     loop records sigs.  In a single parallel batch
     ``[bash, bash, bash]`` the clear runs once and then three
     ``record`` calls accumulate to count=3 in the same call → fires.
     But across three sequential turns, each turn calls
     ``_apply_post_execute_advisories`` fresh, the clear runs at the
     top of each call, and only one ``record`` per call follows — so
     the count never gets above 1 and the canonical
     "small local model stuck on ``bash('echo test')``" pattern
     never triggered the nudge.

The asymmetry only existed for successful calls — failures don't
satisfy the ``not _tool_error_flags.get(tc["id"])`` predicate, so
the clear didn't fire and sequential failures already worked.  The
fix is to drop the clear entirely; ``RepeatDetector``'s
consecutive-streak semantics handle every case uniformly.

Tests:

  - ``test_successful_write_clears_streak`` →
    ``test_intervening_different_call_resets_streak`` —
    rewords the assertion to reflect the actual mechanism (any
    different sig resets, write-or-otherwise) since "writes clear"
    was the bug, not the contract.
  - ``test_failed_write_does_not_clear_streak`` →
    ``test_sequential_bash_failures_fire_repeat`` — same shape, just
    framing fixed.
  - New ``test_sequential_bash_same_command_fires_repeat`` —
    regression for the bug user hit (three sequential successful
    ``bash('echo test')`` calls now correctly fire the nudge).
2026-04-30 03:15:22 -07:00
Patrick Buckley cf44841624 feat(metacog): themed reminder bubble unifies user + tool channels
The yellow themed reminder card introduced for user-channel nudges
(correction / denial / resume / start / completion) now also fronts
tool-channel nudges (tool_error / repeat).  Pre-fix the tool channel
shipped its reminders inside the tool-result envelope via
``wrap_tool_result``, leaking the ``<system-reminder>`` block into
``self.messages`` content (same problem the user channel had before
the side-channel refactor) and surfacing the legacy gray
``[metacognition: nudge injected — …]`` info line as the only
operator-visible signal — duplicated alongside the new themed bubble
for user-channel nudges.

Tool-channel parity:

  - ``_collect_advisories`` now returns
    ``(persistent_advisories, metacog_reminders)``.  Persistent
    advisories (``GuardAdvisory`` / ``UserInterjection``) keep
    riding ``wrap_tool_result`` because they ARE conversation
    history.  Metacognitive reminders extract to the second tuple
    element; the caller attaches them to the tool message dict's
    ``_reminders`` side-channel and emits ``on_tool_reminder``.
  - ``_apply_reminders_for_provider`` already handles ``_reminders``
    on any role, so the tool-channel splice into wire content is
    free.  ``_build_history`` also already propagates
    ``entry["reminders"]`` regardless of role, so reload renders the
    bubble too.
  - ``SessionUI`` Protocol gains ``on_tool_reminder(reminders,
    tool_call_id)``; ``SessionUIBase`` enqueues a ``tool_reminder``
    SSE event with the ``tool_call_id`` anchor.
  - ``_emit_nudge_ping`` had no remaining callers and was removed —
    the themed bubble (live SSE + ``/history`` reload) is the
    canonical operator signal for both channels now.

UI polish (the four fixes the screenshot caught for the user
channel + their tool-channel mirror):

  - Bubble renders BELOW the message it advises (semantically: a
    hint to the model right before its turn).  ``addUserReminder``
    swaps ``insertBefore`` for ``insertAdjacentElement('afterend',
    el)``; ``addToolReminder`` anchors below the ``.ts-approval``
    block whose tool result triggered the batch's reminder.
  - Label uses the full feature name ``metacognition`` (was the
    ``metacog`` shorthand).
  - Card width / alignment inherits from the base ``.msg`` rule —
    ``align-self: flex-end`` and the explicit ``max-width`` are
    gone, so the card matches the user / assistant column instead
    of pinning right-aligned narrow.
  - The legacy ``[metacognition: nudge injected — …]`` gray info
    line is gone for both channels.

Frontend additions:

  - ``Pane.prototype.addToolReminder(reminders, toolCallId)``
    anchors below the ``.ts-approval`` block (live: by
    ``data-call-id``; replay: by "last block in messagesEl"
    fallback, which is correct because messages render in order).
  - SSE switch case ``"tool_reminder"`` calls ``addToolReminder``.
  - ``replayHistory``'s tool-message branch now calls
    ``addToolReminder`` when ``msg.reminders`` is present.
  - ``addUserReminder`` advances its anchor on each loop iteration
    so multiple reminders stack in queued order rather than
    reversed.

Coord console parity:

  - ``coordinator.js`` gains ``appendReminderBubble`` /
    ``appendUserReminderLive`` / ``appendToolReminderLive`` mirroring
    the interactive UI.  The tool-channel anchor walks
    ``toolRows[callId].batch`` to attach below the
    ``.coord-tool-batch`` construct (one bubble per dispatch turn,
    matching the "one nudge per batch even with many failing tools"
    drain).
  - SSE switch handles ``user_reminder`` and ``tool_reminder`` on
    the coord conversation surface.
  - ``/history`` replay propagates ``msg.reminders`` for user and
    tool messages — same wire shape as the interactive pane.
  - ``.msg.user-reminder`` styles moved to
    ``shared_static/chat.css`` so both surfaces inherit the same
    yellow themed bubble from the shared base.

Defensive read on ``_apply_reminders_for_provider`` (per Copilot
review on the closed PR): a malformed ``_reminders`` entry (string,
None, etc. — corruption / partial state) used to abort ``send`` via
AttributeError on the ``.get("text", "")`` call.  Filter to dicts
before building the block, mirroring the same filter
``_build_history`` already applies on the wire-out side; an
all-malformed list passes through as no-reminders.

Tests:

  - ``test_collect_advisories_drains_tool_buffer_on_last_result``
    rewritten to assert the ``(persistent, metacog)`` tuple shape
    and that ``MetacognitiveAdvisory`` no longer appears in the
    persistent list.
  - ``test_collect_advisories_holds_*`` and ``_drops_*`` updated for
    tuple return.
  - ``test_attach_emits_visibility_ping`` /
    ``test_collect_advisories_emits_visibility_ping`` inverted to
    assert the legacy gray line is gone on both channels.
  - ``TestSessionUIBaseToolReminderHook`` covers the new SSE event
    shape with the ``tool_call_id`` anchor.
  - ``test_malformed_reminders_filtered_out`` and
    ``test_all_malformed_reminders_passes_through`` cover the
    Copilot-flagged defensive filter.
2026-04-30 03:15:22 -07:00
Patrick Buckley 7ffab6a272 fix(session): metacog reminders ride a side-channel, not user content
User-channel metacognitive nudges (correction, denial, resume, start,
completion) used to be spliced into ``user_msg["content"]`` permanently,
which leaked the ``<system-reminder>`` envelope into every consumer of
``self.messages`` — UI replay (mitigated by a regex strip in /history),
compaction, title generation, and any future channel adapter that
echoes conversation context.  The /history strip was a band-aid;
compaction and title-gen still saw the raw spliced text.

Switch to a side-channel: ``_attach_pending_user_reminders`` writes the
rendered reminder list to ``user_msg["_reminders"]`` (sibling key,
leading-underscore convention shared with ``_attachments_meta`` /
``_provider_content``).  At the provider boundary, a new
``_apply_reminders_for_provider`` builds a transient shallow-copy with
the reminder spliced into ``content``; the original message dict
stays clean.  ``sanitize_messages`` drops the sibling key on the wire.

Once-per-session-not-per-turn semantics for the wire: after stream
success the loop calls ``_mark_reminders_delivered``, which flips a
``_reminders_delivered`` flag on every user message that carried
reminders into that call.  ``_apply_reminders_for_provider`` skips
already-delivered messages so the model sees each reminder exactly
once (the turn it advised).  ``_build_history`` ignores the delivered
flag entirely, so reconnecting tabs render the same nudge bubble the
originating tab saw via the live ``user_reminder`` SSE event.

UI surface:

  - ``SessionUIBase.on_user_reminder`` enqueues a
    ``{type: "user_reminder", reminders: [...]}`` SSE event with the
    same shape ``_build_history`` surfaces.
  - ``app.js`` renders a ``.msg.user-reminder`` bubble (yellow accent,
    pill-styled) anchored above the user message it advises, both
    live and on history replay.
  - ``replayHistory`` renders ``addUserMessage`` before
    ``addUserReminder`` so the anchor lookup finds the just-rendered
    turn (not a prior one).
  - Multi-tab caveat documented inline: non-originating tabs receive
    no ``user_message`` SSE event today, so a reminder may anchor to
    a stale prior bubble until ``/history`` reload corrects it.

Pre-existing bug surfaced by the audit: cancel handlers
(``GenerationCancelled`` / ``KeyboardInterrupt`` / generic
``Exception``) in ``ChatSession.send`` cleared
``_pending_tool_advisories`` but not the user-channel buffer.  Both
now drain through a shared ``_drain_pending_advisories`` helper.

Removed the ``/history`` regex strip — the side-channel approach
makes it redundant.  Hoisted ``escape_wrapper_tags`` +
``render_system_reminder`` imports to module top (called 2-3× per
turn).

Tests:

  - ``TestApplyRemindersForProvider`` — pass-through-by-reference,
    string + list content splice, escape on user-typed wrapper tags,
    multi-reminder ordering, source-untouched invariant, delivered
    flag skip path, fallback for unexpected content shape.
  - ``TestMarkRemindersDelivered`` — flag idempotency, no-reminders
    no-flag, only marks user messages with reminders.
  - ``TestUpdateTokenTableMsgsParam`` — calibration uses pre-built
    msgs when provided, falls back when not.
  - ``TestUserAdvisoryCancelClear`` — all three cancel branches drain
    the user buffer.
  - ``TestReminderSidechannelIsolation`` — compaction's
    ``_format_messages_for_summary`` and the title-gen extraction
    loop cannot see reminders by construction.
  - ``TestSessionUIBaseUserReminderHook`` — ``on_user_reminder``
    enqueues the right SSE shape.
  - ``TestBuildHistoryReminderPropagation`` — ``entry["reminders"]``
    propagation, absent / empty / multi / coexist-with-attachments
    cases, malformed input filtering, all-malformed elision.
  - ``test_sanitize_messages_strips_underscore_sibling_keys`` covers
    ``_reminders`` and ``_reminders_delivered``.
2026-04-30 03:15:22 -07:00
Patrick Buckley ba3bc9d989 fix(metacog): N>=3 streak detector + drop redundant error-prefix list
Cleanup pass on the metacognitive nudge stack — restores pre-split
errored-counts-toward-repeat behaviour and tightens the is_error
plumbing through the per-batch advisory hook.

The per-batch hook in ``_run_loop`` was duplicating the is_error
signal: ``self._tool_error_flags`` (set by ``_report_tool_result``)
and a string-prefix tuple (``Error`` / ``JSON parse error`` / …).
Two truth sources is what got us here — bash commands that exit
non-zero with normal stdout matched the flag but not the prefix,
the deny path matched the prefix but not the flag, and the result
was that stuck-loop detection silently broke for the most common
failure mode (the model bashing the same broken command).

Single source of truth now:

- ``_execute_tools.run_one`` deny branch routes through
  ``_report_tool_result(is_error=True)`` so denied calls populate
  ``_tool_error_flags`` like every other error path.
- The error-prefix tuple is gone; the write-success-clear gate and
  the tool-error-nudge gate both read ``_tool_error_flags`` only.

Repeat-detection state moves from a ``set[str]`` (fired on the second
identical call, ignored errors entirely) to a ``RepeatDetector``
helper in ``metacognition.py`` with consecutive-streak semantics:

- Threshold raised from 2 to 3 — two-in-a-row was noisy on
  legitimate transient retries; three is the cheapest stuck-loop
  signal.
- Recording a different signature resets the count, so [A, A, B, A]
  is two short streaks of 2 and not a streak of 4. Bounded by O(1)
  state regardless of session length.
- Errored calls now count toward the streak (the split into a
  separate metacog module unintentionally introduced a "skip errors"
  branch — restored).

While there:

- ``metacognition._COOLDOWN_SECS`` default aligned to 300s (matches
  ``MemoryConfig.nudge_cooldown`` and the ``memory.nudge_cooldown``
  config-store default; was set to 30 by an earlier investigation).
- The per-batch advisory block (~80 lines of mixed orchestration
  inside ``_run_loop``) is extracted to
  ``ChatSession._apply_post_execute_advisories`` so the wired
  behaviour is testable without driving ``_run_loop`` end-to-end.
  Producer extraction to a dedicated module is deferred to a
  follow-up; advisory producers all live on ``ChatSession`` for
  now per existing convention.
- Frontend ``appendToolOutput`` (turnstone/ui/static/app.js) now
  skips rendering when the parent approval block is denied or
  the output starts with ``Denied by user`` / ``Blocked``,
  mirroring the history-replay guard at ``_build_history``.
  Previously the live SSE path didn't need this guard because
  the deny path never emitted a ``tool_result`` event; the
  is_error routing change above means it does now, so without
  this guard the badge from ``resolveApproval`` and the SSE
  output would both render.

Tests: 8 unit tests for ``RepeatDetector`` covering streak,
threshold, clear, and intervening-sig reset; 9 integration tests
for ``_apply_post_execute_advisories`` covering the wired
behaviour (3-identical fires warning + advisory + UI line, errored
calls count toward streak as a regression guard, intervening sig
resets streak, successful write clears, failed write does not,
JSON outputs tracked but not inline-warned, tool_error nudge gates
on memory_count, repeat UI line emitted on streak fire).
2026-04-30 03:15:22 -07:00
Patrick Buckley dbe023b4dd chore: bump version to 1.5.1 2026-04-29 20:21:12 -07:00
Patrick Buckley 3d3a8b7367 docs(coord): tighten handleChildState comment per Copilot review
The pre-existing comment said pending_approval_detail "rides on
every ws_state event" — that overstated the case.  The node-side
emit is gated on ``_pending_approval is not None`` so the field is
absent on the steady-state broadcast and possibly null on a node
mid-rolling-upgrade.  The handleChildState fallback already
handles both cases; only the comment was wrong.
2026-04-29 20:20:38 -07:00
Patrick Buckley 99eff73a97 feat(coord): pass pending_approval_detail on child_ws_state SSE events
Inline child approve/deny in the coord tree UI was rendering downstream
of the bulk-live cache (``GET /v1/api/cluster/ws/live``), not the SSE
stream. ``child_ws_state`` events were tiny notifications that fired
an urgent live-bulk fetch on every activity_state transition into/out
of "approval", just to pick up the rich ``pending_approval_detail``
payload. With multiple coord tabs and multi-child workstreams, that
urgent-fetch pattern compounded the SSE-executor pressure Shape A
is unwinding.

Thread the field through every layer so the SSE event itself carries
the rich payload — browser mutates ``liveBadgeCache`` directly,
no urgent fetch:

  1. Node ``WebUI._broadcast_state`` emits ``pending_approval_detail``
     on ``ws_state`` events. Gated on ``_pending_approval is not None``
     so the per-broadcast verdict-cache deepcopy only runs when there
     is actually an approval pending. ``_build_node_snapshot`` also
     projects the field so the console's reconnect-via-snapshot
     resync path delivers it (without this the new collector
     forwarding would never see the field on a snapshot row).

  2. Console ``ClusterCollector._apply_delta`` (live ``ws_state``
     forwarding) and ``_reconcile_node`` (snapshot resync diff) both
     forward the field on the emitted ``cluster_state`` event, AND
     ``_apply_delta`` persists it on the cached ``ws`` dict so the
     ``get_node_detail`` / ``get_snapshot`` endpoints between
     reconciliations don't render stale approve/deny buttons.

  3. ``CoordinatorAdapter._dispatch_child_event`` re-emits the field
     on the ``child_ws_state`` event sent to coord listener queues.

  4. Frontend ``handleChildState`` reads ``ev.pending_approval_detail``
     and writes it directly into ``liveBadgeCache``, tagging the
     entry with ``sseUpdatedAt``. ``flushLiveFetches`` honors that
     tag for ``SSE_AUTHORITATIVE_MS`` (3s) — the upstream
     ``/dashboard`` cache has its own ~2s TTL, so a bulk-poll
     landing right after a transition can otherwise clobber the
     fresh SSE-set state with pre-transition data.

The pre-fix ``enteredApproval`` / ``leftApproval`` urgent-fetch
branch is removed. The 409 stale-call_id retry path keeps its own
urgent fetch — that's a different scenario.

Tests cover the forwarding contract at every layer, the broadcast
gate (event includes the field when an approval is pending,
omits it otherwise, and clears after resolution), and the
``flushLiveFetches`` merge-guard structural shape so a refactor
that keeps the symbols but inverts the comparison or drops the
``prev.live`` check can't pass silently.
2026-04-29 20:20:38 -07:00
Patrick Buckley 99fcd30299 fix(console): offload sync DB calls in coord children/tasks handlers
``coordinator_children`` was calling ``storage.list_workstreams``
directly on the event loop, ``coordinator_tasks`` did the same with
``load_task_envelope``, and ``_resolve_coordinator_or_404`` (called
from both handlers, plus ``coordinator_history`` and
``_resolve_coord_session``) did the same with
``storage.get_workstream`` on its cold-cache path.

The cold-cache resolver path is hit on every console restart,
coordinator eviction, and console proxy hop — exactly when the
event loop is most contended. Three coord tabs reconnecting after a
brief network blip = three serial event-loop blocks per call site.
Other lifted handlers in this file already use
``asyncio.to_thread``; bring all four call sites onto the same
pattern.

Convert ``_resolve_coordinator_or_404`` to ``async def`` and update
its four call sites to ``await``. Exception flow is unchanged.
2026-04-29 20:20:38 -07:00
Patrick Buckley 423c2e80b7 fix(console): isolate coord SSE polling on a dedicated 200-thread pool
Each coord ``events`` SSE listener parks a thread on
``client_queue.get(timeout=5)`` for the connection lifetime. The
console's coord endpoint was wiring no ``sse_executor_lookup`` on
``coord_endpoint_config``, so those parks landed on Python's default
ThreadPoolExecutor (~min(32, cpu_count+4)) and competed with every
other ``asyncio.to_thread`` caller (storage, router, audit). A few
coord tabs against a multi-child workstream would stall new request
handlers waiting for a worker thread.

Mirror the interactive-side precedent (the ``sse_executor`` /
``sse_executor_lookup`` pattern in ``turnstone/server.py``) — build a
dedicated 200-thread ``coord_sse_executor`` in the console lifespan
and wire ``sse_executor_lookup`` onto ``coord_endpoint_config``.
Drain order matters: shut the pool down AFTER ``coord_adapter.shutdown()``
so no new listeners arrive at a dying pool. ``cancel_futures=True``
discards queued-but-not-started futures during teardown.

Update the stale comment on the interactive-side wiring that claimed
"coord wires None and falls back to the default executor" — it now
points at the console's matching wire.
2026-04-29 20:20:38 -07:00
Patrick Buckley a0eb77360d fix(coord): tighten coord_registry refresh logging + comments per round-2 review
Three follow-ups from Copilot's round-2 review on #453.

ValueError logging surfaced the wrong reason
The catch-all ``except ValueError:`` logged ``reason=no_enabled_rows``
unconditionally, but ``ModelRegistry.__init__`` raises ValueError for
five distinct config issues (empty models, default / fallback / agent /
plan / task alias not present).  Operator looking at logs for a
config.toml typo would see the wrong cause.  Switch to
``log.warning("...reason=%s", exc)`` so the actual error message
threads through.  Behavior unchanged — existing registry still
preserved on every ValueError path.

Misleading shutdown() comment
The ``finally`` comment claimed shutdown() was closing clients the
throwaway registry created during DB load.  ``load_model_registry`` only
constructs ModelConfigs and the bare ``ModelRegistry(...)``;
``ModelRegistry.__init__`` leaves ``_clients`` / ``_providers`` empty
and they populate lazily on first resolve.  Today shutdown() iterates
empty dicts.  Comment now says so explicitly while keeping the call
(and its try/except) for forward-compat against an eager-init future.

Stale "probe" wording in test docstring
``test_helper_preserves_registry_when_db_probe_fails`` →
``test_helper_preserves_registry_when_strict_load_fails``.  The
explicit probe was removed in commit 1ba17ed when the helper switched
to ``load_model_registry(..., strict=True)``; the test name and
docstring still talked about a probe.  Updated wording reflects that
the loader's strict-mode re-raise is what the helper catches now.

132 tests pass.
2026-04-29 20:20:38 -07:00
Patrick Buckley 3dd0e196fe refactor(coord): hygiene pass on coord_registry refresh — async + selective teardown + test cleanup
Hygiene follow-ups from the multi-stage code review on #453.

perf-1 — sync helper called from async route handlers
``_refresh_coord_registry`` runs two sync DB reads and a registry reload
that takes ``_client_lock``; calling it directly from an async handler
held the event loop for the duration.  All four call sites now
``await asyncio.to_thread(_refresh_coord_registry, ...)``, matching the
pattern from commit ``1f7d6ad`` (offloaded ``tenant_check``).

perf-3 — ModelRegistry.reload() tore down all clients unconditionally
The reload always closed every cached client and provider, even when
the changed fields (``model``, ``temperature``, ``context_window``)
didn't touch the connection target.  Now selective: clients drop only
when alias removed or ``(base_url, api_key, provider)`` differs;
providers drop only when alias removed or ``provider`` string differs.
Keeps connection pools warm across the common admin-edit case where
only metadata changed.  Two new ``test_model_registry`` cases lock the
keep-warm vs drop-on-change behaviour, and the existing
``test_reload_clears_clients`` was updated (it asserted the old
overly-aggressive contract) into
``test_reload_keeps_clients_when_connection_target_unchanged``.

q-5 — helper rename
``_refresh_console_coord_registry`` → ``_refresh_coord_registry``.  The
``console_`` prefix was redundant given the function lives in
``turnstone/console/server.py`` and sibling helpers there
(``_notify_nodes_model_reload``, ``_publish_config_change``,
``_collect_model_status``) all omit it.

q-1 — shared test middleware
``tests/test_admin_model_registry_refresh`` now imports the
header-driven ``_AuthMiddleware`` from ``tests/_coord_test_helpers``
and sets default ``X-Test-User`` / ``X-Test-Perms`` headers on the
``TestClient``.  The local hardcoded variant duplicated infrastructure
the helper module exists to centralise.

q-3 — multi-alias test registry
``_make_registry`` extracted a ``_make_config`` helper and gained an
``extras={alias: model}`` param so multi-alias scenarios stop
hand-building ``ModelConfig`` literals.
``test_delete_endpoint_refreshes_registry`` now uses the helper.

310 tests pass across the related coordinator + model surfaces.
2026-04-29 20:20:38 -07:00
Patrick Buckley 19c3db5329 test(coord): lock the empty-body gate with a refresh-call spy
bug-3 / q-2 from the multi-stage review on #453: the previous test
``test_update_endpoint_with_empty_body_does_not_blow_up`` asserted only
that the registry's model name was unchanged after an empty PUT, which
holds whether or not the refresh ran (DB row matches registry → refresh
is idempotent).  A regression that always called
``_refresh_console_coord_registry`` — exactly the gate this test was
meant to lock — would have left the assertion green.

Rename to ``test_update_endpoint_skips_refresh_on_empty_body`` and spy
on the helper via ``monkeypatch.setattr``.  Empty-body PUT must register
zero calls; any future change that drops the ``if updates:`` gate now
fails loudly.
2026-04-29 20:20:38 -07:00
Patrick Buckley 0bea72019e fix(coord): strict-mode loader + guarded shutdown for coord_registry refresh
Two correctness follow-ups from the multi-stage code review on #453.

bug-2 / perf-2 (DB probe was theatre + double scan)
The previous probe defended nothing the loader didn't already swallow
on the next line: ``load_model_registry``'s row-loop catches Exception
internally, so a transient DB error after the probe still degrades to
a config.toml-only registry that ``existing.reload()`` would apply,
silently dropping every DB-sourced alias.  And on the happy path each
CRUD paid for two scans of ``model_definitions``.

Add a ``strict: bool = False`` flag to ``load_model_registry``.  When
strict, the row-loop's except re-raises instead of swallowing.  The
helper passes ``strict=True`` and drops the probe — single DB scan,
real failure isolation, the loader's silent fallback can no longer
mask a partial-result regression.  Default ``strict=False`` so CLI /
lifespan callers keep their boot-with-config-fallback behaviour.

bug-1 (shutdown could escape after a successful reload)
``ModelRegistry.shutdown()`` calls ``client.close()`` unguarded, and the
helper's ``finally`` block ran it outside the try/except.  A raising
close() after a successful ``existing.reload()`` would surface as 500
with the registry already mutated and the audit row already recording
success.  Wrap ``new_registry.shutdown()`` in its own try/except that
matches the helper's belt-and-suspenders error policy elsewhere.

The helper's docstring also drops the obsolete probe paragraph; the
``if existing is None: return`` branch gets a one-line inline comment
about the boot-from-empty case (the multi-paragraph version restated
behaviour the line itself documents).

129 tests pass (test_admin_model_registry_refresh + test_model_registry).
2026-04-29 20:20:38 -07:00
Patrick Buckley b9ff52d582 fix(coord): tighten coord_registry refresh — DB probe + accurate boot-from-empty docstring
Two follow-ups from Copilot review of #453:

1. ``load_model_registry`` swallows storage read errors internally
   (logs + continues with config.toml-only models).  Without a strict
   probe in the helper, a transient DB outage on an admin CRUD would
   apply a truncated registry that drops every DB-sourced alias —
   silently, since the loader returns a non-empty registry built from
   ``[models.*]`` config.toml entries.  Add an explicit
   ``storage.list_model_definitions(enabled_only=True)`` probe before
   the loader call so the failure is visible here and the existing
   registry is preserved on outage.

2. The previous docstring claimed ``admin_model_reload`` "has its own
   boot-from-empty story."  It doesn't — it just calls this helper,
   which no-ops when ``coord_registry`` is None.  When no model rows
   existed at boot, lifespan leaves the entire coord subsystem
   uninitialized (no ``coord_mgr``, no ``coord_adapter``, no
   ``session_factory``), and a console restart remains required after
   the operator adds the first row.  Tighten the docstring to admit
   that limitation rather than overstating the helper's reach.

New test ``test_helper_preserves_registry_when_db_probe_fails``
monkeypatches ``list_model_definitions`` to raise and asserts the
existing registry stays intact.
2026-04-29 20:20:38 -07:00
Patrick Buckley 961f999c93 fix(coord): auto-refresh console coord_registry on model-definition changes
The console builds ``app.state.coord_registry`` once at lifespan startup
and the coordinator session factory closes over that exact instance.
Until now, the model-definition admin endpoints (create/update/delete)
wrote to the DB but never touched the in-process registry — and the
explicit reload button only fanned out to nodes via HTTP, also leaving
the console's own registry stale.

Symptom: an operator who changed the underlying model name behind a
local-LLM alias (same alias, same endpoint) saw the DB row update
immediately, but coordinator sessions kept calling the prior model
name until the console process was restarted.

Fix: a new helper ``_refresh_console_coord_registry`` rebuilds a fresh
ModelRegistry from DB and applies it to ``app.state.coord_registry``
via the existing thread-safe ``ModelRegistry.reload()`` — in-place
mutation preserves object identity so the factory closure keeps
working, and active coord sessions auto-pick up the swap on their
next ``send()`` via ``ChatSession._refresh_model_from_registry``.

Wired into four endpoints in ``console/server.py``:

- ``admin_create_model_definition`` — after the DB write
- ``admin_update_model_definition`` — after the DB write, gated on
  ``if updates:`` so a no-op PUT skips the rebuild
- ``admin_delete_model_definition`` — after the DB write
- ``admin_model_reload`` — between ``_publish_config_change`` and
  ``_notify_nodes_model_reload`` so the console mirrors what the
  reload broadcasts to nodes

Failure isolation: a load or reload error leaves the existing registry
intact (logged + swallowed). Coord stays usable while the operator
investigates; the explicit reload remains the user-facing recovery path.

No node fan-out on CRUD — the explicit reload button continues to gate
cluster-wide HTTP propagation, preserving today's UX semantics on shared
clusters.

Tests in ``tests/test_admin_model_registry_refresh.py`` cover:

- helper-level: rebuild from DB, identity preservation, no-op when
  registry is None, preservation on load failure / no-enabled-rows /
  reload validation error
- endpoint-level: create / update / delete / explicit-reload all
  refresh the registry; an empty PUT skips the rebuild
2026-04-29 20:20:38 -07:00
Patrick Buckley 8bdb916064 fix(coord): raise wait_for_workstream message cap to 10 KiB
Production fan-outs are frequently hitting the 6 KiB per-child cap by
just 1-2 KiB, forcing the coordinator into a follow-up inspect_workstream
round-trip per truncated child to recover the tail. Bumping the cap to
10 KiB absorbs the common overshoot without changing the truncation
semantics — truncated=True still fires for genuinely oversized messages,
and inspect_workstream remains the unbounded follow-up.

Worst-case context impact: a 32-child fan-out at the cap is now ~320 KiB
(was ~192 KiB), still well within commercial model context windows.
Typical fan-outs of 1-5 children land at 10-50 KiB.

LAST_ERROR_MAX_LEN (1 KiB) is unchanged — it's intentionally smaller
than the wait cap so error truncation happens at write time, and
1 KiB still sits well below 10 KiB.

WAIT_MESSAGE_MAX_BYTES is referenced by name (not literal 6144) in the
truncation test, so no test value needs updating.
2026-04-29 20:20:38 -07:00
Patrick Buckley 25fe4e728a fix(coord): make coordinator fan out independent work by default
The coordinator system message was descriptive about parallelism rather
than prescriptive — "while multiple children run in parallel" framed
fan-out as incidental, and "a tasks entry, a child to own it" primed
singular delegation. The spawn_batch example (benchmark A, benchmark B,
prototype the winner) showed dependent work under a fan-out framing,
teaching the wrong shape.

In practice the coordinator failed to decompose enumerable requests
("top stories on HN, Lobsters, /r/programming, …") without explicit
"please fan this out" instructions, on both GPT-5.5 and Claude Opus.

base_coordinator.md
- Replace singular "a tasks entry, a child to own it" with plural
  "enumerate the independent units of work, spawn one child per unit,
  run them in parallel by default. Sequential only when one child's
  output feeds the next."
- Tighten the delegation paragraph.

tools_coordinator.md
- Drop the persona repetition that duplicated base_coordinator.md.
- Drop the prescriptive "## Workflow shape" section (the cost note is
  already in wait_for_workstream's tool description; the edit-X
  redirect is already in the persona).
- Drop "in one approval" / "single approval" mentions to avoid
  surfacing approval mechanics to the model.
- Replace the misleading spawn_batch example with truly independent
  items; drop "(up to 10)" which overstated the cap (it's per-call,
  not global, and is documented in the tool schema).
- Add a course-correction example to send_to_workstream — the pattern
  coordinators most often replace with cancel-and-respawn.
- Drop the read action from the tasks examples to keep the lifecycle
  (add → update → remove) coherent.

Coord system message ~16% shorter (4440 → 3722 chars). Both GPT-5.5
and Claude Opus now naturally decompose the news-board prompt without
explicit fan-out instructions. 29 prompt-composition tests pass.
2026-04-29 20:20:38 -07:00
Robert DeAngelis 0d1a32ff65 fix(server): accept --skip-permissions CLI flag (#450)
The server's --help epilog and compose.yaml both reference
--skip-permissions, but the argparser never defined it, so any
container started with SKIP_PERMISSIONS=1 exited with
"unrecognized arguments: --skip-permissions".

Wire the flag through to app.state.skip_permissions, OR-ing it
with the existing tools.skip_permissions config-store setting so
the stored value still works on its own.
2026-04-29 20:20:38 -07:00
241 changed files with 70314 additions and 5665 deletions
+4 -4
View File
@@ -47,9 +47,9 @@ jobs:
# explicit setup, that suite silently skips if the runner
# image happens not to ship Node, masking regressions in
# the browser-side renderer.
- uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "20"
node-version: "24"
- run: pip install -e ".[test]"
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -q
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
@@ -79,9 +79,9 @@ jobs:
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "20"
node-version: "24"
- run: pip install -e ".[test,postgres]"
- run: pytest tests/ -m "not live" --storage-backend=postgresql -q
env:
+523 -3
View File
@@ -8,13 +8,533 @@ version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
Three release tracks are maintained:
- **`stable/1.0`** — patch-only (`v1.0.x`)
- **`stable/1.3`** — patch-only (`v1.3.x`)
- **`stable/1.4`** — patch-only (`v1.4.x`)
- **`main`** — experimental (`v1.5.0aN`)
- **`stable/1.5`** — patch-only (`v1.5.x`)
- **`main`** — experimental (next major)
## [Unreleased]
## [1.5.15]
### Fixed
- **Admin console blank-page on MCP server rows with consented users** — a
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
delimiters on the bulk-revoke button HTML literal, but the literal embeds
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
into bare tokens, and the resulting `SyntaxError` wiped out every global
in `admin.js``showAdmin` and all other admin entry points became
undefined, so the console UI was non-functional whenever the rendered MCP
server list contained at least one row with `consented_users_count > 0`.
Switch the literal to single-quote delimiters to match the surrounding
block.
## [1.5.14]
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
### Added
- **OAuth-MCP Phase 9 — admin status, deferred-consent persistence, operator
docs** — completes the per-(user, server) OAuth-MCP build-out. The sync pool
dispatchers now upsert into a new `mcp_pending_consent` table on
`mcp_consent_required` / `mcp_insufficient_scope`, so a non-interactive run
(scheduled / channel) that hits an unconsented server surfaces the deferred
prompt to the user on their next dashboard load via the gear-icon badge —
rows are cleared automatically by the OAuth callback handler on consent
completion, or via new DELETE endpoints for manual dismiss. The MCP Servers
admin row gains a `consented_users_count` pill and a two-step-confirm
bulk-revoke button for `auth_type=oauth_user` servers (upstream RFC 7009
revoke is intentionally not attempted in bulk to avoid N synchronous
round-trips against the provider). Operator-facing docs land at
`docs/mcp-oauth.md` and `docs/operations/mcp-oauth-headless.md`.
Introduces forward-only migrations `054_mcp_pending_consent` and
`055_mcp_user_tokens_server_index`.
## [1.5.13]
This release introduces one forward-only schema migration:
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
the dispatcher uses in-process fan-out).
### Added
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
per-channel handlers on a separate dispatch thread. The cluster collector
subscribes to a new `services` channel and reacts to node register /
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
loop; the 60 s loop is retained as the backstop for crash-shaped loss
(NOTIFY only fires on real writes). The storage layer also gains a uniform
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
`[database] listen_url` in `config.toml`) points the dispatcher at a
direct-to-Postgres URL; defaults to the main DB URL when unset.
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
whenever a child state change is dispatched to the UI, and the wait loop
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
storage ~2400 times now wakes only on real state transitions, with ~4× lower
SSE traffic in the quiescent case.
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
route previously emitted only `memory.delete`, so tool-initiated mutations
had no audit footprint). All emissions are best-effort and never break the
tool call itself.
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
an optional `skill=<name>` argument that loads the named skill's content as
the sub-agent's persona in place of the hardcoded identity statement. The
fixed operating-guidance block (one-shot, tool-use over narration,
no follow-up questions) is still layered on top of every persona. High- and
critical-risk skills surface their risk tier in the approval header and
emit a `task_agent.high_risk_skill` warning, matching the existing
session-load gate.
### Fixed
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
back-compat `default` alias auto-synthesised by `load_model_registry`
remained visible to the model even when an operator had configured
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
routed to whichever backend the synthesised alias was attached to at boot
instead of the configured per-role default. The synthesised alias is now
only added when neither the DB nor `[models.*]` populates the registry,
filtered out of the LLM-visible alias list, and explicitly rejected at the
validator chokepoint as defense-in-depth.
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
node labels were re-entering the shape parser and producing
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
affected label forms (`|content|` and `ID[content]`) before the SVG cache
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
highlights code blocks progressively as they stream rather than only after
completion.
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
POST went to `/v1/api/auth/login` and was rewritten to
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
public path, and `proxy_api` would have forwarded the login request to the
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
handlers. The dispatch table is a single `(method, path) → handler` mapping
so the test parametrize list can't drift from the implementation.
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
overlay was covering the entire appbar, hiding the proxy-injected node
picker. The overlay now starts at `top: 48px` and the dashboard's role
downgrades from `dialog+aria-modal` to `region` so the appbar above it
remains reachable. The gear icon converts from a direct settings-panel
click into a dropdown with "MCP connections" and "Logout" (the latter with
`.destructive` styling). The settings-menu keydown handler is now attached
synchronously so `Escape` can't fall through the brief window between the
menu opening and its listeners being installed.
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
`services_notify` trigger lives only in the alembic chain, but the test
fixture creates tables via `metadata.create_all`. The trigger function +
trigger are now declared in `_schema.py` and attached via
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
PostgreSQL dialect, with the same SQL constants imported by migration 053
so there's a single source of truth.
## [1.5.12]
### Added
- **Enriched backend error messages** — provider name and attempted URL are now
included in session error responses, so operators can triage connectivity
failures without enabling debug logging.
### Fixed
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
had no messages remaining after a rewind the history event was skipped,
leaving connected UIs with stale content and blocking edit-and-resend flows.
## [1.5.11]
This release introduces one forward-only schema migration:
`052_model_reasoning_persistence``surface_persisted_reasoning` and
`replay_reasoning_to_model` flag columns on `model_definitions`.
### Added
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
restore) now receive an `in_progress_snapshot` event carrying the buffered
partial response, so the UI can resume rendering the in-flight turn without
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
counter so a reconnecting client can skip events it already saw.
- **Reasoning persistence** (Phases 14) — model reasoning text can now be
persisted to conversation history and optionally replayed to the model on
subsequent turns. Phase 1 persists reasoning text on the history payload.
Phase 2 wires a build-time shape filter and a per-model
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
new per-model capability flags (`surface_persisted_reasoning`,
`replay_reasoning_to_model`) both default `False` on unknown and
local-server models.
- **Console home composer: placeholders + toggle** — the console landing-page
composer now shows context-aware placeholder text and a toggle component for
advanced options; an admin polish pass tightened spacing and focus behaviour
across the form.
### Changed
- **`judge.model` now requires a named alias** — raw provider model IDs on
`judge.model` in config are no longer accepted; the judge must reference an
alias registered in the model registry. The session-provider raw-model
fallback is removed. Existing configs using an unregistered model ID need a
corresponding alias entry.
### Fixed
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
flag for a model that does not declare reasoning-replay support now silently
no-ops instead of forwarding reasoning blocks and triggering a provider error.
- **Coordinator alias resolution unified across placeholder + factory** — a
placeholder coordinator and the real coordinator factory could previously
resolve to different model aliases, producing a visible mismatch in the model
display. Both paths now share the same resolution logic.
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
under-initialised coordinator state no longer 500s when the models endpoint
is hit before the coordinator subsystem is fully bootstrapped.
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
counter that broke `state_change` / `in_progress_snapshot` ordering on
reconnect.
- **Reasoning persistence shape + replay fixes** — per-block
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
synthesised alongside non-reasoning `provider_blocks` so both appear
together in the history payload.
## [1.5.10]
This release introduces one forward-only schema migration:
`051_skill_notify_on_complete_array_default` — backfills
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
### Added
- **Skills unlock action** — operators can unlock an installed skill to allow
local customisation. Once unlocked, the skill's resource content, system
prompt additions, and notify configuration are editable through the admin UI.
Skills shipped as part of a bundle remain locked (read-only) until explicitly
unlocked; the unlock is logged to the audit trail. A lock icon in the
top-right of the Skills detail pane doubles as the unlock trigger.
### Fixed
- **`skills.sh` install endpoint** — the install script was targeting an
endpoint removed in an earlier refactor; switched to `/api/download`.
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
(object) instead of `[]` (array), causing notify configurations to be
rejected at schema validation.
- **Skills admin UI modal errors** — `.is-visible` class used consistently
instead of inline `style.display`; stale error text is cleared on submit;
designer-review lock-icon UX applied.
## [1.5.9]
### Fixed
- **`repair=False` on all display-read `load_messages` call sites** —
passing `repair=True` on display paths was silently mutating the stored
message list, causing divergence between what the UI showed and what the
model received on the next turn.
## [1.5.8]
This release introduces two forward-only schema migrations:
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
`050_conversations_source_and_reminders``_source` and `_reminders` columns
on `conversations`.
### Added
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
configured with a client ID and secret through the admin UI. The full token
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
stored encrypted at rest using a key derived from the JWT secret. The consent
flow runs in-browser via a provider redirect. Rolled out in phases:
- Minimum admin form and OAuth schema (`21663d15`).
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
- SDK 401/403 introspection via httpx response hook (`bde09134`).
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
their OAuth token is permitted to call (`cfc8a6c8`).
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
use of an OAuth-gated server and can revoke consent from their profile;
admins see per-server consent counts in the MCP Servers tab (`61051339`).
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
that delivers at most one nudge per turn, preventing multi-channel pile-ups
that inflate context. Observable changes:
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
through to the system message so the model can reason about recency.
- Coordinator idle-children observer: a coordinator with no in-flight
children for longer than the configured idle threshold receives a nudge.
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
rather than the previous `_watch_pending` list, giving them the same
delivery guarantees and priority handling as other advisories.
- **Structured watch-result card** — the UI renders watch results as a styled
card with a system-nudge marker, distinct from the assistant message body.
On history replay, system-nudge turns are visually distinguished from normal
assistant turns.
- **Side-channel persistence** — `_source` and `_reminders` side-channel
fields are persisted to the `conversations` storage table and restored on
session resume, so metacognitive context survives process restarts. A
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
### Fixed
- **Replay consistency** — queued user messages captured mid-loop are now
persisted and replayed in the correct order on a subsequent `events`
subscription. Coordinator history replay fixed: blank assistant cards and
out-of-order tool results on the coordinator tree no longer occur when the
coordinator has mixed queued + delivered messages.
- **Session reminder preservation on fork + resume** — `_source` and
`_reminders` are carried through workstream fork and restored from storage
on resume.
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
columns; `_source` and `_reminders` now strip NUL bytes on write.
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
now committed atomically on first model add; startup teardown is offloaded
to avoid blocking the event loop.
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
now replaces the pool entry with a fresh session; the carrier token is
owned by the pool entry to prevent a race between the 401 handler and a
concurrent request.
- **OIDC hardening** — multiple security and correctness fixes:
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
callback robustness — typed exceptions, shape checks, log sanitization, JS
race (bug-46, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
stranded-user self-heal on role-mapping failure (cumulative bug-1).
## [1.5.7]
### Added
- **Inline node picker** — a compact node-switcher dropdown in the console
header replaces the "← Back to console" banner, so operators can switch
between nodes without a full navigation.
### Fixed
- **Queued user messages injected mid-loop** — messages queued while a
generation was in progress were not being delivered at the correct seam and
could be dropped or reordered when the worker consumed the queue.
- **Search tool output bounded** — pathological inputs (very long lines with
no whitespace) could produce search results exceeding the context budget.
Output is now clamped before reaching the message.
## [1.5.6]
### Added
- **`api_surface` toggle** — model definitions gain an `api_surface` field
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
the provider client uses. Enables Mistral Medium reasoning via the Responses
surface; Chat Completions remains the default for all other models.
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
includes a `healthy_aliases` list per node, so the coordinator and operators
can see which model aliases are currently reachable without a separate
per-model health probe.
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
configure per-kind reasoning effort and alias overrides from the UI rather
than editing `config.toml`. Live-refresh dropdowns update in place when
model definitions change.
### Fixed
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
query-aware candidate-set selection, dramatically improving recall for
queries whose terms span multiple stored entries.
- **Workstream model + config preserved on rehydrate** — reopening a closed
workstream no longer overwrites the model alias and per-workstream config
with session defaults.
- **Console home composer: attachments + user-message pills** — multipart
attachments in the home composer were not forwarded correctly; user-message
pills in the coordinator chat pane were missing.
## [1.5.5]
### Fixed
- **Saved-workstream tool result rendering** — tool results in closed
workstreams were not rendering on history replay. Audit-trail decoration for
tool calls is now applied on the replay path.
## [1.5.4]
### Added
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
first-class citizens in the cluster event bus. `child_ws_state` events are
pushed through the cluster SSE stream so the console tree view updates in
real time without polling. `list_children` and `get_child` primitives on
`SessionManager` provide a consistent cross-node view of the coordinator's
spawn tree.
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
in the console admin panel now supports checkbox multi-select with a
bulk-delete action.
## [1.5.3]
This release introduces one forward-only schema migration:
`048_workstream_reaper_index` — partial composite index on `workstreams` for
the orphan-reaper query.
### Fixed
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
`close_idle` pass now scopes the DB-orphan reaper by
`services.last_heartbeat` so workstreams belonging to a live node are not
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
storage primitives added; a partial composite index keeps the reaper scan
cheap.
- **Coordinator pool idle cleanup** — a periodic task on the console now
closes coordinator pool entries whose session has gone idle past the
configurable threshold, preventing pool exhaustion on long-running consoles.
## [1.5.2]
### Added
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
nudges are rendered as a distinct styled bubble rather than being injected
inline into the assistant message, making it easier to distinguish model
output from metacognitive annotations. The CLI REPL gains matching
`on_user_reminder` / `on_tool_reminder` callbacks.
### Fixed
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
now fires correctly on the third repetition; a write-success-clear that
reset the counter after a successful tool call (preventing streaks across
mixed-outcome sequences) was removed.
- **Metacog reminders isolated to side-channel** — reminder text no longer
appears in the user content turn; it flows through a dedicated side-channel
the session injects into the system context, preventing the model from
attributing it to the user.
## [1.5.1]
### Added
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
now receive the child's pending approval detail in `child_ws_state` events,
enabling the coordinator to surface approval prompts without a separate poll.
### Fixed
- **Coordinator registry auto-refresh** — the console coordinator registry now
refreshes when model definitions change, so a newly added alias is visible
to coordinators without restarting.
- **Coordinator fan-out default** — coordinators now fan out to independent
child workstreams by default instead of serialising them, matching the
documented contract for parallel-work patterns.
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
summaries and tool results from child workstreams were silently truncated at
the previous 4 KiB cap.
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
polling now runs on a dedicated 200-thread executor, matching interactive's
`sse_executor`, so coordinator long-poll blocking no longer contends with
storage and routing workers on the default pool.
## [1.5.0]
User-visible additions: a unified workstream HTTP surface (interactive and
coordinator under one URL family), inline child approvals, coordinator
composer parity, progressive rendering, OIDC authentication, MCP OAuth
foundations, and a redesigned UI built on the Design System v1 token layer.
This release removes the pre-1.5 body-keyed and query-keyed URL family.
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
This release introduces the following forward-only schema migrations that the
server applies automatically on first startup. All are additive; no data loss.
- `039_workstream_kind``kind` + `parent_ws_id` columns on `workstreams`.
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
`admin.cluster.inspect` to the builtin-admin role.
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
introduced by 039.
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
builtin-admin.
- `043_skill_description_required` — backfills empty `description` rows in
`prompt_templates`.
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
(`interactive` / `coordinator` / `any`).
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status`
`risk_level`.
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
by rendezvous routing in 1.4.
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
rows removed from the coordinator in 1.5.0a4.
### Added
- **Inline child approvals** — pending tool approvals on coordinator child
workstreams surface directly in the coordinator tree view. A risk pill shows
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
buttons appear inline so operators do not need to navigate to the child's
workstream. `pending_approval_detail` is exposed on
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
so all connected clients render approval prompts simultaneously. LLM judge
verdicts are cached client-side and replayed on SSE reconnect.
- **Coordinator composer parity** — the coordinator composer now supports
Stop, Send-to-queue, and Attach (file upload), matching the interactive
workstream composer feature set.
- **Per-call model and judge override on coordinator composer** — operators
can override the model alias and judge model for a single coordinator send
from the composer, without changing the node-wide or role-wide defaults. Bad
aliases return a corrective error listing available choices.
- **Coordinator status bar + richer history replay** — each coordinator
workstream gains a per-coordinator status bar showing active children, token
spend, and generation state. History replay in the coordinator panel is
extended to include tool results and thinking blocks.
- **Coordinator child error surfacing + memory tool** — child workstream
errors are surfaced as distinct error rows in the coordinator tree view
rather than disappearing silently. The coordinator gains access to a
`memory` tool (same interface as interactive) for retrieving stored facts.
- **Coordinator inline tool-batch construct** — the coordinator tool approval
UI replaces the separate approval dock with an inline batch construct that
groups all pending tool calls for a given turn into a single review card.
- **Node capability auto-detection** — nodes report kernel-level capabilities
(available memory, CPU count, accelerator presence) via
`/v1/api/node/capabilities` at startup, enabling the console to filter model
aliases offered to coordinators routing to that node.
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
a `SKILL.md` file's content into the modal auto-populates the name,
description, and configuration fields.
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
soon as a complete diagram block is detected in the stream rather than
waiting for the full response; the diagram re-renders in place as the model
extends it.
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
math delimiters are now recognised alongside the existing `$$` fences.
### Removed (BREAKING — 1.5.0)
- **Legacy body-keyed and query-keyed URL family for the workstream
+6 -3
View File
@@ -8,14 +8,17 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.7 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
# System dependencies: psycopg (libpq5), developer tooling for agent workflows
# System dependencies: psycopg (libpq5), developer tooling for agent workflows.
# ripgrep is the preferred backend for the search tool — natively bounds
# per-line, per-file, and per-filesize so pathological inputs (minified
# bundles, training-data JSONL with multi-MB single records) can't OOM us.
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
libpq5 git curl jq man-db manpages procps file \
libpq5 git curl jq man-db manpages procps file ripgrep \
&& rm -rf /var/lib/apt/lists/*
# Node.js LTS (for npx-based MCP servers like @modelcontextprotocol/server-github)
+46 -1
View File
@@ -281,6 +281,7 @@ Each message in the `messages` array has:
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
Each entry in `tool_calls`:
@@ -325,6 +326,44 @@ finalize any in-progress assistant message.
{"type": "stream_end"}
```
**`state_change`** -- the worker thread transitioned to a new state. Drives
the client's busy-mode (composer in send vs. stop, spinner indicators,
auto-focus on idle). Sent live during normal operation AND on every fresh
SSE subscribe (so a mid-stream page refresh restores the correct composer
state without waiting for the next live transition).
```json
{"type": "state_change", "state": "running"}
```
| Field | Type | Description |
|----------|--------|----------------------------------------------------------------------|
| `state` | string | One of `"running"`, `"thinking"`, `"attention"`, `"idle"`, `"error"` |
**`in_progress_snapshot`** -- one-shot replay of the in-progress turn's
content + reasoning text-so-far when this client connects mid-stream.
Lets a refreshing browser tab restore partial assistant text immediately
instead of waiting for the response to complete. Yielded once after the
kind-specific replay phase (history + pending), only when at least one
of `content` / `reasoning` is non-empty. Both halves render into the same
assistant bubble the live `content` / `reasoning` events would target;
clients should treat the snapshot as idempotent (skip overwrite if the
current local buffer is already a superset prefix — covers EventSource
auto-reconnect re-replays).
```json
{
"type": "in_progress_snapshot",
"content": "Here is the answer so far: it depends on ",
"reasoning": "The user is asking about a comparison; let me think about..."
}
```
| Field | Type | Description |
|--------------|--------|------------------------------------------------------------|
| `content` | string | Joined assistant content text accumulated this turn |
| `reasoning` | string | Joined reasoning / chain-of-thought text accumulated |
**`tool_info`** -- one or more tool calls that were auto-approved (no user
action required).
@@ -522,7 +561,13 @@ Each SSE connection to a workstream receives its own delivery queue. Events
produced by the worker thread are fanned out to all registered listener queues,
so multiple consumers (browser, console proxy, SDK) can connect
simultaneously and each receives every event. On reconnect the client receives
a full history replay, so no catch-up mechanism is needed.
the kind-specific replay (`connected` + `status` + `history` + pending
approval / plan for interactive; `connected` + `status` + pending for coord)
followed by a `state_change` carrying the current worker state and an
optional `in_progress_snapshot` carrying any partial content / reasoning
buffered for the in-progress turn — so a mid-stream refresh restores both
the busy-mode UI and the partial assistant text without waiting for the
response to complete.
---
+50 -11
View File
@@ -231,11 +231,13 @@ The engine emits state changes via `_emit_state()` which calls
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 16
methods. Every frontend must implement all of them.
```python
class SessionUI(Protocol):
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -252,6 +254,14 @@ class SessionUI(Protocol):
def on_rename(self, name: str) -> None: ... # propagate alias to tab/UI label
```
`on_turn_start` fires at the top of each iteration of the send-loop;
`on_turn_committed` fires immediately after `messages.append(assistant_msg)`.
`SessionUIBase` uses both to reset the per-turn inflight buffers
(`_ws_inflight_content` / `_ws_inflight_reasoning` / `_ws_inflight_seq`)
that fuel the SSE refresh-resume `in_progress_snapshot` event — see
the per-workstream events stream in
[`docs/api-reference.md`](api-reference.md#get-v1apiworkstreamsws_idevents).
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
### Three Implementations
@@ -546,11 +556,9 @@ adds, removes, or reconnects servers as needed.
6. `_exec_mcp_tool()` calls `call_tool_sync()` which dispatches to the async loop
via `asyncio.run_coroutine_threadsafe()`
**Tool refresh:** Three mechanisms keep tools up-to-date without restart:
**Tool refresh:** Two mechanisms keep tools up-to-date without restart:
- **Push:** Servers declaring `tools.listChanged` send `ToolListChangedNotification`;
the registered `message_handler` triggers immediate single-server refresh.
- **Periodic:** Servers without push support are polled on a staggered interval
(default 4 h, configurable via `[mcp] refresh_interval` or `--mcp-refresh-interval`).
- **Manual:** `/mcp refresh [server]` calls `refresh_sync()` for on-demand refresh
(also attempts reconnection for disconnected servers).
@@ -572,10 +580,11 @@ from a healthy connection do not trip the breaker. When the cooldown expires
(`call_tool_sync`, `read_resource_sync`, `get_prompt_sync`, `refresh_sync`)
cancel orphaned futures on timeout to prevent coroutine accumulation on the
background event loop. Push notification refreshes are debounced (5 s per
server) to protect against notification storms. The periodic refresh loop
attempts reconnection for disconnected servers with exponential backoff
(60 s1 h). Transport stream references are pre-closed before stack teardown to
work around the MCP SDK's anyio cancel-scope CPU busy-loop (SDK #2147).
server) to protect against notification storms. Operators can force a
catalog refresh or full reconnect from the admin panel; reconnects clear
the circuit breaker and run a fresh handshake. Transport stream references
are pre-closed before stack teardown to work around the MCP SDK's anyio
cancel-scope CPU busy-loop (SDK #2147).
**Error isolation:** Per-server connection/refresh failures are caught and logged; other
servers are unaffected. Tool execution errors return error strings to the LLM
@@ -620,14 +629,15 @@ LLMProvider (protocol)
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
| `retryable_error_names` | Exception class names that trigger retry |
| `extract_reasoning_text()` | Walk stored `provider_blocks`, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic `thinking`, OpenAI Responses `reasoning`, OpenAI Chat synthetic `reasoning_text`) |
**Normalized data types:**
| Type | Fields |
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
@@ -715,6 +725,35 @@ and `"openai-compatible"`.
`max_tokens`, and `reasoning_effort` to override the global defaults from
ConfigStore. When unset (`NULL`), the global default is used.
**Per-model reasoning persistence:** Two booleans on `model_definitions`
(migration 052) control how reasoning text round-trips:
* `surface_persisted_reasoning` (default `True`) — gates whether stored
reasoning text is surfaced on `/history` payloads for UI rehydration.
**Storage of reasoning bytes happens regardless of this flag** — they
ride in `provider_data` independently. Phase-1 admin UI label "Surface
persisted reasoning."
* `replay_reasoning_to_model` (default `False`) — gates whether stored
reasoning blocks are sent back to the provider on subsequent turns.
Capability-gated: `ModelCapabilities.supports_reasoning_replay` must
also be `True` for the wire path to actually replay (canonical OpenAI
gpt-5*/o-series and Anthropic Claude entries set it; unknown / local-
server models default to `False`).
Three reasoning paths are recognised:
| Path | Provider | Capture | Persist | Replay |
|------|----------|---------|---------|--------|
| 1 | Anthropic Messages API | `thinking_delta` | `provider_blocks` (`type="thinking"`) | Verbatim via `_provider_content` |
| 2 | OpenAI Responses (gpt-5*, o-series) | `response.reasoning_text.delta` events | `provider_blocks` (`type="reasoning"`) — only when `include=["reasoning.encrypted_content"]` | `ResponseReasoningItemParam` input items |
| 3 | OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat) | `delta.reasoning_content` Pydantic extras | Synthetic `{type: "reasoning_text", text, source}` block stamped at end-of-stream | None — no API surface for replay on Chat Completions |
Cross-provider safety is enforced by `ANTHROPIC_VALID_BLOCK_TYPES` (a
shape filter in `_anthropic.py:_convert_messages`): foreign blocks
(OpenAI `reasoning`, synthetic `reasoning_text`) fall through to the
text+tool_calls rebuild path rather than reaching Anthropic's input
boundary as malformed content.
```toml
[models.local]
base_url = "http://localhost:8000/v1"
+9 -4
View File
@@ -115,7 +115,8 @@ with a `type` field. The recurring shapes a UI has to handle:
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `rename` | Session's display name changed | `name` |
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
@@ -130,9 +131,13 @@ with a `type` field. The recurring shapes a UI has to handle:
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved) and any in-flight `wait_*` / `batch_*`
indicator — so a tab refresh mid-approval doesn't strand the
operator.
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
---
+1 -1
View File
@@ -40,7 +40,7 @@ package "turnstone/core/" <<Rectangle>> {
component [auth.py\nAuthentication] as auth <<core>>
component [healthcheck.py\nBackendHealthMonitor] as healthcheck <<core>>
component [ratelimit.py\nRateLimiter] as ratelimit <<core>>
component [mcp_client.py\nMCPClientManager\n(push + periodic refresh)] as mcp <<core>>
component [mcp_client.py\nMCPClientManager\n(push + manual refresh)] as mcp <<core>>
component [tool_search.py\nToolSearchManager, BM25] as toolsearch <<core>>
component [model_registry.py\nModelRegistry] as registry <<core>>
}
+5 -3
View File
@@ -69,9 +69,10 @@ class "NullUI" as NullUI {
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ...) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ...) → CompletionResult
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
+ convert_tools(tools) → list[dict]
+ extract_reasoning_text(provider_blocks) → str
+ retryable_error_names: frozenset[str] {property}
--
core/providers/_protocol.py
@@ -126,6 +127,7 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_web_search: bool
+ supports_tool_search: bool
+ supports_vision: bool
+ supports_reasoning_replay: bool
}
' ChatSession
@@ -253,7 +255,7 @@ class "MCPClientManager" as MCPMgr {
Background asyncio event loop
bridges async MCP SDK to
sync ChatSession dispatch.
Push + periodic + manual refresh.
Push + manual refresh.
Resources + prompts discovered
alongside tools at startup.
--
+16
View File
@@ -24,6 +24,14 @@ CS -> DB : save_message(ws_id, "user", input)
group loop [while tool_calls present]
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
@@ -73,6 +81,14 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
+15 -11
View File
@@ -190,21 +190,25 @@ group Push Notifications (debounced 5s per server)
MCPMgr -> Storage : sync_prompts_to_storage()
end
group Periodic Polling (default 4h)
MCPMgr -> MCPMgr : _periodic_refresh()
group Manual Refresh
Session -> MCPMgr : refresh_sync()
note right
Only polls capabilities
without push support.
Staggered per-server.
Disconnected servers get
reconnect attempts with
exponential backoff (60s-1h).
/mcp refresh [server] —
re-fetches catalog and
attempts reconnect for
disconnected servers.
end note
end
group Manual Refresh
Session -> MCPMgr : refresh_sync()
note right: /mcp refresh [server]
group Manual Reconnect
Session -> MCPMgr : reconnect_sync(name)
note right
Operator-driven via the
console admin panel —
tears down session, clears
circuit breaker, runs a
fresh handshake.
end note
end
== Policy Evaluation ==
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a3b5c59403a6febd81667fc8fd2a7d22bc59da6130eba0dea5449c42668d0ede
size 387044
oid sha256:95dd5ebc899a1261d516686a5aa3319a7f45015d411302825fa28afbfc82e1ce
size 326766
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:474b900448ec04d1117b48a2b55614524721b2f04ac4bda66170bd0a06aae0f2
size 624573
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3aa8d972bba40d78152f9f0c762b9f5ec616d8052c45fa52b7dd1c679ed81d61
size 325245
oid sha256:c14dfbb2db8dcb22cd332b2cf0e53ba75141adb213dae47dd9dbfd389ed482fe
size 354799
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7623df33be9baf7647ca1c2450640df57e1cd73e8be1f8168aae16e546ad683c
size 459941
oid sha256:d6aff446a062aa08f316985d00c2183148694f786d7f22172bc50b30046c728b
size 379259
+1
View File
@@ -84,6 +84,7 @@ Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
|----------|---------|-------------|
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_LISTEN_URL` | (falls back to `TURNSTONE_DB_URL`) | Direct-to-PostgreSQL URL for the console's dedicated `LISTEN` connection. Set this when `TURNSTONE_DB_URL` points at PgBouncer in transaction pooling mode — LISTEN is session state and the transaction-pooled connection can't hold it. See [pgbouncer.md](pgbouncer.md). |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
+117
View File
@@ -0,0 +1,117 @@
# MCP OAuth — per-user authorization for MCP servers
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
---
## When to use which `auth_type`
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
| `auth_type` | What it means | When to use |
|---|---|---|
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
## Prerequisites for `auth_type=oauth_user`
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata** *or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
3. **OAuth client registration**. Two paths:
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
---
## Configuration
### Per-server fields (admin UI)
| Field | Required | Description |
|---|---|---|
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
| Multitenant Authorization | Yes | `none` / `static` / `oauth_user` (recommended). |
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
| Client Registration | Yes (oauth_user) | `preregistered` or `dynamic`. |
| Client ID | Yes (preregistered) | OAuth 2.0 client ID. Stored unencrypted. |
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
### Encryption key
```toml
[security]
mcp_token_encryption_key = "base64-fernet-key"
# For rotation, list the keys in priority order — first is used for new
# writes, all are tried for reads.
# mcp_token_encryption_keys = ["new-key", "old-key"]
```
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
---
## Admin status indicators
The MCP Servers admin tab shows per-server status pills (Phase 9):
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
---
## Auth-type transitions
| From | To | What happens |
|---|---|---|
| `none` / `static``oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
+82 -13
View File
@@ -39,18 +39,19 @@ are set.
| `TURNSTONE_OIDC_ROLE_CLAIM` | No | — | ID token claim containing role/group values (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_ROLE_MAP` | No | — | Mapping from claim values to Turnstone role IDs (see [Role Mapping](#role-mapping)) |
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | No | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Recommended when running behind a reverse proxy. When unset, derived from the request Host header. |
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
OIDC is enabled when all three required fields (issuer, client ID, client
secret) are non-empty. If any is missing, OIDC is silently disabled and
the login screen shows only the password form.
All four required fields issuer, client ID, client secret, and
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
is disabled at startup (an error is logged when only `redirect_base`
is missing) and the login screen shows only the password form.
### Reverse Proxy / Load Balancer
### Redirect base (required)
When Turnstone runs behind a reverse proxy, the internal `Host` header may
not match the externally-reachable URL. Set `TURNSTONE_OIDC_REDIRECT_BASE`
to the public origin so the redirect URI sent to the identity provider is
correct:
`TURNSTONE_OIDC_REDIRECT_BASE` pins the redirect URI sent to the identity
provider to a known externally-visible origin. Set it to the public origin
of your Turnstone deployment:
```bash
TURNSTONE_OIDC_REDIRECT_BASE=https://app.example.com
@@ -60,6 +61,44 @@ The resulting callback URL will be
`https://app.example.com/v1/api/auth/oidc/callback` — register this as the
authorized redirect URI in your identity provider.
OIDC will refuse to start when this variable is unset. There is no
Host-header fallback: a permissive reverse proxy or direct backend access
would otherwise let an attacker spoof `Host` and steer the IdP redirect
to a callback origin they control.
### Cross-host endpoints
By default, every endpoint in the IdP discovery document
(`token_endpoint`, `jwks_uri`, `userinfo_endpoint`) must share the
issuer's `(scheme, host, port)`. This prevents a hostile or compromised
IdP from redirecting the token-exchange POST (which carries
`client_secret`) to an arbitrary host, and prevents JWKS fetches from
being aimed at internal services.
A few public IdPs legitimately split endpoints across hostnames. Google
is the canonical example:
| Field | Hostname |
|-------|----------|
| issuer | `accounts.google.com` |
| token_endpoint | `oauth2.googleapis.com` |
| jwks_uri | `www.googleapis.com` |
| userinfo_endpoint | `openidconnect.googleapis.com` |
Google's set is built in — operators using `https://accounts.google.com`
need no extra configuration.
For other IdPs whose discovery document references a non-issuer host,
extend the allow-list explicitly:
```bash
TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS=token.example.com,keys.example.com
```
The same scheme / no-userinfo / SSRF rules apply to allow-listed hosts —
this knob only relaxes the same-origin check, not the security gates.
Each entry is a hostname (no scheme, no path).
### config.toml alternative
```toml
@@ -198,6 +237,19 @@ TURNSTONE_OIDC_ROLE_MAP="admin:builtin-admin,engineering:builtin-operator,viewer
the user authenticates via OIDC, so new group memberships are picked
up on the next login.
### `assigned_by` markers
Role assignments record an `assigned_by` value that controls how the
sync logic treats them. OIDC-driven flows use two distinct markers:
- `oidc` — set by claim-driven role mapping; revoked automatically on
the next login when the corresponding claim value is no longer
present.
- `oidc-default` — applied to brand-new OIDC users who have no
claim-mapped roles, as a safety net so they still get
`builtin-viewer` access on first login. Survives subsequent logins
regardless of claim contents and is never revoked by `apply_role_mapping`.
### Built-in Roles
| Role ID | Permissions |
@@ -375,10 +427,27 @@ callback validation. Entries are automatically cleaned up after 5 minutes.
### "OIDC not configured"
All three required environment variables must be set:
`TURNSTONE_OIDC_ISSUER`, `TURNSTONE_OIDC_CLIENT_ID`, and
`TURNSTONE_OIDC_CLIENT_SECRET`. Check that none are empty or
whitespace-only.
All four required environment variables must be set:
`TURNSTONE_OIDC_ISSUER`, `TURNSTONE_OIDC_CLIENT_ID`,
`TURNSTONE_OIDC_CLIENT_SECRET`, and `TURNSTONE_OIDC_REDIRECT_BASE`.
Check that none are empty or whitespace-only.
### "OIDC enabled but TURNSTONE_OIDC_REDIRECT_BASE is unset"
This error is logged when the three credential variables are set but
`TURNSTONE_OIDC_REDIRECT_BASE` is missing. OIDC is disabled at startup
to prevent Host-header-derived redirect URI spoofing. Set the variable
to your service's externally-visible origin (e.g.
`https://app.example.com`) and restart the server. See
[Redirect base](#redirect-base-required) for the rationale.
### Discovery silently disables OIDC with "host does not match issuer"
The IdP discovery document points `token_endpoint`, `jwks_uri`, or
`userinfo_endpoint` at a hostname that doesn't share the issuer's
origin. If the IdP is legitimate, add the additional hostname(s) to
`TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS`. Google is allow-listed
automatically; see [Cross-host endpoints](#cross-host-endpoints).
### "Login session expired"
+29
View File
@@ -0,0 +1,29 @@
# MCP OAuth in headless / scheduled / channel-driven runs
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
**Affected surfaces**:
- Scheduled workstreams (`turnstone-console` task scheduler).
- Discord adapter runs.
- Slack adapter runs.
- Any future channel adapter without an interactive browser session.
**What happens when consent is missing**:
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
**Recovery**:
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
**Pre-consent recipe**:
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
1. Open the dashboard.
2. Open the settings modal (gear icon).
3. Click Connect on each MCP server the schedule will use.
4. Confirm consent in the popup.
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
+24
View File
@@ -199,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
not use server-side prepared statements by default, so this is not an
issue.
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
transaction pooling assigns a real server connection only for the
duration of each transaction, then returns it to the pool. PostgreSQL
`LISTEN` is session state — a transaction-pooled client can't hold the
multi-statement session a long-lived `LISTEN` needs. The console's
`NotifyDispatcher` (reactive node discovery via the `services` channel)
therefore opens a **dedicated, direct-to-Postgres** connection that
bypasses PgBouncer.
Configure via `config.toml` `[database] listen_url` (preferred —
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
(config.toml wins when both are set). Defaults to the main DB URL when
unset.
| Setting | Behaviour |
|---|---|
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
Set this whenever PgBouncer is in transaction mode (the recommended
setting per this doc). The override only adds one long-lived PG
connection per console process — sized into the cluster's
`max_connections` budget alongside the pool.
See also: [Docker deployment](docker.md) · [Security](security.md)
+3
View File
@@ -138,6 +138,9 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `error` | `ErrorEvent` | `message` |
| `info` | `InfoEvent` | `message` |
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `cancelled` | `CancelledEvent` | — |
**Global events** (from `stream_global_events()`):
+16 -1
View File
@@ -59,6 +59,21 @@ from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Reasoning persistence (per-model)
Two boolean flags on `model_definitions` (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|------|---------|--------|
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
provider-side mechanics (Anthropic `thinking`, OpenAI Responses
`reasoning` + `include=["reasoning.encrypted_content"]`, synthetic
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
### Plan / task agent overrides
`plan_agent` and `task_agent` sub-sessions resolve independently from the
@@ -100,7 +115,7 @@ initialization:
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
| `cluster` | node_fan_out_limit, mcp_max_servers |
| `mcp` | config_path, refresh_interval, registry_url |
| `mcp` | config_path, registry_url |
| `ratelimit` | enabled, requests_per_second, burst, trusted_proxies |
| `health` | backend_probe_interval, backend_probe_timeout, circuit_breaker_threshold, circuit_breaker_cooldown |
| `judge` | enabled, model, provider, base_url, api_key, confidence_threshold, max_context_ratio, timeout, read_only_tools, output_guard, redact_secrets, cancel_on_approval |
@@ -0,0 +1,233 @@
---
name: import-conversation-history
description: Use this skill when the user wants to import or migrate conversation history from another LLM chat or coding tool (e.g. ChatGPT, Claude.ai, Cursor, Copilot Chat, Aider, Gemini, a custom JSON export) into Turnstone. The skill teaches Turnstone's destination contracts — workstream identity, the OpenAI-shaped message rows, tool-call/result pairing, provider-fidelity blobs, attachments, and archive-vs-resumable choice — so the agent can map any source format onto them. Trigger phrases: "import my chats", "migrate this transcript into Turnstone", "bring my Claude.ai history over", "load this export as a workstream".
version: 1.0.0
---
# Importing Conversation History into Turnstone
## Overview
Source formats vary; the destination does not. Your job is to translate whatever the user hands you (JSON dump, ZIP export, scraped HTML, screenshot OCR, raw transcript) into Turnstone's internal shape: **one workstream row** plus an ordered sequence of **conversation rows** in OpenAI message format. This skill documents the destination so you can write a correct mapper for any source.
Two questions to settle with the user before writing anything:
1. **Archive or resumable?** An archive ("saved" workstream — `state="closed"`) is read-only history. A resumable workstream (`state="idle"`) lets the user continue the conversation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. **One workstream per source thread, or merge?** Default to one-to-one unless the user explicitly asks to merge.
Default to **archive** when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.
## Turnstone Data Model (the destination)
Two tables carry the conversation:
### `workstreams` (one row per imported thread)
| Column | Required | Notes |
|---|---|---|
| `ws_id` | yes | 32-char lowercase hex. Auto-generate with `secrets.token_hex(16)` if you don't already have one. **First 4 hex chars are the routing bucket** — see "Identity & Routing" below. |
| `name` | yes | Short title. Pull from source thread title; fall back to first ~60 chars of first user message. |
| `state` | yes | `"closed"` for archive, `"idle"` for resumable. Never set `"running"` on import. |
| `kind` | yes | `"interactive"` for normal threads. Do NOT use `"coordinator"` for imports — that's reserved for cluster-spawned coordinator workstreams. |
| `parent_ws_id` | no | Leave NULL. Only set if you're importing a coordinator-spawned subtree and re-parenting it; rare. |
| `user_id` | yes | Owner. Must exist in `users`; importer must know which Turnstone user owns the imported history. |
| `node_id` | yes (multi-node) | Denormalized cache of the node that owns this `ws_id`'s bucket. Single-node deployments can leave it NULL or set it to the only node. |
| `alias` | no | Human-typeable short name. Optional; must be unique cluster-wide if set. |
| `title` | no | Auto-titled later by the LLM; safe to leave NULL on import. |
| `skill_id`, `skill_version` | yes | Default `""` and `0` unless the source thread was scoped to a Turnstone skill. |
| `created`, `updated` | yes | ISO8601 strings. Use the source's first/last message timestamps when available. |
### `conversations` (many rows per thread, ordered by `id`/`timestamp`)
| Column | Notes |
|---|---|
| `ws_id` | The workstream this row belongs to. |
| `timestamp` | ISO8601 string. Preserve source timestamps; fall back to monotonically increasing values if unknown. **Order is canonical via `id` (autoincrement), not `timestamp`** — but always insert in conversational order so both agree. |
| `role` | One of `system`, `user`, `assistant`, `tool`, `developer`. See role mapping below. |
| `content` | Text. May be NULL for assistant rows that are *only* tool calls. |
| `tool_name` | Set on `role="tool"` rows (the tool whose result this is). NULL otherwise. |
| `tool_call_id` | Set on `role="tool"` rows (matches the assistant row's `tool_calls[].id`). NULL otherwise. |
| `tool_calls` | JSON-encoded list, on `role="assistant"` rows that issued tool calls. OpenAI shape — see "Tool Calls" below. |
| `provider_data` | JSON blob preserving provider-native content blocks (Anthropic `signature`, Gemini `thought_signature`, etc.). Optional; only matters for **resumable** imports against the same provider. Skip for archives. |
The internal format is **OpenAI-shaped**, even when the source was Anthropic or Gemini. Providers translate at their own API boundary; storage stays uniform.
## Identity & Routing (`ws_id`)
- `ws_id` is **32-char lowercase hex** (i.e. `secrets.token_hex(16)`).
- The **routing bucket** is `int(ws_id[:4], 16)` — the first 4 hex chars place this workstream on a specific node via the consistent hash ring.
- For multi-node imports: either insert through the console's routing proxy (which forwards to the owning node), or generate `ws_id`s and write directly to each node's database in batches grouped by bucket.
- For single-node imports: bucket math is irrelevant; any `ws_id` works.
- **Do not reuse the source platform's IDs as `ws_id`** unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in `workstream_config` under a key like `import.source_id`.
## Recommended Import Path
Three options, in order of preference:
### 1. Storage protocol (recommended for full history)
Use `turnstone.core.storage.Storage.save_messages_bulk(rows)`. This is the canonical bulk-insert primitive and bypasses the LLM round-trip entirely.
```python
from turnstone.core.storage import get_storage # construct via the same path the server uses
storage = get_storage(...) # see turnstone.core.storage.__init__ for the project's wiring
storage.create_workstream( # or whatever the project's exposed creator is — check turnstone/core/storage/_protocol.py
ws_id=ws_id,
user_id=user_id,
name=name,
state="closed",
kind="interactive",
...
)
storage.save_messages_bulk([
{"ws_id": ws_id, "role": "user", "content": "Hello"},
{"ws_id": ws_id, "role": "assistant", "content": "Hi! What can I help with?"},
{"ws_id": ws_id, "role": "assistant", "content": None,
"tool_calls": json.dumps([{"id": "call_1", "type": "function",
"function": {"name": "search", "arguments": "{\"q\":\"x\"}"}}])},
{"ws_id": ws_id, "role": "tool", "tool_name": "search", "tool_call_id": "call_1",
"content": "result text"},
# ...
])
```
`save_messages_bulk` handles `timestamp` and the workstream's `updated` column internally, so you don't need to compute them per row. **Verify the exact creator signature** by reading `turnstone/core/storage/_protocol.py` — table layout has shifted across migrations and the Storage protocol is the source of truth.
### 2. SDK `create_workstream(resume_ws=...)` (when the source is already a Turnstone workstream)
Only useful for *Turnstone → Turnstone* re-parenting. Not relevant for foreign sources.
### 3. SDK `create_workstream(initial_message=...)` + `send()` per turn (last resort)
Only fits archives where the source had **no tool calls** and you don't care about preserving assistant turns verbatim. Each `send()` triggers a real LLM round-trip, which is expensive and rewrites assistant content. Don't use this for full history.
## Role Mapping
Common source-role conventions and how they map to Turnstone:
| Source role | Turnstone `role` | Notes |
|---|---|---|
| `user`, `human` | `user` | Direct map. |
| `assistant`, `ai`, `model`, `bot` | `assistant` | Direct map. |
| `system` | `system` | Preserve only if it's content the user wrote (custom instructions). Drop boilerplate provider preambles — Turnstone composes its own system message. |
| `developer` (OpenAI o-series) | `developer` | Preserve. |
| `tool`, `function`, `tool_result` | `tool` | Must carry `tool_name` and `tool_call_id` matching the prior assistant row's `tool_calls[].id`. |
| `tool_use` (Anthropic) | `assistant` with `tool_calls` | Anthropic emits tool calls *inside* an assistant message; flatten to OpenAI shape. |
| `human_feedback`, `revision` | `user` | Treat as a follow-up user turn. |
## Tool Calls (the most error-prone part)
Turnstone stores tool calls in OpenAI's nested-function shape on the assistant row, and matches them with `role="tool"` result rows by `tool_call_id`.
### Assistant row with tool calls
```json
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "search_web",
"arguments": "{\"query\":\"turnstone import\"}"
}
}
]
}
```
`tool_calls[].function.arguments` is **a JSON-encoded string**, not an object. Source formats commonly get this wrong — Anthropic stores arguments as a parsed object, Gemini as a struct. Always re-serialize to a string.
### Tool result row
```json
{
"role": "tool",
"tool_name": "search_web",
"tool_call_id": "call_abc123",
"content": "..."
}
```
Pairing rules:
- Every assistant `tool_calls[].id` MUST be followed by exactly one `role="tool"` row with the matching `tool_call_id`, before the next user/assistant turn.
- If the source dropped the tool result (cut-off transcript), insert a synthetic `role="tool"` row with `content="[tool result missing in source]"` to keep the chain valid. An assistant row with an unanswered `tool_calls[].id` will break replay and any LLM round-trip.
- Multi-tool assistant turns: one `role="tool"` row per call, in any order, all before the next non-tool row.
### Tool ID generation
If the source used opaque tool IDs that aren't unique within a thread (some platforms reuse them), regenerate with a stable scheme like `f"call_{i}"` where `i` is a per-thread counter. Update both the assistant and tool rows together.
## Provider Fidelity (`provider_data`)
Skip this entirely for **archive** imports.
For **resumable** imports against the same provider, populate `provider_data` to preserve provider-specific tool-call metadata that the next API round-trip will require:
- **Anthropic**: `signature` field on thinking blocks; required for round-tripping extended-thinking responses.
- **Gemini**: `thought_signature` on tool calls; required for fidelity.
- **OpenAI**: typically nothing to preserve.
The runtime-side dict key is `_provider_content` (a list of provider-native blocks); the persisted column is `provider_data` (the same list, JSON-encoded). If you don't have provider-native blocks from the source — and you usually won't, because a foreign export won't include them — leave `provider_data` NULL. The first new turn will succeed without it, but the previous assistant turn's reasoning won't replay back to the model.
## Attachments
If the source thread had image or file attachments:
- **Size limits**: images ≤ 4 MiB, text documents ≤ 512 KiB. Reject or downsample anything bigger.
- **Allowed types**: server validates magic bytes for images and UTF-8-decodes for text. Binary blobs that aren't images won't pass.
- **Lifecycle**: pending → reserved → consumed. For imports, the cleanest path is to upload as pending and immediately consume by attaching to the relevant `conversations.id`.
Two import paths:
1. **Bulk-insert + post-attach**: insert messages first, get back the assistant/user `conversations.id`, then write `workstream_attachments` rows linking the file to `message_id`.
2. **SDK multipart create**: `create_workstream(attachments=[...], initial_message=...)` for the *first* turn only — the server reserves and consumes them onto that turn. Doesn't help for mid-thread attachments.
For full-history imports with multiple attachments at different turns, path (1) is the only option.
## Validation Checklist
Before declaring success, verify:
- [ ] `ws_id` is 32-char lowercase hex.
- [ ] `workstreams` row exists with the right `user_id`, `state`, `kind`.
- [ ] Conversation rows are inserted **in order** (autoincrement `id` will reflect insert order).
- [ ] Every assistant `tool_calls[].id` has a matching `role="tool"` row with the same `tool_call_id`.
- [ ] `tool_calls[].function.arguments` is a JSON-encoded **string**, not a parsed object.
- [ ] First message is typically `role="user"` (not `system`) — Turnstone composes its own system prompt at runtime.
- [ ] No empty assistant rows (`content=NULL` AND `tool_calls=NULL` is invalid).
- [ ] If multi-node: the `ws_id`'s bucket maps to a node that exists; `workstreams.node_id` matches.
- [ ] Round-trip test: run `Storage.load_messages(ws_id)` and confirm the reconstructed list matches what you inserted (modulo timestamps).
## Anti-patterns
- **Don't import the source provider's system prompt verbatim.** Provider boilerplate ("You are Claude...", "You are ChatGPT...") will conflict with Turnstone's composed system message and confuse the model on resume. Drop it; preserve only user-authored custom instructions.
- **Don't preserve foreign tool definitions as Turnstone tools.** If the source had custom tools that don't exist in Turnstone, the assistant rows that called them are still valid history (archive), but the workstream is **not resumable** — mark `state="closed"`.
- **Don't fabricate `tool_call_id`s without re-pairing.** Mismatched ids silently break the replay chain on the next turn.
- **Don't skip the `tool_name` field on `role="tool"` rows.** Some load paths use it for display and audit; NULL there will render as "unknown tool".
- **Don't write through the LLM (`send()` per turn) for full history.** It's expensive, rewrites assistant turns, and rate-limits will bite long imports.
## Quick Reference
| Task | Path |
|---|---|
| Generate ws_id | `secrets.token_hex(16)` |
| Bulk insert messages | `Storage.save_messages_bulk(rows)` |
| Archive (read-only) | `state="closed"`, skip `provider_data` |
| Resumable | `state="idle"`, populate `provider_data` if same provider |
| Tool call id | OpenAI shape: `{"id": ..., "type": "function", "function": {"name": ..., "arguments": "<json string>"}}` |
| Tool result row | `role="tool"`, `tool_name`, `tool_call_id`, `content` |
| Source role → Turnstone role | See "Role Mapping" table |
| Per-thread metadata | Store source IDs in `workstream_config` under `import.*` keys |
## Files to read before writing the importer
- `turnstone/core/storage/_schema.py` — authoritative table definitions.
- `turnstone/core/storage/_protocol.py``save_message`, `save_messages_bulk`, `load_messages` signatures.
- `turnstone/core/session.py` (around the message-save section) — how the runtime constructs in-memory message dicts; mirror this shape on import to round-trip cleanly.
- `turnstone/api/server_schemas.py` — Pydantic shapes for the SDK paths if you go through HTTP.
+5 -14
View File
@@ -758,22 +758,18 @@ MCP tools (3):
### Dynamic tool refresh
MCP tool lists stay up-to-date without restart through three mechanisms:
MCP tool lists stay up-to-date without restart through two mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
2. **Periodic timer** -- Servers that do *not* support push notifications are polled
on a configurable interval (default 4 hours). The timer is staggered using a
launch-time seed (`monotonic_ns ^ pid`) so cluster nodes don't all hit MCP
servers simultaneously. Configure via `[mcp] refresh_interval` in `config.toml`
or `--mcp-refresh-interval SECONDS` on the CLI. Set to `0` to disable.
3. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
2. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
manual refresh attempts reconnection.
manual refresh attempts reconnection. The console admin panel exposes the
same controls (refresh / reconnect buttons per server) for cluster-wide
fan-out.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
@@ -781,11 +777,6 @@ instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`, `_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```toml
[mcp]
refresh_interval = 14400 # seconds (default 4h), 0 to disable
```
```
/mcp refresh
MCP refresh complete:
+3 -2
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.5.0"
version = "1.5.15"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -24,7 +24,7 @@ classifiers = [
dependencies = [
"openai>=2.24",
"httpx>=0.28",
"mcp>=1.6",
"mcp>=1.27",
"starlette>=0.45",
"uvicorn>=0.34",
"sse-starlette>=2.0",
@@ -35,6 +35,7 @@ dependencies = [
"structlog>=24.1",
"PyJWT>=2.8",
"bcrypt>=4.0",
"cryptography>=42",
"python-frontmatter>=1.0",
]
+92 -92
View File
@@ -74,9 +74,9 @@
}
},
"node_modules/@oxc-project/types": {
"version": "0.127.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz",
"integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==",
"version": "0.129.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.129.0.tgz",
"integrity": "sha512-3oz8m3FGdr2nDXVqmFUw7jolKliC4MoyXYIG2c7gpjBnzUWQpUGIYcXYKxTdTi+N2jusvt610ckTMkxdwHkYEg==",
"dev": true,
"license": "MIT",
"funding": {
@@ -84,9 +84,9 @@
}
},
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0.tgz",
"integrity": "sha512-TWMZnRLMe63C2Lhyicviu7ZHaU4kxa6PS3rofvc9GmcvptzNN11BcfQ4Sl7MwTOsisQoa2keB/EBdNCAnUo8vA==",
"cpu": [
"arm64"
],
@@ -101,9 +101,9 @@
}
},
"node_modules/@rolldown/binding-darwin-arm64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0.tgz",
"integrity": "sha512-6XcD+8k0gPVItNagEw78/qqcBDwKcwDYS8V2hRmVsfUSIrd8cWe/CBvRDI5toqFyPfj+FJr6t8U6Xj2P2prEew==",
"cpu": [
"arm64"
],
@@ -118,9 +118,9 @@
}
},
"node_modules/@rolldown/binding-darwin-x64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz",
"integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0.tgz",
"integrity": "sha512-iN/tWVXRQDWvmZlKdceP1Dwug9GDpEymhb9p4xnEe6zvCg5lFmzVljl+1qR1NVx3yfGpr2Na+CuLmv5IU8uzfQ==",
"cpu": [
"x64"
],
@@ -135,9 +135,9 @@
}
},
"node_modules/@rolldown/binding-freebsd-x64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz",
"integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0.tgz",
"integrity": "sha512-jjQMDvvwSOuhOwMszD/klSOjyWMM3zI64hWTj9KT5x4MxRbZAf+7vLQ6qouRhtsLVFHr3f0ILaJAfgENPiQdAQ==",
"cpu": [
"x64"
],
@@ -152,9 +152,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz",
"integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0.tgz",
"integrity": "sha512-d//Dtg2x6/m3mbV64yUGNnDGNZaDGRpDLLNGerHQUVObuNaIQaaDp25yUiqGXtHEXX+NP2d0wAlmKgpYgIAJ2A==",
"cpu": [
"arm"
],
@@ -169,9 +169,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0.tgz",
"integrity": "sha512-n7Ofp0mx+aB2cC+Sdy5YtMnXtY9lchnHbY+3Yt0uq9JsWQExf4f5Whu0tK0R8Jdc9S6RchTHjIFY7uc92puOVQ==",
"cpu": [
"arm64"
],
@@ -189,9 +189,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-musl": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz",
"integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0.tgz",
"integrity": "sha512-EIVjy2cgd7uuMMo94FVkBp7F6DhcZAUwNURkSG3RwUmvAXR6s0ISxM81U+IydcZByPG0pZIHsf1b6kTxoFDgJA==",
"cpu": [
"arm64"
],
@@ -209,9 +209,9 @@
}
},
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0.tgz",
"integrity": "sha512-JEwwOPcwTLAcpDQlqSmjEmfs63xJnSiUNIGvLcDLUHCWK4XowpS/7c7tUsUH6uT/ct6bMUTdXKfI8967FYj6mg==",
"cpu": [
"ppc64"
],
@@ -229,9 +229,9 @@
}
},
"node_modules/@rolldown/binding-linux-s390x-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0.tgz",
"integrity": "sha512-0wjCFhLrihtAubnT9iA0N++0pSV0z5Hg7tNGdNJ4RFaINceHadoF+kiFGyY1qSSNVIAZtLotG8Ju1bgDPkjnFA==",
"cpu": [
"s390x"
],
@@ -249,9 +249,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0.tgz",
"integrity": "sha512-Dfn7iak9BcMMePxcoJfpSbWqnEyrp/dRF63/8qW/eHBdOZov6x5aShLLEYGYdIeSJ6vMLK/XCVB+lGIxm41bQA==",
"cpu": [
"x64"
],
@@ -269,9 +269,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-musl": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz",
"integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0.tgz",
"integrity": "sha512-5/utzzDmD/pD/bmuaUcbTf/sZYy0aztwIVlfpoW1fTjCZ0BaPOMVWGZL1zvgxyi7ZIVYWlxKONHmSbHuiOh8Jw==",
"cpu": [
"x64"
],
@@ -289,9 +289,9 @@
}
},
"node_modules/@rolldown/binding-openharmony-arm64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0.tgz",
"integrity": "sha512-ouJs8VcUomfLfpbUECqFMRqdV4x6aeAK3MA4m6vTrJJjKyWTV5KnxZx7Jd9G+GlDaQQxubcba00x16OyJ1meig==",
"cpu": [
"arm64"
],
@@ -306,9 +306,9 @@
}
},
"node_modules/@rolldown/binding-wasm32-wasi": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz",
"integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0.tgz",
"integrity": "sha512-E+oHKGiDA+lsKMmFtffDDw91EryDT7uJocrIuCHqhm6bCTM6xFK+3gaCkYOHfPwQr0cCNarSM2xaELoQDz9jJg==",
"cpu": [
"wasm32"
],
@@ -325,9 +325,9 @@
}
},
"node_modules/@rolldown/binding-win32-arm64-msvc": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz",
"integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0.tgz",
"integrity": "sha512-yYK02n8Rngo+gbm1y6G0+7jk1sJ/2Wt7K0me0Y7k/ErBpyf+LJ2gFpqWVTcRV1rUepBlQRmpgWkTQCiiwrK0Ow==",
"cpu": [
"arm64"
],
@@ -342,9 +342,9 @@
}
},
"node_modules/@rolldown/binding-win32-x64-msvc": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz",
"integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0.tgz",
"integrity": "sha512-14bpChMahXRRXiTwahSl+zzHPW6qQTXtkMuJBFlbo+pqSAews2d4BdCSHfrJ/MBsCZtpmTafsY+1QhBzitcmdg==",
"cpu": [
"x64"
],
@@ -359,9 +359,9 @@
}
},
"node_modules/@rolldown/pluginutils": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz",
"integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0.tgz",
"integrity": "sha512-aKs/3GSWyV0mrhNmt/96/Z3yczC3yvrzYATCiCXQebBsGyYzjNdUphRVLeJQ67ySKVXRfMxt2lm12pmXvbPFQQ==",
"dev": true,
"license": "MIT"
},
@@ -373,9 +373,9 @@
"license": "MIT"
},
"node_modules/@tybys/wasm-util": {
"version": "0.10.1",
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz",
"integrity": "sha512-9tTaPJLSiejZKx+Bmog4uSubteqTvFrVrURwkmHixBo0G4seD0zUxp98E1DzUBJxLQ3NPwXrGKDiVjwx/DpPsg==",
"version": "0.10.2",
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.2.tgz",
"integrity": "sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==",
"dev": true,
"license": "MIT",
"optional": true,
@@ -402,9 +402,9 @@
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
"integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==",
"dev": true,
"license": "MIT"
},
@@ -902,9 +902,9 @@
}
},
"node_modules/nanoid": {
"version": "3.3.11",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz",
"integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==",
"version": "3.3.12",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.12.tgz",
"integrity": "sha512-ZB9RH/39qpq5Vu6Y+NmUaFhQR6pp+M2Xt76XBnEwDaGcVAqhlvxrl3B2bKS5D3NH3QR76v3aSrKaF/Kiy7lEtQ==",
"dev": true,
"funding": [
{
@@ -959,9 +959,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.12",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.12.tgz",
"integrity": "sha512-W62t/Se6rA0Az3DfCL0AqJwXuKwBeYg6nOaIgzP+xZ7N5BFCI7DYi1qs6ygUYT6rvfi6t9k65UMLJC+PHZpDAA==",
"version": "8.5.14",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
"dev": true,
"funding": [
{
@@ -988,14 +988,14 @@
}
},
"node_modules/rolldown": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz",
"integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==",
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0.tgz",
"integrity": "sha512-yD986aXDESFGS95spT1LAv0jssywP4npMEjmMHyN2/5+eE8qQJUype2AaKkRiLgBgyD0LFlubwAht7VmY8rGoA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@oxc-project/types": "=0.127.0",
"@rolldown/pluginutils": "1.0.0-rc.17"
"@oxc-project/types": "=0.129.0",
"@rolldown/pluginutils": "1.0.0"
},
"bin": {
"rolldown": "bin/cli.mjs"
@@ -1004,21 +1004,21 @@
"node": "^20.19.0 || >=22.12.0"
},
"optionalDependencies": {
"@rolldown/binding-android-arm64": "1.0.0-rc.17",
"@rolldown/binding-darwin-arm64": "1.0.0-rc.17",
"@rolldown/binding-darwin-x64": "1.0.0-rc.17",
"@rolldown/binding-freebsd-x64": "1.0.0-rc.17",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17",
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17",
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.17",
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.17",
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.17",
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17",
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17"
"@rolldown/binding-android-arm64": "1.0.0",
"@rolldown/binding-darwin-arm64": "1.0.0",
"@rolldown/binding-darwin-x64": "1.0.0",
"@rolldown/binding-freebsd-x64": "1.0.0",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0",
"@rolldown/binding-linux-arm64-gnu": "1.0.0",
"@rolldown/binding-linux-arm64-musl": "1.0.0",
"@rolldown/binding-linux-ppc64-gnu": "1.0.0",
"@rolldown/binding-linux-s390x-gnu": "1.0.0",
"@rolldown/binding-linux-x64-gnu": "1.0.0",
"@rolldown/binding-linux-x64-musl": "1.0.0",
"@rolldown/binding-openharmony-arm64": "1.0.0",
"@rolldown/binding-wasm32-wasi": "1.0.0",
"@rolldown/binding-win32-arm64-msvc": "1.0.0",
"@rolldown/binding-win32-x64-msvc": "1.0.0"
}
},
"node_modules/siginfo": {
@@ -1060,9 +1060,9 @@
"license": "MIT"
},
"node_modules/tinyexec": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.1.1.tgz",
"integrity": "sha512-VKS/ZaQhhkKFMANmAOhhXVoIfBXblQxGX1myCQ2faQrfmobMftXeJPcZGp0gS07ocvGJWDLZGyOZDadDBqYIJg==",
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.1.2.tgz",
"integrity": "sha512-dAqSqE/RabpBKI8+h26GfLq6Vb3JVXs30XYQjdMjaj/c2tS8IYYMbIzP599KtRj7c57/wYApb3QjgRgXmrCukA==",
"dev": true,
"license": "MIT",
"engines": {
@@ -1119,16 +1119,16 @@
}
},
"node_modules/vite": {
"version": "8.0.10",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
"version": "8.0.12",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.12.tgz",
"integrity": "sha512-w2dDofOWv2QB09ZITZBsvKTVAlYvPR4IAmrY/v0ir9KvLs0xybR7i48wxhM1/oyBWO34wPns+bPGw5ZrZqDpZg==",
"dev": true,
"license": "MIT",
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
"postcss": "^8.5.10",
"rolldown": "1.0.0-rc.17",
"postcss": "^8.5.14",
"rolldown": "1.0.0",
"tinyglobby": "^0.2.16"
},
"bin": {
@@ -1145,7 +1145,7 @@
},
"peerDependencies": {
"@types/node": "^20.19.0 || >=22.12.0",
"@vitejs/devtools": "^0.1.0",
"@vitejs/devtools": "^0.1.18",
"esbuild": "^0.27.0 || ^0.28.0",
"jiti": ">=1.21.0",
"less": "^4.0.0",
+29
View File
@@ -13,6 +13,20 @@ export interface ConnectedEvent {
export interface HistoryEvent {
type: "history";
/**
* Per-message dicts the frontend consumes directly. Common optional keys:
* - `role`: "user" | "assistant" | "tool"
* - `content`: string or list (image/document parts)
* - `tool_calls`: assistant turns list of `{id, name, arguments, verdict?, output_assessment?}`
* - `tool_call_id`: tool turns id of the originating call
* - `reminders`: metacognitive nudge bubbles (user/tool channels)
* - `advisories`: extracted `UserInterjection` payloads on tool turns
* - `reasoning`: concatenated reasoning text for assistant turns whose
* `provider_data` carried reasoning-bearing blocks (Anthropic
* `thinking`, OpenAI Responses `reasoning`, or synthetic
* `reasoning_text` from path-3 servers). Present only when the
* active model's `surface_persisted_reasoning` flag is true.
*/
messages: Array<Record<string, unknown>>;
}
@@ -38,6 +52,14 @@ export interface StreamEndEvent {
type: "stream_end";
}
/** One-shot replay of the in-progress turn's content + reasoning emitted
* by the events SSE handler when a fresh subscriber connects mid-stream. */
export interface InProgressSnapshotEvent {
type: "in_progress_snapshot";
content: string;
reasoning: string;
}
export interface StateChangeEvent {
type: "state_change";
state: "idle" | "thinking" | "running" | "attention" | "error";
@@ -162,6 +184,7 @@ export type ServerEvent =
| ContentEvent
| ReasoningEvent
| StreamEndEvent
| InProgressSnapshotEvent
| StateChangeEvent
| ToolInfoEvent
| ApproveRequestEvent
@@ -261,6 +284,12 @@ export function isStreamEndEvent(e: ServerEvent): e is StreamEndEvent {
return e.type === "stream_end";
}
export function isInProgressSnapshotEvent(
e: ServerEvent,
): e is InProgressSnapshotEvent {
return e.type === "in_progress_snapshot";
}
export function isStateChangeEvent(e: ServerEvent): e is StateChangeEvent {
return e.type === "state_change";
}
+3 -4
View File
@@ -35,11 +35,10 @@ def _seed_children(
The production path populates the registry via the cluster-event
fan-out thread observing ``ws_created`` events. These tests just
need a known-children set for the endpoint handlers to iterate
inject directly under ``_children_lock`` rather than spinning up
the collector + fan-out plumbing.
inject directly via the registry's bulk-merge surface rather than
spinning up the collector + fan-out plumbing.
"""
with adapter._children_lock:
adapter._merge_child_ids_locked(coord_ws_id, child_ws_ids)
adapter._registry.merge_children(coord_ws_id, child_ws_ids)
class _AuthMiddleware(BaseHTTPMiddleware):
+53
View File
@@ -0,0 +1,53 @@
"""Shared test helpers — kept out of conftest.py since these are factories,
not fixtures, and several test files want to import them directly."""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
def make_chat_session(**overrides: Any) -> Any:
"""Build a minimal ``ChatSession`` with sane test defaults.
Caller passes any constructor arg as a kwarg to override the default
e.g. ``make_chat_session(memory_config=MemoryConfig(fetch_limit=5))``.
"""
from turnstone.core.session import ChatSession
defaults: dict[str, Any] = {
"client": MagicMock(),
"model": "test-model",
"ui": MagicMock(),
"instructions": None,
"temperature": 0.5,
"max_tokens": 4096,
"tool_timeout": 30,
}
defaults.update(overrides)
return ChatSession(**defaults)
def patch_session_storage(
monkeypatch: Any,
*,
active: bool = True,
raise_on_is_active: bool = False,
) -> list[str]:
"""Patch ``session.get_storage`` to a stub whose ``is_watch_active``
returns *active* (or raises if *raise_on_is_active*). Returns the
list of ``watch_id``s the predicate was called with.
"""
from turnstone.core import session as session_mod
calls: list[str] = []
class _Stub:
def is_watch_active(self, watch_id: str) -> bool:
calls.append(watch_id)
if raise_on_is_active:
raise RuntimeError("storage down")
return active
monkeypatch.setattr(session_mod, "get_storage", lambda: _Stub())
return calls
+45
View File
@@ -0,0 +1,45 @@
"""Shared session-test helpers.
Two reasoning-test modules (``test_session_replay_reasoning.py`` and
``test_session_synth_reasoning_block.py``) need the same minimal
``ChatSession`` factory + a ``SessionUIBase`` no-op subclass. Hoisting
keeps a future third caller from drifting on the defaults the third
existing ``_make_session`` (``test_model_registry.py``) deliberately
takes a different signature (registry / model_alias / reasoning_effort
+ ``_FakeUI``) and is NOT a candidate for sharing this helper.
Module is named with a leading underscore so pytest doesn't try to
collect it as a test file it's an importable utility, not a test.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
from turnstone.core.session_ui_base import SessionUIBase
class NullUI(SessionUIBase):
"""Bare-bones UI satisfying the SessionUIBase contract for tests
that don't care about UI side effects."""
def __init__(self) -> None:
super().__init__()
def make_session(**kwargs: Any) -> ChatSession:
"""Build a ChatSession with minimal defaults; tests override
individual fields via kwargs."""
defaults: dict[str, Any] = {
"client": MagicMock(),
"model": "test-model",
"ui": NullUI(),
"instructions": None,
"temperature": 0.5,
"max_tokens": 4096,
"tool_timeout": 30,
}
defaults.update(kwargs)
return ChatSession(**defaults)
+69
View File
@@ -1,10 +1,79 @@
from __future__ import annotations
import os
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock
import pytest
if TYPE_CHECKING:
from turnstone.core.mcp_client import MCPClientManager, StaticServerState
from turnstone.core.mcp_crypto import MCPTokenCipher
from turnstone.core.oidc import OIDCConfig
def make_mcp_token_cipher() -> MCPTokenCipher:
"""Build a single-key MCP token cipher for tests.
Used by test files that need to exercise ``MCPTokenStore`` round-
trips without the lifespan-side configuration loader; centralised
here so the key/material defaults stay aligned across files.
"""
import base64
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import MCPTokenCipher, MCPTokenCipherConfig
raw = base64.urlsafe_b64decode(Fernet.generate_key())
return MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,)))
def _seed_static_state(mgr: MCPClientManager, name: str, **overrides: Any) -> StaticServerState:
"""Get-or-create a ``StaticServerState`` on ``mgr`` and apply ``overrides``.
Shared across MCP test files so the helper stays in one place. Imported
where needed; ``StaticServerState`` is constructed lazily so non-MCP
tests don't pay the import cost.
"""
from turnstone.core.mcp_client import StaticServerState
state = mgr._static_servers.get(name)
if state is None:
state = StaticServerState(name=name)
mgr._static_servers[name] = state
for k, v in overrides.items():
setattr(state, k, v)
return state
def make_oidc_test_config(**overrides: Any) -> OIDCConfig:
"""Build a test ``OIDCConfig`` with sensible defaults.
Shared between ``test_oidc.py`` and ``test_oidc_handlers.py`` so the
defaults (including the now-required ``redirect_base``) stay aligned.
"""
from turnstone.core.oidc import OIDCConfig
defaults: dict[str, Any] = {
"enabled": True,
"issuer": "https://idp.example.com",
"client_id": "my-client",
"client_secret": "my-secret",
"scopes": "openid email profile",
"provider_name": "TestIDP",
"role_claim": "",
"role_map": {},
"password_enabled": True,
"redirect_base": "https://app.example.com",
"authorization_endpoint": "https://idp.example.com/authorize",
"token_endpoint": "https://idp.example.com/token",
"userinfo_endpoint": "https://idp.example.com/userinfo",
"jwks_uri": "https://idp.example.com/.well-known/jwks.json",
}
defaults.update(overrides)
return OIDCConfig(**defaults)
def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption(
+391
View File
@@ -0,0 +1,391 @@
"""Spike 1 — validate MCP SDK behavior for the per-(user, server) session pool.
Three scenarios:
1. N=20 concurrent ClientSession instances to the same URL.
Verifies: no FD blow-up, no shared transport state, each session's
tools/list returns independently.
2. Two concurrent tools/call on a shared ClientSession with interleaving
payloads. Verifies: request_id demux works under contention.
3. Per-session Authorization header isolation. Verifies: different Bearer
tokens per ClientSession reach the server with the expected
Authorization header i.e. httpx connection pooling does not cross
headers between sessions.
Run: uv run python tests/spike_sdk_concurrency.py
Outcome gates Phase 5's pool architecture; if any scenario fails, fall
back to per-call header injection (Alternative F in the OAuth-MCP RFC).
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
import os
import socket
import sys
import threading
import time
from collections import defaultdict
from typing import TYPE_CHECKING
import uvicorn
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
# Reduce uvicorn / mcp log noise so spike output is readable.
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
# Records (auth_header, tool_name) per request — populated by the
# AuthHeaderRecorder middleware below. Indexed by call sequence.
SERVER_OBSERVATIONS: list[tuple[str | None, str | None]] = []
# Tool-call payloads observed (for request_id demux verification).
TOOL_CALL_PAYLOADS: list[str] = []
class AuthHeaderRecorder(BaseHTTPMiddleware):
"""Records the Authorization header on every request the server sees."""
async def dispatch(self, request: Request, call_next: Callable) -> Response:
auth = request.headers.get("authorization")
# We only record the auth header here; tool name comes from the
# body payload which we can't read non-destructively. The tool
# handler logs the payload it received.
SERVER_OBSERVATIONS.append((auth, None))
return await call_next(request)
def find_free_port() -> int:
"""Bind to port 0, return the assigned port."""
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def build_server(port: int) -> uvicorn.Server:
"""Create a minimal FastMCP server with one echo tool."""
mcp = FastMCP(name="spike-target", streamable_http_path="/mcp")
@mcp.tool()
async def echo(payload: str) -> str:
"""Echo the payload back. Records the payload server-side."""
TOOL_CALL_PAYLOADS.append(payload)
# Add a small await so two concurrent calls can interleave
# on the wire if the SDK pools the requests.
await asyncio.sleep(0.05)
return f"echoed:{payload}"
app = mcp.streamable_http_app()
app.add_middleware(AuthHeaderRecorder)
config = uvicorn.Config(
app,
host="127.0.0.1",
port=port,
log_level="warning",
access_log=False,
)
return uvicorn.Server(config)
def run_server_in_thread(server: uvicorn.Server) -> threading.Thread:
"""Boot the server in a background thread on its own asyncio loop."""
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="spike-server")
t.start()
return t
async def wait_for_server_ready(url: str, timeout: float = 5.0) -> None:
"""Poll the server until it accepts connections."""
import urllib.parse
parsed = urllib.parse.urlparse(url)
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
reader, writer = await asyncio.open_connection(parsed.hostname, parsed.port)
writer.close()
await writer.wait_closed()
return
except OSError:
await asyncio.sleep(0.05)
raise TimeoutError(f"server at {url} not ready within {timeout}s")
def fd_count() -> int:
"""Count open file descriptors for the current process."""
try:
return len(os.listdir(f"/proc/{os.getpid()}/fd"))
except OSError:
return -1
# ---------------------------------------------------------------------------
# Scenario 1: N=20 concurrent ClientSession instances
# ---------------------------------------------------------------------------
async def scenario_1_concurrent_sessions(url: str, n: int = 20) -> dict:
"""Open N concurrent ClientSession instances and call tools/list on each."""
print(f"\n=== Scenario 1: {n} concurrent ClientSession instances ===")
fd_before = fd_count()
async def one_session(idx: int) -> dict:
headers = {"Authorization": f"Bearer test-token-{idx}"}
async with (
streamablehttp_client(url=url, headers=headers) as (read, write, _),
ClientSession(read, write) as session,
):
await session.initialize()
tools = await session.list_tools()
return {
"idx": idx,
"tool_count": len(tools.tools),
"tool_names": [t.name for t in tools.tools],
}
start = time.monotonic()
results = await asyncio.gather(*[one_session(i) for i in range(n)], return_exceptions=True)
elapsed = time.monotonic() - start
fd_after = fd_count()
# Allow some settling time for FDs to release.
await asyncio.sleep(0.5)
fd_settled = fd_count()
successes = [r for r in results if isinstance(r, dict)]
failures = [r for r in results if isinstance(r, Exception)]
# Verify every session got the same tool catalog.
catalog_consistent = (
len(successes) == n and len({tuple(r["tool_names"]) for r in successes}) == 1
)
return {
"scenario": "concurrent_sessions",
"n": n,
"successes": len(successes),
"failures": len(failures),
"elapsed_seconds": round(elapsed, 3),
"fd_before": fd_before,
"fd_during_peak": fd_after,
"fd_settled": fd_settled,
"fd_growth_during": fd_after - fd_before,
"fd_growth_settled": fd_settled - fd_before,
"catalog_consistent": catalog_consistent,
"first_failure": str(failures[0]) if failures else None,
}
# ---------------------------------------------------------------------------
# Scenario 2: 2 concurrent tools/call on a shared session
# ---------------------------------------------------------------------------
async def scenario_2_concurrent_calls_shared_session(url: str) -> dict:
"""Two concurrent tools/call on one ClientSession with interleaving payloads.
The echo tool sleeps 50ms, so concurrent calls overlap on the wire.
Each call passes a distinct payload (~10KB) to make request bodies
spannable across multiple stream frames.
"""
print("\n=== Scenario 2: 2 concurrent tools/call on shared session ===")
# Generous-size payloads so both bodies live during the await.
payload_a = "A" * 10000
payload_b = "B" * 10000
headers = {"Authorization": "Bearer shared-session-token"}
async with (
streamablehttp_client(url=url, headers=headers) as (read, write, _),
ClientSession(read, write) as session,
):
await session.initialize()
TOOL_CALL_PAYLOADS.clear()
start = time.monotonic()
results = await asyncio.gather(
session.call_tool("echo", {"payload": payload_a}),
session.call_tool("echo", {"payload": payload_b}),
return_exceptions=True,
)
elapsed = time.monotonic() - start
successes = [r for r in results if not isinstance(r, Exception)]
failures = [r for r in results if isinstance(r, Exception)]
# Each result.content[0].text should be "echoed:{payload}".
response_payloads: list[str] = []
if len(successes) == 2:
for r in successes:
text = r.content[0].text if r.content else ""
response_payloads.append(text)
# Order may not match call order — what matters is both payloads echo.
expected = {f"echoed:{payload_a}", f"echoed:{payload_b}"}
received = set(response_payloads)
demux_ok = received == expected
# Did both calls actually overlap? If sequential, elapsed ~= 0.1+s;
# if concurrent, ~0.05s.
concurrent_observed = elapsed < 0.09
return {
"scenario": "concurrent_calls_shared_session",
"successes": len(successes),
"failures": len(failures),
"elapsed_seconds": round(elapsed, 3),
"demux_ok": demux_ok,
"expected_payloads_received": list(received) if demux_ok else None,
"actual_payloads_received_count": len(received),
"appears_concurrent_on_wire": concurrent_observed,
"first_failure": str(failures[0]) if failures else None,
}
# ---------------------------------------------------------------------------
# Scenario 3: per-session header isolation
# ---------------------------------------------------------------------------
async def scenario_3_header_isolation(url: str, n: int = 5) -> dict:
"""Open N sessions with distinct Authorization headers, call echo on each.
Verifies the server sees each session's own header — i.e. httpx
connection pooling does not cross headers between concurrent
ClientSession instances against the same URL.
"""
print(f"\n=== Scenario 3: {n}-session Authorization-header isolation ===")
SERVER_OBSERVATIONS.clear()
async def one_session(idx: int) -> str | None:
headers = {"Authorization": f"Bearer iso-token-{idx}"}
async with (
streamablehttp_client(url=url, headers=headers) as (read, write, _),
ClientSession(read, write) as session,
):
await session.initialize()
# One call per session.
await session.call_tool("echo", {"payload": f"session-{idx}"})
return f"Bearer iso-token-{idx}"
start = time.monotonic()
expected_tokens = await asyncio.gather(*[one_session(i) for i in range(n)])
elapsed = time.monotonic() - start
# Tally observed Authorization headers, ignoring None entries (initial
# handshake sometimes lacks auth).
observed_auth = [auth for auth, _ in SERVER_OBSERVATIONS if auth]
expected_set = set(expected_tokens)
observed_set = set(observed_auth)
# Every expected token must show up at least once on the server.
all_present = expected_set.issubset(observed_set)
# No spurious tokens.
no_extras = observed_set.issubset(expected_set)
# Frequency: at least one observation per token.
counts = defaultdict(int)
for a in observed_auth:
counts[a] += 1
each_seen = all(counts[t] >= 1 for t in expected_tokens)
return {
"scenario": "header_isolation",
"n": n,
"elapsed_seconds": round(elapsed, 3),
"expected_tokens": sorted(expected_set),
"observed_tokens": sorted(observed_set),
"all_expected_present": all_present,
"no_extra_tokens_observed": no_extras,
"each_token_seen_at_least_once": each_seen,
"header_counts_per_token": dict(counts),
"total_requests_observed": len(observed_auth),
}
# ---------------------------------------------------------------------------
# Driver
# ---------------------------------------------------------------------------
async def main() -> None:
port = find_free_port()
url = f"http://127.0.0.1:{port}/mcp"
server = build_server(port)
server_thread = run_server_in_thread(server)
try:
await wait_for_server_ready(url)
print(f"server up at {url}\n")
result_1 = await scenario_1_concurrent_sessions(url, n=20)
print_scenario_result(result_1)
result_2 = await scenario_2_concurrent_calls_shared_session(url)
print_scenario_result(result_2)
result_3 = await scenario_3_header_isolation(url, n=5)
print_scenario_result(result_3)
# Final verdict
verdict_1 = (
result_1["successes"] == result_1["n"]
and result_1["catalog_consistent"]
and result_1["fd_growth_settled"] < 30 # 20 sessions, generous bound
)
verdict_2 = result_2["demux_ok"] and result_2["successes"] == 2
verdict_3 = (
result_3["all_expected_present"]
and result_3["no_extra_tokens_observed"]
and result_3["each_token_seen_at_least_once"]
)
print("\n=== VERDICT ===")
print(f" Scenario 1 (concurrent sessions): {'PASS' if verdict_1 else 'FAIL'}")
print(f" Scenario 2 (concurrent calls shared): {'PASS' if verdict_2 else 'FAIL'}")
print(f" Scenario 3 (header isolation): {'PASS' if verdict_3 else 'FAIL'}")
all_pass = verdict_1 and verdict_2 and verdict_3
print(
f"\n Phase 5 per-(user, server) pool architecture: "
f"{'VIABLE' if all_pass else 'NEEDS REWORK (Alternative F fallback)'}"
)
sys.exit(0 if all_pass else 1)
finally:
server.should_exit = True
server_thread.join(timeout=5)
def print_scenario_result(result: dict) -> None:
print(f"\nresult[{result['scenario']}]:")
for k, v in result.items():
if k == "scenario":
continue
print(f" {k}: {v}")
if __name__ == "__main__":
with contextlib.suppress(KeyboardInterrupt):
asyncio.run(main())
File diff suppressed because it is too large Load Diff
+434
View File
@@ -92,3 +92,437 @@ def test_tool_error_does_not_overwrite_approval_badge() -> None:
"badge instead so the approval verdict stays visible alongside "
"the error."
)
def test_replay_history_renders_content_before_tool_block() -> None:
"""In ``replayHistory``'s ``role === "assistant"`` branch, the
``msg.content`` render must precede the ``msg.tool_calls`` render.
Two reasons, both load-bearing:
1. **Structural** the next loop iteration's ``role === "tool"``
message anchors to ``lastToolBlock``. The tool-block branch sets
that anchor; the content branch clears it. If content runs after
the tool block, the clear silently drops the upcoming tool
result. Pre-fix, every interactive tool result was missing from
saved-workstream replays whenever the assistant turn carried
both narration and tool calls (very common output shape).
2. **Visual** the live SSE path renders content first
(``stream_text`` streams before ``tool_info`` /
``approve_request``), so replay should match.
The test pins the order via the offsets of the ``msg.content`` and
``msg.tool_calls`` branch headers inside the function body."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("Pane.prototype.replayHistory = function")
end = body.index("Pane.prototype._attachRetryToLastAssistant", start)
fn = body[start:end]
# Locate the assistant branch and bound the search to its body —
# the function also handles user / tool roles which would otherwise
# confuse the offset comparison.
asst_start = fn.index('msg.role === "assistant"')
asst_end = fn.index('msg.role === "tool"', asst_start)
asst = fn[asst_start:asst_end]
# ``if (msg.content && msg.content.trim())`` guards against a
# whitespace-only content row (Qwen-style "\n\n" left over after a
# reasoning-parser model strips ``<think>…</think>`` and emits
# nothing else before the tool call). Pre-trim guard, those rows
# rendered as a visible-but-empty ``.msg.assistant`` card on
# replay. Match the substring up to ``msg.content`` so the test
# tolerates either guard shape without locking the trim() in.
content_idx = asst.index("if (msg.content")
tool_calls_idx = asst.index("if (msg.tool_calls && msg.tool_calls.length)")
assert content_idx < tool_calls_idx, (
"replayHistory must render msg.content BEFORE msg.tool_calls "
"inside the assistant branch — otherwise the lastToolBlock "
"anchor is clobbered before the next iteration's tool result "
"can attach to it (and the visual order also drifts from the "
"live SSE flow)."
)
def test_replay_history_renders_persisted_verdict_badge() -> None:
"""Saved-workstream replays must paint the persisted intent verdict
next to each tool div, using the same ``renderVerdictBadge`` helper
the live ``showInlineToolBlock`` path uses. Pre-fix the audit trail
was complete in storage (``intent_verdicts`` table) but never
surfaced on replay operators reviewing a saved workstream
couldn't see what the heuristic / LLM judge thought of any tool
call. This test pins the call site so a refactor that drops the
decoration regresses the audit surface."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("Pane.prototype.replayHistory = function")
end = body.index("Pane.prototype._attachRetryToLastAssistant", start)
fn = body[start:end]
# Match a `renderVerdictBadge(<something>.verdict, ...)` call inside
# the replay loop. Loose on whitespace + identifier so a future
# rename of the iteration variable doesn't trip CI.
badge_call_re = re.compile(
r"renderVerdictBadge\(\s*\w+\.verdict\b",
)
assert badge_call_re.search(fn), (
"replayHistory must call renderVerdictBadge(tc.verdict, ...) "
"when a persisted verdict is attached to a tool_call entry — "
"otherwise the audit-trail data persisted to intent_verdicts "
"doesn't surface on saved-workstream replays."
)
def test_shared_utils_defines_replay_advisories_after_tool() -> None:
"""The shared ``replayAdvisoriesAfterTool`` helper in
``shared_static/utils.js`` is the single source of advisory-walk +
type-filter logic for both ``app.js`` (interactive) and
``coordinator.js`` (coord). A refactor that drops the helper
breaks both surfaces, so guard its definition + filter shape here.
"""
utils_js = Path(__file__).resolve().parent.parent / "turnstone/shared_static/utils.js"
body = utils_js.read_text(encoding="utf-8")
assert "function replayAdvisoriesAfterTool" in body, (
"shared/utils.js must define replayAdvisoriesAfterTool — "
"interactive and coord both invoke it."
)
# The type filter — ``adv.type !== 'user_interjection'`` — must
# remain in the helper so a future advisory shape (output_guard,
# metacognitive nudge, etc.) doesn't silently render as a user
# bubble.
assert 'adv.type !== "user_interjection"' in body, (
"replayAdvisoriesAfterTool must filter by advisory type so a "
"future non-user_interjection advisory shape doesn't silently "
"render as a user bubble."
)
def test_replay_renders_user_interjection_advisory_after_tool_block() -> None:
"""Queued user messages spliced into the last tool-result envelope
of a batch (Seam 1) persist on the tool DB row as a wrapped
``<tool_output>`` envelope. ``decorate_history_messages`` extracts
the advisory back out and the wire layer projects it onto
``msg.advisories``; ``replayHistory`` must invoke the shared
``replayAdvisoriesAfterTool`` helper (defined in
``shared/utils.js``) so each ``user_interjection`` renders through
``addUserMessage`` and the bubble looks identical to a Seam 2/3
user row.
This test pins the call site so a refactor that drops the helper
invocation regresses the queued-during-batch replay shape
silently."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("Pane.prototype.replayHistory = function")
end = body.index("Pane.prototype._attachRetryToLastAssistant", start)
fn = body[start:end]
# The replay loop must invoke the shared helper, passing
# ``msg.advisories`` and a renderer that routes through
# ``addUserMessage``. The helper itself filters on
# ``adv.type !== "user_interjection"``; that branch lives in
# ``shared/utils.js`` (test_shared_utils_js or runtime smoke covers
# the helper's body).
assert "replayAdvisoriesAfterTool(msg.advisories" in fn, (
"replayHistory must invoke replayAdvisoriesAfterTool with "
"msg.advisories so queued messages spliced into the tool "
"envelope render as user bubbles after the tool block."
)
assert "addUserMessage(text" in fn, (
"replayHistory's renderer callback must route the extracted "
"advisory text through addUserMessage so the rendered bubble "
"matches a normal user-row replay."
)
# ---------------------------------------------------------------------------
# Phase 8 — Chunk D: MCP error embed + settings panel UX
# ---------------------------------------------------------------------------
_INDEX_HTML = Path(__file__).resolve().parent.parent / "turnstone/ui/static/index.html"
_STYLE_CSS = Path(__file__).resolve().parent.parent / "turnstone/ui/static/style.css"
# The Phase-8 D-chunk pins the absence of an unsafe DOM-write API
# in two regions of app.js. Spell the property name out of literal
# concatenation so the tooling that flags occurrences in code
# strings doesn't false-positive on the test source.
_UNSAFE_DOM_WRITE_RE = re.compile(r"\.inner" + r"HTML\s*=")
def test_phase8_mcp_error_helpers_defined_in_app_js() -> None:
"""The Phase 8 dashboard renderer adds three load-bearing helpers
next to the existing media-embed pattern: ``tryParseMcpError``
(envelope detector), ``buildMcpErrorEmbed`` (interactive card),
and the ``_pendingConsentServers`` set that drives the gear-icon
badge. A regression that drops any of them silently degrades the
OAuth consent UX to a plain JSON dump, so guard their existence
here."""
body = _APP_JS.read_text(encoding="utf-8")
assert "function tryParseMcpError" in body, (
"tryParseMcpError must remain defined — appendToolOutput's "
"error branch depends on it to detect the MCP error envelope."
)
assert "function buildMcpErrorEmbed" in body, (
"buildMcpErrorEmbed must remain defined — it renders the "
"interactive consent / forbidden / operator card."
)
assert "_pendingConsentServers" in body, (
"_pendingConsentServers state must remain — it backs the "
"gear-icon badge so a user who scrolls past a consent prompt "
"still has a stable signal that consent is pending."
)
# The buildMcpErrorEmbed pattern must also wire the "actionable"
# branch (consent_required / insufficient_scope) into the badge
# via _onConsentDetected; pin the helper name.
assert "_onConsentDetected" in body, (
"_onConsentDetected must remain — buildMcpErrorEmbed calls it "
"for the actionable category to surface the gear-icon badge."
)
def test_phase8_settings_panel_handlers_defined() -> None:
"""The settings modal exposes four entry points that the inline
``onclick`` attributes in index.html depend on. Renaming or
deleting any of them breaks the modal silently (the buttons are
still rendered but click-to-action is dead). Catch that here."""
body = _APP_JS.read_text(encoding="utf-8")
for name in [
"function openSettingsPanel",
"function closeSettingsPanel",
"function confirmRevokeMcp",
"function cancelRevokeMcp",
]:
assert name in body, f"Missing required handler: {name}"
# The connections list is fetched against the Phase-7 endpoint —
# pin the URL so a server-side rename forces an explicit UI bump.
assert "/v1/api/mcp/oauth/connections" in body, (
"Settings panel must fetch /v1/api/mcp/oauth/connections — "
"a server-side rename needs an explicit UI update."
)
def test_phase8_appendtooloutput_dispatches_mcp_error_before_renderer() -> None:
"""``appendToolOutput`` must call ``tryParseMcpError`` inside its
``isError`` branch BEFORE falling through to the plain
``renderToolOutput`` path. The ordering is what makes the
interactive consent card replace the JSON dump; reverse the calls
and the user sees the raw error envelope as text again."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("Pane.prototype.appendToolOutput = function")
end = body.index("Pane.prototype.", start + 10)
fn = body[start:end]
parse_idx = fn.find("tryParseMcpError(")
render_idx = fn.find("renderToolOutput(")
assert parse_idx >= 0, (
"appendToolOutput must call tryParseMcpError on the error path "
"before renderToolOutput, otherwise the consent card never "
"replaces the plain JSON output."
)
assert render_idx >= 0, "renderToolOutput call must remain present"
assert parse_idx < render_idx, (
"tryParseMcpError must run BEFORE renderToolOutput so the "
"interactive card path takes precedence over plain rendering."
)
def test_phase8_no_unsafe_dom_write_in_settings_panel() -> None:
"""Defensive XSS guard: the settings panel renders user-controlled
server names, scope strings, and timestamp values into the DOM.
The whole section MUST go through ``textContent``-style APIs; an
unsafe-DOM-write assignment would be a regression vector. Bound
the check to the section 15 body to avoid false positives
elsewhere."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("// 15. MCP server connections settings panel")
# Bound to the full settings section (terminates at the next
# top-level keydown handler block).
end = body.index('document.addEventListener("keydown"', start)
section = body[start:end]
assert not _UNSAFE_DOM_WRITE_RE.search(section), (
"Section 15 must not assign to the unsafe DOM-write property — "
"server names and scope values flow through here and would be "
"XSS-injectable. Use textContent / DOM APIs instead."
)
def test_phase8_settings_button_in_index_html() -> None:
"""The gear-icon entry-point for the settings menu must remain
in the appbar's actions span. The console proxy IIFE prepends a
node pill to ``header.firstChild`` (turnstone/console/server.py:
202); our button is appended inside ``<span class='appbar-actions'>``
on the right, so they don't collide. Pin both shape constraints
here so a future appbar refactor keeps them disjoint."""
body = _INDEX_HTML.read_text(encoding="utf-8")
assert 'id="settings-btn"' in body, (
"index.html must keep the #settings-btn — onclick handlers "
"and the consent badge target it by id."
)
assert 'onclick="toggleSettingsMenu(this)"' in body, (
"settings-btn must wire onclick=toggleSettingsMenu(this) — "
"the gear opens a dropdown with MCP connections + Logout; "
"losing the binding leaves the menu unreachable."
)
# The button must live inside <span class="appbar-actions"> so the
# console proxy's header.insertBefore(pill, header.firstChild)
# leaves it untouched.
actions_open = body.index('class="appbar-actions"')
actions_close = body.index("</span>", actions_open)
assert 'id="settings-btn"' in body[actions_open:actions_close], (
"settings-btn must be inside <span class='appbar-actions'> "
"so the console proxy's firstChild prepend doesn't shift it."
)
def test_settings_menu_handlers_defined() -> None:
"""The gear-icon dropdown exposes a toggle/open/close trio that the
inline ``onclick="toggleSettingsMenu(this)"`` in index.html depends
on, plus the menu items themselves must wire to existing entry
points (``openSettingsPanel`` for MCP connections, ``logout`` for
sign-out). Pin all four so a rename or deletion fails loudly here
instead of silently leaving the gear's menu broken or wired to a
stale function."""
body = _APP_JS.read_text(encoding="utf-8")
for name in [
"function toggleSettingsMenu",
"function openSettingsMenu",
"function closeSettingsMenu",
]:
assert name in body, f"Missing required handler: {name}"
# Bound to the settings-menu region so we don't accidentally match
# an unrelated openSettingsPanel/logout call elsewhere in the file.
start = body.index("function openSettingsMenu(")
end = body.index("function closeSettingsMenu(", start)
section = body[start:end]
assert "openSettingsPanel()" in section, (
"Settings menu's MCP-connections item must call openSettingsPanel() "
"— otherwise the existing settings overlay is unreachable from the "
"new dropdown."
)
assert "logout()" in section, (
"Settings menu's Logout item must call logout() — that's the "
"shared auth.js entry point that clears the cookie + session state."
)
def test_dashboard_overlay_is_region_not_dialog() -> None:
"""The dashboard overlay must be role='region' (not role='dialog' +
aria-modal='true'). The role downgrade is what allows ui-header to
stay interactive while the dashboard is open see the comment at
showDashboard() in app.js. A revert to role='dialog' + aria-modal
would re-trap focus and break the gear/theme buttons + the console
proxy's node-picker pill while the dashboard is open."""
body = _INDEX_HTML.read_text(encoding="utf-8")
idx = body.index('id="dashboard"')
# Bound to ~600 chars after the tag so we only check this element's
# attributes — same shape as test_phase8_settings_modal_in_index_html.
chunk = body[idx : idx + 600]
assert 'role="region"' in chunk, (
"dashboard must be role='region' — see showDashboard() comment."
)
assert "aria-modal" not in chunk, (
"dashboard must NOT be aria-modal — re-trapping focus breaks "
"the appbar's interactive controls (theme toggle, settings menu, "
"proxy node-picker pill) while the dashboard is open."
)
def test_close_settings_menu_resets_aria() -> None:
"""closeSettingsMenu must reset aria-expanded='false' AND remove
aria-controls from the gear trigger. Without the reset the gear
keeps reporting 'expanded' to assistive tech after the menu closes;
without the removal aria-controls points at a dead DOM id."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("function closeSettingsMenu(")
# Bound to ~600 chars so we don't catch unrelated handlers.
section = body[start : start + 600]
assert 'setAttribute("aria-expanded", "false")' in section, (
"closeSettingsMenu must set aria-expanded='false' on the gear."
)
assert 'removeAttribute("aria-controls")' in section, (
"closeSettingsMenu must remove aria-controls from the gear."
)
def test_phase8_settings_modal_in_index_html() -> None:
"""Both the settings overlay and the revoke-confirmation overlay
must remain in the modal area. The Escape-key deferral list in
app.js targets these ids, so removing them silently breaks the
handler chain."""
body = _INDEX_HTML.read_text(encoding="utf-8")
assert 'id="settings-overlay"' in body
assert 'id="revoke-mcp-overlay"' in body
# Each overlay must have role="dialog" + aria-modal="true" so
# screen readers and the existing modal-deferral handlers can
# treat them like the rest of the modal stack.
for overlay_id in ("settings-overlay", "revoke-mcp-overlay"):
idx = body.index(f'id="{overlay_id}"')
# Bound to ~600 chars after the open tag so we only check this
# overlay's attributes.
chunk = body[idx : idx + 600]
assert 'role="dialog"' in chunk, f"{overlay_id} missing role=dialog"
assert 'aria-modal="true"' in chunk, f"{overlay_id} missing aria-modal=true"
def test_phase8_xss_safe_render_in_build_mcp_error_embed() -> None:
"""Adversarial input — the renderer for an MCP error envelope
must use ``textContent`` (not the unsafe DOM-write API) for every
field that flows from the server: ``err.detail``, ``err.server``,
scopes list. The card builder uses createElement + textContent
throughout so a script-tag server name renders harmlessly. Pin
the absence of the unsafe-write inside ``buildMcpErrorEmbed``."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("function buildMcpErrorEmbed(")
# Bound to the function body — find its closing brace at column 0.
rest = body[start:]
# Closing function brace at line start (matches existing functions)
end_match = re.search(r"\n}\n", rest)
assert end_match is not None
fn = rest[: end_match.end()]
assert not _UNSAFE_DOM_WRITE_RE.search(fn), (
"buildMcpErrorEmbed must not use the unsafe-DOM-write API — "
"server names and detail strings flow through here. An "
"adversarial server name must render harmlessly via "
"textContent."
)
def test_phase8_css_classes_present_in_stylesheet() -> None:
"""The card / badge / modal classes referenced from app.js must
have CSS rules. Without them the DOM still works but the visual
treatment is gone, which would silently degrade the consent UX."""
css = _STYLE_CSS.read_text(encoding="utf-8")
for selector in [
".mcp-error-card",
".mcp-error-icon",
".mcp-error-action-btn",
".mcp-scope-pill",
"#settings-overlay",
"#settings-box",
".settings-revoke-btn",
".settings-consent-badge",
"#revoke-mcp-overlay",
]:
assert selector in css, f"Missing CSS rule for {selector}"
def test_phase8_consent_url_prefix_check_in_click_handler() -> None:
"""Defence-in-depth: the consent button's click handler must reject
any ``consent_url`` that doesn't start with the dispatcher's known
prefix (``/v1/api/mcp/oauth/start``). ``_build_consent_url`` always
emits a path-relative URL with that exact prefix; a non-prefix
value implies the producer drifted (or was compromised) and a
``window.open("javascript:...")`` would be catastrophic.
The renderer is the last line of defence before ``window.open`` and
must not rely on the producer-side guarantee alone. Pin the prefix
string and the ``startsWith`` form so a future refactor can't
silently weaken the guard.
"""
body = _APP_JS.read_text(encoding="utf-8")
# Bound the search to the click handler region (between the
# ``buildMcpErrorEmbed`` function and the next top-level helper) to
# avoid false positives from unrelated string occurrences.
start = body.index("function buildMcpErrorEmbed(")
end = body.index("\n}\n", start) + 1
fn = body[start:end]
assert 'consentUrl.startsWith("/v1/api/mcp/oauth/start")' in fn, (
"Click handler must guard window.open with "
'consentUrl.startsWith("/v1/api/mcp/oauth/start"). Without it '
"a future producer drift to a non-path-relative URL (or a "
'"javascript:" injection) would be passed straight to '
"window.open."
)
+57
View File
@@ -83,6 +83,39 @@ class TestIsPublicPath:
def test_shared_static_public(self):
assert is_public_path("/shared/base.css") is True
# Console proxy: a public proxied path must still be public, otherwise
# the login modal can never re-authenticate from inside a ``/node/{id}/``
# proxied page once the cookie expires.
def test_proxy_v1_login_public(self):
assert is_public_path("/node/node-a/v1/api/auth/login") is True
def test_proxy_no_v1_login_public(self):
assert is_public_path("/node/node-a/api/auth/login") is True
def test_proxy_v1_status_public(self):
assert is_public_path("/node/node-a/v1/api/auth/status") is True
def test_proxy_v1_setup_public(self):
assert is_public_path("/node/node-a/v1/api/auth/setup") is True
def test_proxy_v1_logout_public(self):
assert is_public_path("/node/node-a/v1/api/auth/logout") is True
def test_proxy_v1_oidc_authorize_public(self):
assert is_public_path("/node/node-a/v1/api/auth/oidc/authorize") is True
def test_proxy_v1_oidc_callback_public(self):
assert is_public_path("/node/node-a/v1/api/auth/oidc/callback") is True
def test_proxy_v1_workstreams_still_not_public(self):
"""Proxy prefix must not turn protected paths into public ones."""
assert is_public_path("/node/node-a/v1/api/workstreams") is False
def test_proxy_v1_refresh_still_requires_auth(self):
"""Refresh isn't in PUBLIC_PATHS — the caller must already have
a valid cookie. Proxy-prefix shouldn't change that."""
assert is_public_path("/node/node-a/v1/api/auth/refresh") is False
# ---------------------------------------------------------------------------
# TestRequiredRole
@@ -207,6 +240,30 @@ class TestRequiredScope:
"""Only POST is elevated — GET falls through to read."""
assert required_scope("GET", "/api/_internal/mcp-reload") == "read"
def test_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/api/_internal/mcp-refresh/srv") == "approve"
def test_v1_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/v1/api/_internal/mcp-refresh/srv") == "approve"
def test_proxy_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/node/n1/v1/api/_internal/mcp-refresh/srv") == "approve"
def test_proxy_no_v1_internal_mcp_refresh_one_needs_approve(self):
assert required_scope("POST", "/node/n1/api/_internal/mcp-refresh/srv") == "approve"
def test_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/api/_internal/mcp-reconnect/srv") == "approve"
def test_v1_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/v1/api/_internal/mcp-reconnect/srv") == "approve"
def test_proxy_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/node/n1/v1/api/_internal/mcp-reconnect/srv") == "approve"
def test_proxy_no_v1_internal_mcp_reconnect_one_needs_approve(self):
assert required_scope("POST", "/node/n1/api/_internal/mcp-reconnect/srv") == "approve"
# Workstream sub-resource mutations (parametric paths)
def test_ws_delete_needs_write(self):
assert required_scope("POST", "/api/workstreams/abc123/delete") == "write"
+266
View File
@@ -0,0 +1,266 @@
"""Tests for ``turnstone.server._build_history`` reminder + source surfacing.
The replay path (``_build_history``) projects the ``_source`` and
``_reminders`` side-channels onto the wire entry the frontend
consumes. Persisted via migration 050 (Commit 1) so multi-tab /
multi-device replay sees the same metacognitive bubble shape the
originating tab saw live.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
from unittest.mock import patch
from turnstone.server import _build_history
def _make_stub_session(messages: list[dict[str, Any]]) -> Any:
"""Minimal ChatSession-shaped stub. ``_build_history`` only reads
``session.messages`` plus calls ``_load_verdict_indexes(ws_id)``
the latter we patch out below.
"""
return SimpleNamespace(messages=messages, _ws_id="ws-test")
def _build(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Run ``_build_history`` against a stub session, bypassing the
verdicts / output-assessment storage round-trip (no tool_calls in
these tests, so the indexes are unused anyway).
"""
session = _make_stub_session(messages)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
return _build_history(session)
class TestSourceSurfacing:
def test_source_surfaces_when_set(self) -> None:
msg = {
"role": "user",
"content": "",
"_source": "system_nudge",
}
history = _build([msg])
assert len(history) == 1
assert history[0]["source"] == "system_nudge"
def test_source_absent_when_unset(self) -> None:
msg = {"role": "user", "content": "hello"}
history = _build([msg])
assert "source" not in history[0]
class TestRemindersWidening:
def test_watch_triggered_optional_fields_propagate(self) -> None:
"""The widened payload (Commit 2) carries watch_name / command /
poll_count / max_polls / is_final on each ``watch_triggered``
reminder so the frontend renders ``.msg.watch-result``.
"""
msg = {
"role": "user",
"content": "",
"_source": "system_nudge",
"_reminders": [
{
"type": "watch_triggered",
"text": "$ ls\nfile.txt",
"watch_name": "w1",
"command": "ls",
"poll_count": 2,
"max_polls": 100,
"is_final": False,
}
],
}
history = _build([msg])
assert history[0]["source"] == "system_nudge"
assert history[0]["reminders"] == [
{
"type": "watch_triggered",
"text": "$ ls\nfile.txt",
"watch_name": "w1",
"command": "ls",
"poll_count": 2,
"max_polls": 100,
"is_final": False,
}
]
def test_legacy_two_field_reminders_still_work(self) -> None:
"""Producers without optional fields (correction / denial /
idle_children) keep the legacy ``{type, text}`` shape the
widened filter just doesn't add anything beyond that."""
msg = {
"role": "user",
"content": "noted",
"_reminders": [{"type": "correction", "text": "watch out"}],
}
history = _build([msg])
assert history[0]["reminders"] == [{"type": "correction", "text": "watch out"}]
def test_unknown_keys_are_dropped(self) -> None:
"""The wire-layer filter projects on a known set of keys so a
future producer accidentally stuffing arbitrary fields can't
leak them through replay.
"""
msg = {
"role": "user",
"content": "x",
"_reminders": [
{
"type": "correction",
"text": "hi",
"secret": "leak-me",
"internal_id": 42,
}
],
}
history = _build([msg])
clean = history[0]["reminders"][0]
assert "secret" not in clean
assert "internal_id" not in clean
assert clean == {"type": "correction", "text": "hi"}
def test_malformed_reminder_skipped(self) -> None:
"""A non-dict / empty entry is filtered out instead of breaking
the rest of the list (mirrors the defensive filter in
``_apply_reminders_for_provider``).
"""
msg = {
"role": "user",
"content": "x",
"_reminders": [
"garbage string",
{"type": "", "text": ""}, # empty type + text → drop
{"type": "denial", "text": "ok"},
],
}
history = _build([msg])
assert history[0]["reminders"] == [{"type": "denial", "text": "ok"}]
class _StubRegistry:
"""Minimal model registry — only ``get_config`` is read by
``_build_history``."""
def __init__(self, surface_persisted_reasoning: bool = True) -> None:
self._cfg = SimpleNamespace(surface_persisted_reasoning=surface_persisted_reasoning)
def get_config(self, alias: str) -> Any:
return self._cfg
def _build_with_registry(
messages: list[dict[str, Any]],
surface_persisted_reasoning: bool = True,
) -> list[dict[str, Any]]:
session = SimpleNamespace(
messages=messages,
_ws_id="ws-test",
_registry=_StubRegistry(surface_persisted_reasoning=surface_persisted_reasoning),
_model_alias="claude-opus-4-7",
)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
return _build_history(session)
class TestReasoningSurfacing:
"""Phase 1 — surface stored Anthropic thinking blocks on the
history payload so refresh-the-page rehydrates the reasoning bubble.
Drives through the real ``AnthropicProvider`` extractor (no mock-of-
extractor) only the model registry is stubbed.
"""
def test_reasoning_surfaces_for_anthropic_thinking_msg(self) -> None:
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "s"},
{"type": "text", "text": "Final answer."},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert len(history) == 1
assert history[0]["reasoning"] == "let me think"
def test_reasoning_empty_when_persist_flag_false(self) -> None:
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "hidden", "signature": "s"},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=False)
assert "reasoning" not in history[0]
def test_provider_content_never_in_wire_entry(self) -> None:
# The build path does not copy ``_provider_content`` into the
# entry dict regardless of flag — wire payload stays tight.
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "x", "signature": "s"},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert "_provider_content" not in history[0]
def test_no_reasoning_field_when_provider_content_missing(self) -> None:
msg = {"role": "assistant", "content": "plain answer"}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert "reasoning" not in history[0]
def test_no_reasoning_field_for_non_assistant_messages(self) -> None:
# Defensive — user/tool messages with a stray _provider_content
# do not get the reasoning field stamped.
msgs: list[dict[str, Any]] = [
{"role": "user", "content": "hi"},
{
"role": "tool",
"tool_call_id": "c1",
"content": "out",
"_provider_content": [{"type": "thinking", "thinking": "leak", "signature": "s"}],
},
]
history = _build_with_registry(msgs, surface_persisted_reasoning=True)
assert "reasoning" not in history[0]
assert "reasoning" not in history[1]
def test_default_true_when_registry_lookup_raises(self) -> None:
# Conservative default — Phase 1 spec mandates rehydration on
# refresh. A registry/alias mismatch must not silently kill the
# bubble.
class BrokenRegistry:
def get_config(self, alias: str) -> Any:
raise KeyError(alias)
session = SimpleNamespace(
messages=[
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "still works", "signature": "s"}
],
}
],
_ws_id="ws-test",
_registry=BrokenRegistry(),
_model_alias="missing-alias",
)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
history = _build_history(session)
assert history[0]["reasoning"] == "still works"
+112
View File
@@ -19,6 +19,12 @@ class NullUI:
self.infos = []
self.stream_ends = 0
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
@@ -820,3 +826,109 @@ class TestForceCancelThreaded:
assert "idle" in ui.states
assistant_msgs = [m for m in session.messages if m["role"] == "assistant"]
assert any("Fresh response" in m.get("content", "") for m in assistant_msgs)
class TestSynthesizeCancelledResults:
"""Regression coverage for ``_synthesize_cancelled_results`` — must
fire ``on_tool_result`` for each synthesized cancellation so live
SSE listeners (e.g. coord's ``--running`` indicator added by
tool_info) can complete the in-DOM tool batch. Without this, the
coord JS would spin the running indicator forever on cancelled
batches because ``state_change`` doesn't strip ``--running`` from
individual batches."""
def _ui_with_tool_result_tracking(self):
class _TrackingUI(NullUI):
def __init__(self) -> None:
super().__init__()
self.tool_results: list[tuple[str, str, str, bool]] = []
def on_tool_result(self, call_id, name, output, **kwargs):
self.tool_results.append(
(call_id, name, output, bool(kwargs.get("is_error", False))),
)
return _TrackingUI()
def test_synthesizes_tool_result_for_unanswered_calls(self, tmp_db):
ui = self._ui_with_tool_result_tracking()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"content": "calling tools",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
session._synthesize_cancelled_results("Cancelled by user.")
# Both unanswered calls fired ``on_tool_result``.
assert len(ui.tool_results) == 2
ids = {tr[0] for tr in ui.tool_results}
assert ids == {"call_a", "call_b"}
# All emitted as errors so the live UI renders them as
# ``coord-tool-row-result--error``.
assert all(tr[3] is True for tr in ui.tool_results)
# Reason text propagates as the synthetic tool output.
assert all(tr[2] == "Cancelled by user." for tr in ui.tool_results)
# And the message list has the synthesized tool entries
# (preserves the prior contract).
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
assert len(tool_msgs) == 2
def test_skips_calls_already_answered(self, tmp_db):
ui = self._ui_with_tool_result_tracking()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
# call_a already answered.
session.messages.append(
{"role": "tool", "tool_call_id": "call_a", "content": "result"},
)
session._msg_tokens.append(1)
session._synthesize_cancelled_results("Cancelled by user.")
# Only call_b synthesized.
assert len(ui.tool_results) == 1
assert ui.tool_results[0][0] == "call_b"
def test_ui_emit_failure_does_not_break_synthesis(self, tmp_db):
"""The UI hook is wrapped in try/except — a hook failure
during cancel must NOT compound the problem. Synthesis still
appends to messages + storage."""
class _ExplodingUI(NullUI):
def on_tool_result(self, call_id, name, output, **kwargs):
raise RuntimeError("ui hook blew up")
ui = _ExplodingUI()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
# Must not raise.
session._synthesize_cancelled_results("Cancelled by user.")
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
assert len(tool_msgs) == 1
@@ -0,0 +1,66 @@
"""ChatSession interactivity flag tests (Phase 9).
Validates that ``ChatSession._is_interactive_for_consent`` is computed
correctly from ``client_type`` on construction. This is the front of
the Phase 9 plumb-through: the flag flows from here to
``_dispatch_pool_sync`` to the structured-error pending-consent
write path.
"""
from __future__ import annotations
from tests._session_helpers import make_session
from turnstone.prompts import INTERACTIVE_CONSENT_CLIENT_TYPES, ClientType
def test_web_is_interactive() -> None:
s = make_session(client_type=ClientType.WEB)
assert s._is_interactive_for_consent is True
def test_cli_is_interactive() -> None:
s = make_session(client_type=ClientType.CLI)
assert s._is_interactive_for_consent is True
def test_chat_is_not_interactive() -> None:
# Discord / Slack adapters cannot drive a browser redirect from
# inside the channel — consent prompts must be deferred to the
# dashboard badge.
s = make_session(client_type=ClientType.CHAT)
assert s._is_interactive_for_consent is False
def test_scheduled_is_not_interactive() -> None:
# The scheduler runs autonomously; the user isn't online to
# complete the OAuth redirect.
s = make_session(client_type=ClientType.SCHEDULED)
assert s._is_interactive_for_consent is False
def test_interactive_set_matches_module_constant() -> None:
# Pin the module-level frozenset against the flag computation —
# a future reorganisation that drifts the set vs the per-session
# logic would silently break the gating.
for ct in ClientType:
s = make_session(client_type=ct)
assert s._is_interactive_for_consent == (ct in INTERACTIVE_CONSENT_CLIENT_TYPES), ct
def test_default_client_type_is_cli_interactive() -> None:
# Defaults preserved — make_session uses ChatSession's default
# which is CLI. Sanity check that the default user experience
# stays interactive-for-consent.
s = make_session()
assert s._client_type == ClientType.CLI
assert s._is_interactive_for_consent is True
def test_scheduled_env_file_exists() -> None:
"""The SCHEDULED env module must exist; otherwise
``compose_system_message`` for a scheduled session would 500."""
from turnstone.prompts import _load
text = _load("env/scheduled.md")
assert "Output Environment" in text
assert "consent" in text.lower()
+215
View File
@@ -0,0 +1,215 @@
"""Unit tests for :class:`turnstone.core.child_event_bus.ChildEventBus`.
The bus is the in-process wakeup primitive for ``wait_for_workstream``
(see :mod:`turnstone.console.coordinator_client`). It's a small dict
of ws_id set[threading.Event] under a lock focused tests for
register/notify symmetry, no-subscriber notify, multi-waiter fan-out,
multi-child waiter, and concurrent register/notify (smoke). End-to-end
integration with the dispatch sink lives in
``test_coordinator_adapter.py`` and ``test_coordinator_client.py``.
"""
from __future__ import annotations
import threading
import time
import pytest
from turnstone.core.child_event_bus import ChildEventBus
def test_register_returns_event_that_starts_unset() -> None:
"""A waiter must not see leftover state from before it registered —
a fresh wait should always block until the first notify."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
assert isinstance(event, threading.Event)
assert not event.is_set()
def test_notify_wakes_waiter_on_matching_ws_id() -> None:
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify("ws-1")
assert event.is_set()
def test_notify_does_not_wake_waiter_on_unrelated_ws_id() -> None:
"""Different ws_ids must keep independent waiter sets — a notify on
a stranger ws can't wake the wait or the bus stops being keyed."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify("ws-other")
assert not event.is_set()
def test_notify_with_no_subscribers_is_noop() -> None:
"""The dispatch sink calls notify on every translated event; the
steady state has no wait tool active. Must not raise."""
bus = ChildEventBus()
bus.notify("ws-nobody-cares") # no exception
def test_multi_waiter_each_gets_independent_event() -> None:
"""Two waits on the same ws_id must wake independently — clearing
one Event must not silence the other."""
bus = ChildEventBus()
e1 = bus.register_waiter(["ws-1"])
e2 = bus.register_waiter(["ws-1"])
assert e1 is not e2
bus.notify("ws-1")
assert e1.is_set()
assert e2.is_set()
def test_multi_child_waiter_fires_on_any_listed_ws_id() -> None:
"""A wait on [A, B, C] returns a single Event registered against
all three. Notify on ANY of A/B/C must wake the wait the
caller's snapshot re-read disambiguates which one changed."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-a", "ws-b", "ws-c"])
bus.notify("ws-b")
assert event.is_set()
def test_unregister_removes_event_from_all_listed_ws_ids() -> None:
"""After unregister, notify on any of the previously-watched ws_ids
must NOT wake the Event leaks would mean every future notify on
that ws_id wakes a long-dead wait."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-a", "ws-b"])
bus.unregister_waiter(["ws-a", "ws-b"], event)
bus.notify("ws-a")
bus.notify("ws-b")
assert not event.is_set()
def test_unregister_is_idempotent() -> None:
"""A double-unregister must silently no-op — finally blocks may
run twice in odd shutdown paths, the bus must not raise."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.unregister_waiter(["ws-1"], event)
bus.unregister_waiter(["ws-1"], event) # no exception
def test_unregister_pops_empty_buckets() -> None:
"""Empty per-ws_id buckets must be popped so a long-lived bus
doesn't accumulate dead keys after many waits have churned through.
Reaches into the private state the property is structural, not
behavioral, so the assertion is also."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
assert "ws-1" in bus._waiters
bus.unregister_waiter(["ws-1"], event)
assert "ws-1" not in bus._waiters
def test_unregister_keeps_bucket_with_remaining_waiters() -> None:
"""Removing one waiter from a multi-waiter bucket must not drop
the others popping the bucket would silently disable notifies
for every concurrent wait on the same ws_id."""
bus = ChildEventBus()
e1 = bus.register_waiter(["ws-1"])
e2 = bus.register_waiter(["ws-1"])
bus.unregister_waiter(["ws-1"], e1)
bus.notify("ws-1")
assert not e1.is_set()
assert e2.is_set()
def test_empty_and_falsy_ws_ids_are_skipped_on_register() -> None:
"""Defensive: ``wait_for_workstream`` cleans its inputs but the bus
is reachable from other callers in future use; falsy ids should be
silently dropped, not registered against an empty-string key."""
bus = ChildEventBus()
event = bus.register_waiter(["", "ws-1", ""])
# Only the real ws_id should bucket the waiter.
assert list(bus._waiters.keys()) == ["ws-1"]
bus.notify("") # no crash, no spurious wake
assert not event.is_set()
bus.notify("ws-1")
assert event.is_set()
def test_notify_wakes_waiter_blocking_on_event_wait() -> None:
"""End-to-end wake-up latency: a wait blocked on ``Event.wait``
must return promptly after a notify on a watched ws_id. This is
the property that retires the 0.5s polling cadence."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
woken_at = [0.0]
def _waiter() -> None:
event.wait(timeout=2.0)
woken_at[0] = time.monotonic()
t = threading.Thread(target=_waiter, daemon=True)
t.start()
# Give the waiter a beat to enter Event.wait, then notify.
time.sleep(0.05)
notified_at = time.monotonic()
bus.notify("ws-1")
t.join(timeout=1.0)
assert not t.is_alive(), "waiter did not wake within 1s of notify"
# Latency budget is generous; the contract is "well under the legacy
# 0.5s poll cadence", not microsecond timing.
assert woken_at[0] - notified_at < 0.2
def test_clear_before_check_race_does_not_lose_wake() -> None:
"""The wait-loop pattern is ``clear(); snapshot(); ...; wait()``.
A notify between clear and wait must leave the Event set, so the
next wait returns immediately and the loop re-snapshots. Same
standard subscribe/check race the wait loop guards against."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
# Simulate wait-loop ordering: clear, then notify "between" clear
# and the next wait.
event.clear()
bus.notify("ws-1")
# The next wait must return True immediately (set is sticky until
# the next clear).
assert event.wait(timeout=0.1) is True
def test_concurrent_register_and_notify_is_safe() -> None:
"""Smoke test: many threads registering / notifying / unregistering
in parallel must not raise or deadlock. Doesn't assert specific
interleavings only structural safety of the lock discipline."""
bus = ChildEventBus()
stop = threading.Event()
errors: list[BaseException] = []
def _worker(ws_id: str) -> None:
try:
for _ in range(200):
if stop.is_set():
return
ev = bus.register_waiter([ws_id])
bus.notify(ws_id)
bus.unregister_waiter([ws_id], ev)
except BaseException as e: # noqa: BLE001
errors.append(e)
threads = [threading.Thread(target=_worker, args=(f"ws-{i}",), daemon=True) for i in range(8)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5.0)
stop.set()
assert not errors, f"worker threads raised: {errors!r}"
# All buckets should have been popped (every register paired with
# unregister).
assert bus._waiters == {}
@pytest.mark.parametrize("ws_id", ["", None])
def test_notify_silently_ignores_falsy_ws_id(ws_id: object) -> None:
"""Defensive: the dispatch sink already guards against empty
ws_ids, but a falsy slip-through must not raise."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify(ws_id) # type: ignore[arg-type]
assert not event.is_set()
+270
View File
@@ -0,0 +1,270 @@
"""Unit tests for :mod:`turnstone.core.child_source`.
Covers both strategies in isolation against fakes no live collector,
no live SessionManager. Adapter-level integration coverage continues to
live in ``test_coordinator_adapter.py``.
"""
from __future__ import annotations
import contextlib
import time
from typing import TYPE_CHECKING, Any
from turnstone.core.child_source import ClusterChildSource, SameNodeChildSource
from turnstone.core.children_registry import ChildrenRegistry
from turnstone.core.workstream import WorkstreamState
if TYPE_CHECKING:
import queue
# ---------------------------------------------------------------------------
# SameNodeChildSource
# ---------------------------------------------------------------------------
class _FakeManager:
"""Minimal SessionManager stand-in implementing the subscribe API."""
def __init__(self) -> None:
self.subscribers: list[Any] = []
def subscribe_to_state(self, callback: Any) -> None:
self.subscribers.append(callback)
def unsubscribe_from_state(self, callback: Any) -> None:
with contextlib.suppress(ValueError):
self.subscribers.remove(callback)
def fire(self, ws_id: str, state: WorkstreamState) -> None:
for cb in self.subscribers:
cb(ws_id, state)
class TestSameNodeChildSource:
def test_start_subscribes_to_manager(self) -> None:
mgr = _FakeManager()
registry = ChildrenRegistry()
src = SameNodeChildSource(mgr, registry)
sink_calls: list[dict[str, Any]] = []
src.start(sink=sink_calls.append)
assert len(mgr.subscribers) == 1
def test_state_change_for_known_child_pushes_to_sink(self) -> None:
mgr = _FakeManager()
registry = ChildrenRegistry()
registry.install("p1", object())
registry.add_child("p1", "c1")
src = SameNodeChildSource(mgr, registry)
sink_calls: list[dict[str, Any]] = []
src.start(sink=sink_calls.append)
mgr.fire("c1", WorkstreamState.RUNNING)
assert len(sink_calls) == 1
ev = sink_calls[0]
assert ev["type"] == "cluster_state"
assert ev["ws_id"] == "c1"
assert ev["state"] == "running"
def test_state_change_for_unknown_workstream_is_dropped(self) -> None:
mgr = _FakeManager()
registry = ChildrenRegistry()
src = SameNodeChildSource(mgr, registry)
sink_calls: list[dict[str, Any]] = []
src.start(sink=sink_calls.append)
# No registry entry — pre-filter drops the event without
# invoking the sink.
mgr.fire("ws-unknown", WorkstreamState.IDLE)
assert sink_calls == []
def test_shutdown_unsubscribes(self) -> None:
mgr = _FakeManager()
registry = ChildrenRegistry()
src = SameNodeChildSource(mgr, registry)
src.start(sink=lambda ev: None)
assert len(mgr.subscribers) == 1
src.shutdown()
assert mgr.subscribers == []
def test_start_is_idempotent(self) -> None:
mgr = _FakeManager()
registry = ChildrenRegistry()
src = SameNodeChildSource(mgr, registry)
src.start(sink=lambda ev: None)
src.start(sink=lambda ev: None)
# Second start is a no-op; only one subscription.
assert len(mgr.subscribers) == 1
def test_sink_exception_does_not_propagate(self) -> None:
mgr = _FakeManager()
registry = ChildrenRegistry()
registry.install("p1", object())
registry.add_child("p1", "c1")
src = SameNodeChildSource(mgr, registry)
def bad_sink(ev: dict[str, Any]) -> None:
raise RuntimeError("sink boom")
src.start(sink=bad_sink)
# Should not raise — the strategy catches sink failures and logs.
mgr.fire("c1", WorkstreamState.RUNNING)
# ---------------------------------------------------------------------------
# ClusterChildSource
# ---------------------------------------------------------------------------
class _FakeCollector:
"""Minimal ClusterCollector stand-in providing the listener API."""
def __init__(self, snapshot: dict[str, Any] | None = None) -> None:
self._snapshot = snapshot or {"nodes": []}
self.queues: list[queue.Queue[dict[str, Any]]] = []
self.unregistered: list[queue.Queue[dict[str, Any]]] = []
def get_snapshot_and_register(self, q: queue.Queue[dict[str, Any]]) -> dict[str, Any]:
self.queues.append(q)
return self._snapshot
def unregister_listener(self, q: queue.Queue[dict[str, Any]]) -> None:
self.unregistered.append(q)
def emit(self, event: dict[str, Any]) -> None:
"""Push an event to all registered listener queues."""
for q in self.queues:
q.put(event)
class TestClusterChildSource:
def test_start_subscribes_to_collector(self) -> None:
coll = _FakeCollector()
registry = ChildrenRegistry()
src = ClusterChildSource(
collector=coll,
registry=registry,
parents_provider=list,
)
try:
src.start(sink=lambda ev: None)
assert len(coll.queues) == 1
finally:
src.shutdown()
def test_start_primes_registry_from_snapshot(self) -> None:
snapshot = {
"nodes": [
{
"workstreams": [
{"id": "c1", "parent_ws_id": "p1"},
{"id": "c2", "parent_ws_id": "p1"},
# Unknown parent — dropped
{"id": "x", "parent_ws_id": "p-unknown"},
],
},
],
}
coll = _FakeCollector(snapshot)
registry = ChildrenRegistry()
registry.install("p1", object())
src = ClusterChildSource(
collector=coll,
registry=registry,
parents_provider=lambda: ["p1"],
)
try:
src.start(sink=lambda ev: None)
assert set(registry.children_of("p1")) == {"c1", "c2"}
assert registry.parent_for("x") is None
finally:
src.shutdown()
def test_event_dispatched_to_sink(self) -> None:
coll = _FakeCollector()
registry = ChildrenRegistry()
src = ClusterChildSource(
collector=coll,
registry=registry,
parents_provider=list,
)
sink_calls: list[dict[str, Any]] = []
try:
src.start(sink=sink_calls.append)
coll.emit({"type": "cluster_state", "ws_id": "c1", "state": "running"})
# Daemon thread loop has 1.0s queue timeout; poll briefly.
for _ in range(20):
if sink_calls:
break
time.sleep(0.05)
assert len(sink_calls) == 1
assert sink_calls[0]["ws_id"] == "c1"
finally:
src.shutdown()
def test_shutdown_unregisters_and_joins_thread(self) -> None:
coll = _FakeCollector()
registry = ChildrenRegistry()
src = ClusterChildSource(
collector=coll,
registry=registry,
parents_provider=list,
)
src.start(sink=lambda ev: None)
src.shutdown()
assert coll.unregistered == coll.queues
# Second shutdown is a no-op (idempotent).
src.shutdown()
def test_start_is_idempotent(self) -> None:
coll = _FakeCollector()
registry = ChildrenRegistry()
src = ClusterChildSource(
collector=coll,
registry=registry,
parents_provider=list,
)
try:
src.start(sink=lambda ev: None)
src.start(sink=lambda ev: None)
assert len(coll.queues) == 1
finally:
src.shutdown()
def test_sink_exception_does_not_kill_thread(self) -> None:
coll = _FakeCollector()
registry = ChildrenRegistry()
src = ClusterChildSource(
collector=coll,
registry=registry,
parents_provider=list,
)
survived_calls: list[dict[str, Any]] = []
call_count = [0]
def flaky_sink(ev: dict[str, Any]) -> None:
call_count[0] += 1
if call_count[0] == 1:
raise RuntimeError("first one boom")
survived_calls.append(ev)
try:
src.start(sink=flaky_sink)
coll.emit({"type": "cluster_state", "ws_id": "c1", "state": "x"})
coll.emit({"type": "cluster_state", "ws_id": "c2", "state": "y"})
for _ in range(40):
if survived_calls:
break
time.sleep(0.05)
assert len(survived_calls) == 1
assert survived_calls[0]["ws_id"] == "c2"
finally:
src.shutdown()
# Multi-subscriber observer tests for ``SessionManager.subscribe_to_state``
# / ``unsubscribe_from_state`` live in ``test_session_manager.py`` where
# the proper FakeAdapter / FakeStorage construction helpers already exist.
+239
View File
@@ -0,0 +1,239 @@
"""Unit tests for :class:`turnstone.core.children_registry.ChildrenRegistry`.
The registry was lifted from ``CoordinatorAdapter`` in Stage 3 Step 1.
Adapter-level coverage for the integrated behavior already lives in
``test_coordinator_adapter.py``; this file pins the data structure
invariants in isolation so the registry can be reused by future
``ChildSource`` strategies (Step 2) without re-deriving the behavior
from the adapter test surface.
"""
from __future__ import annotations
import threading
import pytest
from turnstone.core.children_registry import ChildrenRegistry
class _Sentinel:
"""Lightweight UI stand-in; identity-comparable, no behavior."""
@pytest.fixture
def registry() -> ChildrenRegistry:
return ChildrenRegistry()
# ---------------------------------------------------------------------------
# install / uninstall
# ---------------------------------------------------------------------------
class TestInstallUninstall:
def test_install_seeds_empty_child_set_and_presence(self, registry: ChildrenRegistry) -> None:
ui = _Sentinel()
registry.install("p1", ui)
assert registry.children_of("p1") == []
assert registry.ui_for("p1") is ui
assert registry.parents() == ["p1"]
def test_install_is_idempotent_repoints_ui_keeps_children(
self, registry: ChildrenRegistry
) -> None:
ui_a = _Sentinel()
ui_b = _Sentinel()
registry.install("p1", ui_a)
registry.merge_children("p1", ["c1", "c2"])
registry.install("p1", ui_b)
assert registry.ui_for("p1") is ui_b
assert set(registry.children_of("p1")) == {"c1", "c2"}
def test_uninstall_clears_forward_reverse_and_presence(
self, registry: ChildrenRegistry
) -> None:
ui = _Sentinel()
registry.install("p1", ui)
registry.merge_children("p1", ["c1", "c2"])
registry.uninstall("p1")
assert registry.children_of("p1") == []
assert registry.ui_for("p1") is None
assert registry.parents() == []
assert registry.parent_for("c1") is None
assert registry.parent_for("c2") is None
def test_uninstall_unknown_parent_is_noop(self, registry: ChildrenRegistry) -> None:
registry.uninstall("never-installed") # must not raise
def test_uninstall_does_not_clobber_other_parents(self, registry: ChildrenRegistry) -> None:
registry.install("p1", _Sentinel())
registry.install("p2", _Sentinel())
registry.merge_children("p1", ["c1"])
registry.merge_children("p2", ["c2"])
registry.uninstall("p1")
assert registry.parent_for("c1") is None
assert registry.parent_for("c2") == "p2"
assert registry.parents() == ["p2"]
# ---------------------------------------------------------------------------
# add_child — atomic check-and-route
# ---------------------------------------------------------------------------
class TestAddChild:
def test_add_child_returns_ui_on_success(self, registry: ChildrenRegistry) -> None:
ui = _Sentinel()
registry.install("p1", ui)
assert registry.add_child("p1", "c1") is ui
assert registry.parent_for("c1") == "p1"
assert registry.children_of("p1") == ["c1"]
def test_add_child_returns_none_when_parent_not_installed(
self, registry: ChildrenRegistry
) -> None:
assert registry.add_child("absent", "c1") is None
assert registry.parent_for("c1") is None
def test_add_child_returns_none_on_duplicate(self, registry: ChildrenRegistry) -> None:
ui = _Sentinel()
registry.install("p1", ui)
assert registry.add_child("p1", "c1") is ui
# second add for same child returns None — caller must not
# double-dispatch.
assert registry.add_child("p1", "c1") is None
assert registry.children_of("p1") == ["c1"]
# ---------------------------------------------------------------------------
# merge_children — bulk seeding
# ---------------------------------------------------------------------------
class TestMergeChildren:
def test_merge_seeds_forward_and_reverse(self, registry: ChildrenRegistry) -> None:
registry.merge_children("p1", ["c1", "c2", "c3"])
assert set(registry.children_of("p1")) == {"c1", "c2", "c3"}
for cid in ("c1", "c2", "c3"):
assert registry.parent_for(cid) == "p1"
def test_merge_is_idempotent(self, registry: ChildrenRegistry) -> None:
registry.merge_children("p1", ["c1"])
registry.merge_children("p1", ["c1"])
assert registry.children_of("p1") == ["c1"]
def test_merge_skips_empty_or_falsy_ids(self, registry: ChildrenRegistry) -> None:
registry.merge_children("p1", ["", "c1", "", "c2"])
assert set(registry.children_of("p1")) == {"c1", "c2"}
def test_merge_does_not_require_install(self, registry: ChildrenRegistry) -> None:
# Snapshot-priming may run before the parent's install fires —
# the merge still seeds the forward set so the install picks
# the children up. (Storage-seeded rebuild relies on this.)
registry.merge_children("p1", ["c1"])
assert registry.children_of("p1") == ["c1"]
# ui_for is still None because install hasn't run
assert registry.ui_for("p1") is None
# ---------------------------------------------------------------------------
# Lookups — return copies, not live refs
# ---------------------------------------------------------------------------
class TestLookups:
def test_children_of_returns_copy(self, registry: ChildrenRegistry) -> None:
registry.install("p1", _Sentinel())
registry.merge_children("p1", ["c1", "c2"])
snap = registry.children_of("p1")
snap.append("c3-injected")
assert "c3-injected" not in registry.children_of("p1")
def test_children_of_unknown_parent_returns_empty(self, registry: ChildrenRegistry) -> None:
assert registry.children_of("absent") == []
def test_parent_for_unknown_child_returns_none(self, registry: ChildrenRegistry) -> None:
assert registry.parent_for("absent") is None
def test_parents_returns_copy(self, registry: ChildrenRegistry) -> None:
registry.install("p1", _Sentinel())
snap = registry.parents()
snap.append("p2-injected")
assert "p2-injected" not in registry.parents()
# ---------------------------------------------------------------------------
# Concurrency — concurrent add_child must not exceed the unique-set
# invariant or leave a half-installed reverse-index entry.
# ---------------------------------------------------------------------------
class TestConcurrency:
def test_concurrent_add_child_returns_ui_exactly_once_per_unique(
self, registry: ChildrenRegistry
) -> None:
ui = _Sentinel()
registry.install("p1", ui)
results: list[object] = []
results_lock = threading.Lock()
def attempt_add(child_id: str) -> None:
r = registry.add_child("p1", child_id)
with results_lock:
results.append(r)
threads = [threading.Thread(target=attempt_add, args=("c1",)) for _ in range(20)]
for t in threads:
t.start()
for t in threads:
t.join()
# Exactly one thread sees the UI; the remaining 19 see None
# (duplicate). The forward + reverse indexes carry exactly one
# entry for c1.
successes = [r for r in results if r is ui]
nones = [r for r in results if r is None]
assert len(successes) == 1
assert len(nones) == 19
assert registry.children_of("p1") == ["c1"]
assert registry.parent_for("c1") == "p1"
def test_concurrent_install_and_add_child_no_resurrect(
self, registry: ChildrenRegistry
) -> None:
# add_child racing with uninstall: either lands first (registry
# populated) or the parent is gone (returns None). Must NOT
# leave a forward-set entry without presence — that would be
# the "resurrected after close" leak the locked dispatch path
# was guarding against.
ui = _Sentinel()
registry.install("p1", ui)
outcomes: list[object] = []
def adder() -> None:
outcomes.append(registry.add_child("p1", "c1"))
def uninstaller() -> None:
registry.uninstall("p1")
threads = [
threading.Thread(target=adder),
threading.Thread(target=uninstaller),
]
for t in threads:
t.start()
for t in threads:
t.join()
# If add_child landed first: c1 is in the forward set, then
# uninstall clears everything. End state: nothing.
# If uninstall landed first: add_child sees no presence,
# returns None, no entry added. End state: nothing.
# Either way, the leak invariant holds: child set is empty or
# parent is gone, never "child set populated but no presence".
children = registry.children_of("p1")
ui_present = registry.ui_for("p1") is not None
if children:
assert ui_present, "registry leaked: children set without presence"
+18 -2
View File
@@ -30,9 +30,25 @@ def _full_hdr() -> dict[str, str]:
}
@pytest.fixture(autouse=True)
def _isolate_metrics(monkeypatch):
"""Swap ``turnstone.server._metrics`` for a fresh collector
per-test, with auto-restore.
Bare ``srv_mod._metrics = MetricsCollector()`` (the prior
pattern) leaks into any test file that already bound the name
via ``from turnstone.server import _metrics`` at import time
those tests' patches then operate on a different instance from
the one the live ``_publish_models_metadata`` reads, and the
monkeypatch silently no-ops. ``monkeypatch.setattr`` restores
after the test, so the leak is contained.
"""
fresh = MetricsCollector()
fresh.model = "test-model"
monkeypatch.setattr(srv_mod, "_metrics", fresh)
def _make_app(storage: Any) -> TestClient:
srv_mod._metrics = MetricsCollector()
srv_mod._metrics.model = "test-model"
mock_session = MagicMock()
mock_ws = MagicMock()
mock_ws.id = "ws-target"
+489 -19
View File
@@ -3,11 +3,13 @@
import asyncio
import json
import queue
from typing import Any
from unittest.mock import MagicMock
import pytest
from turnstone.console.collector import ClusterCollector, NodeSnapshot
from turnstone.console.server import _PROXY_AUTH_LOCAL_HANDLERS
# Shared test auth — JWT-based
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
@@ -149,6 +151,78 @@ class TestCollectorDiscovery:
assert c._nodes["node-a"].started == 1234567890.0
class TestCollectorNotifyWireIn:
"""NotifyDispatcher-driven discovery — reactive node visibility."""
def test_start_subscribes_to_services_channel(self):
# Stub dispatcher records subscriptions without spawning threads.
class _StubDispatcher:
def __init__(self):
self.subscriptions: list[tuple[str, Any]] = []
def subscribe(self, channel, handler):
self.subscriptions.append((channel, handler))
return lambda: None
stub = _StubDispatcher()
storage = MockStorage()
c = ClusterCollector(
storage=storage,
discovery_interval=999,
notify_dispatcher=stub,
)
try:
c.start()
assert len(stub.subscriptions) == 1
channel, handler = stub.subscriptions[0]
assert channel == "services"
assert handler == c._on_services_notify
finally:
c.stop()
def test_no_dispatcher_means_no_subscribe(self):
# Collector without a dispatcher (single-node / SQLite dev) just
# falls back to the 60 s discovery-loop polling — no error.
c = _make_collector(MockStorage())
try:
c.start()
assert c._notify_unsubscribe is None
finally:
c.stop()
def test_on_notify_runs_discovery(self):
# Construct a synthetic Notify and invoke the handler directly —
# asserts the wire-in delegates back to ``_discover_nodes``.
from turnstone.core.storage._notify import Notify
storage = MockStorage()
c = _make_collector(storage)
c._running = True # bypass start() so we don't spawn threads
q: queue.Queue[dict[str, Any]] = queue.Queue()
c.register_listener(q)
storage.services = [
{"service_id": "node-z", "url": "http://z:8080", "metadata": "{}"},
]
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
event = q.get_nowait()
assert event["type"] == "node_joined"
assert event["node_id"] == "node-z"
def test_on_notify_when_not_running_is_noop(self):
# If a stray notify arrives after stop, the handler doesn't run
# discovery on a half-torn-down collector.
from turnstone.core.storage._notify import Notify
storage = MockStorage()
storage.services = [{"service_id": "node-y", "url": "http://y:8080", "metadata": "{}"}]
c = _make_collector(storage)
# _running stays False (never called start()).
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
assert c.get_overview()["nodes"] == 0
class TestCollectorSnapshot:
"""Applying node_snapshot SSE events."""
@@ -314,6 +388,46 @@ class TestCollectorSnapshot:
assert event["ws_id"] == "ws1"
assert event["state"] == "running"
def test_apply_snapshot_state_change_does_not_carry_pending_approval_detail(self):
"""Stage 3 cleanup — the snapshot-resync cluster_state event no
longer piggybacks ``pending_approval_detail`` (the field is
gone from cluster_state entirely). On reconnect the browser's
bulk fetch triggered by the ``activity_state="approval"``
transition in the reducer pulls the items directly from
``ui.serialize_pending_approval_detail()`` via the dashboard
endpoint."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "same", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_snapshot(
"node-a",
{
"type": "node_snapshot",
"node_id": "node-a",
"workstreams": [
{
"id": "ws1",
"name": "same",
"state": "running",
"activity_state": "approval",
}
],
"health": {},
"aggregate": {},
},
)
event = q.get_nowait()
assert event["type"] == "cluster_state"
assert event["activity_state"] == "approval"
assert "pending_approval_detail" not in event
def test_apply_snapshot_skips_empty_id_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
@@ -359,6 +473,36 @@ class TestCollectorDelta:
# Verify in-memory state was updated
assert c._nodes["node-a"].workstreams["ws1"]["state"] == "running"
def test_apply_delta_ws_state_does_not_carry_pending_approval_detail(self):
"""Stage 3 cleanup — ``cluster_state`` no longer carries the
``pending_approval_detail`` piggyback. Approval items now arrive
via bulk fetch on activity_state transition; verdicts via the
explicit ``intent_verdict`` event class. Symmetric event flow,
no piggyback to dedupe against."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "test", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_delta(
"node-a",
{
"type": "ws_state",
"ws_id": "ws1",
"state": "running",
"activity_state": "approval",
},
)
event = q.get_nowait()
assert event["type"] == "cluster_state"
assert event["activity_state"] == "approval"
assert "pending_approval_detail" not in event
def test_apply_delta_ws_created(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
@@ -404,6 +548,139 @@ class TestCollectorDelta:
assert event["name"] == "new-name"
assert c._nodes["node-a"].workstreams["ws1"]["name"] == "new-name"
def test_apply_delta_intent_verdict_forwards_verbatim(self):
"""Stage 3 Step 5 — node-emitted intent_verdict events flow
through _apply_delta to cluster fan-out so coord adapters can
re-emit as child_ws_intent_verdict on the parent's SSE."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "test", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
verdict = {
"call_id": "c1",
"risk_level": "low",
"confidence": 0.9,
"recommendation": "approve",
}
c._apply_delta(
"node-a",
{"type": "intent_verdict", "ws_id": "ws1", "verdict": verdict},
)
event = q.get_nowait()
assert event["type"] == "intent_verdict"
assert event["ws_id"] == "ws1"
assert event["node_id"] == "node-a"
assert event["verdict"] == verdict
def test_apply_delta_intent_verdict_drops_when_ws_id_missing(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_delta("node-a", {"type": "intent_verdict", "verdict": {}})
assert q.empty()
def test_apply_delta_approval_resolved_forwards_verbatim(self):
"""Stage 3 Step 5 — paired with intent_verdict; clears the
coord tree's pending-approval pill in lockstep with the
actual decision rather than waiting for the state-change
piggyback."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "test", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_delta(
"node-a",
{
"type": "approval_resolved",
"ws_id": "ws1",
"approved": True,
"feedback": "lgtm",
"always": False,
},
)
event = q.get_nowait()
assert event["type"] == "approval_resolved"
assert event["ws_id"] == "ws1"
assert event["node_id"] == "node-a"
assert event["approved"] is True
assert event["feedback"] == "lgtm"
assert event["always"] is False
def test_apply_delta_approve_request_forwards_detail(self):
"""Push path for the initial approval items — eliminates the
bulk-fetch race that left the coord row stuck on a loading
placeholder when the bulk fetch landed in the gap between
_emit_state(ATTENTION) and approve_tools setting _pending_approval."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "test", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
detail = {
"type": "approve_request",
"items": [{"call_id": "c1", "header": "tool x"}],
"judge_pending": True,
}
c._apply_delta(
"node-a",
{"type": "approve_request", "ws_id": "ws1", "detail": detail},
)
event = q.get_nowait()
assert event["type"] == "approve_request"
assert event["ws_id"] == "ws1"
assert event["node_id"] == "node-a"
assert event["detail"] == detail
def test_apply_delta_approve_request_drops_when_ws_id_missing(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_delta("node-a", {"type": "approve_request", "detail": {}})
assert q.empty()
def test_apply_delta_approval_resolved_coerces_missing_fields(self):
"""Defensive: ``approved`` / ``always`` / ``feedback`` may be
omitted by older nodes mid-rolling-upgrade; collector coerces
to safe defaults."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "test", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_delta("node-a", {"type": "approval_resolved", "ws_id": "ws1"})
event = q.get_nowait()
assert event["approved"] is False
assert event["feedback"] == ""
assert event["always"] is False
def test_apply_delta_health_changed(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
@@ -1286,6 +1563,142 @@ class TestConsoleProxy:
assert sse_mock.await_count == 1
assert sse_mock.await_args.kwargs.get("use_service_auth") is False
# -------------------------------------------------------------------
# Proxied auth endpoints — handled locally by the console, not
# forwarded to the upstream node. Cases derive directly from
# ``_PROXY_AUTH_LOCAL_HANDLERS`` so a new dispatch entry can't be
# added without a matching test (or vice versa). See proxy_api's
# docstring for the JWT-audience reasoning.
# -------------------------------------------------------------------
@pytest.mark.parametrize(
("method", "path", "handler_name"),
[
(method, path, handler_name)
for (method, path), handler_name in sorted(_PROXY_AUTH_LOCAL_HANDLERS.items())
],
)
def test_proxy_auth_endpoint_dispatches_to_local_handler(
self, client, method, path, handler_name
):
"""Every entry in ``_PROXY_AUTH_LOCAL_HANDLERS`` must route to its
local console handler and never reach the upstream proxy. The
lockout class of bug this dispatch was added to fix is exactly
what a regression here would reintroduce silently covering all
eight branches keeps each path tied to its handler."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
with (
patch(
f"turnstone.console.server.{handler_name}",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "ok"}),
) as local_mock,
patch(
"turnstone.console.server._proxy_post",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as post_mock,
patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as get_mock,
):
resp = client.request(method, f"/node/node-a/v1/api/{path}")
assert resp.status_code == 200
assert local_mock.await_count == 1
assert post_mock.await_count == 0
assert get_mock.await_count == 0
def test_proxy_auth_login_works_without_cookie(self, mock_collector):
"""Without this fix the AuthMiddleware 401s before any handler
runs the user is locked out of the proxied UI once the cookie
expires. Test bypasses _TEST_AUTH_HEADERS to reproduce."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
from starlette.testclient import TestClient
from turnstone.console.server import _load_static, create_app
_load_static()
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
unauth_client = TestClient(app, raise_server_exceptions=False)
try:
with patch(
"turnstone.console.server.auth_login",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "ok"}),
) as local_mock:
resp = unauth_client.post(
"/node/node-a/v1/api/auth/login",
json={"username": "x", "password": "y"},
)
# AuthMiddleware must classify the proxied login path as
# public (is_public_path change) AND proxy_api must
# dispatch to the local handler (proxy_api change).
assert resp.status_code == 200, (
f"login locked out: got {resp.status_code}, body={resp.text}"
)
assert local_mock.await_count == 1
finally:
unauth_client.close()
def test_proxy_auth_wrong_method_returns_405_not_forwarded(self, client):
"""A non-canonical method on an auth path (e.g. PUT on auth/login)
must short-circuit with 405 instead of falling through to the
upstream proxy falling through would forward the request
authenticated as the console's service token (``_proxy_auth_headers``
fallback)."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
with (
patch(
"turnstone.console.server._proxy_post",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as post_mock,
patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as get_mock,
):
# PUT on a POST-only auth path → 405
put_resp = client.put("/node/node-a/v1/api/auth/login")
assert put_resp.status_code == 405
# POST on a GET-only auth path → 405
post_resp = client.post("/node/node-a/v1/api/auth/status")
assert post_resp.status_code == 405
assert post_mock.await_count == 0
assert get_mock.await_count == 0
def test_proxy_non_auth_endpoint_still_forwarded(self, client, mock_collector):
"""Sanity: only auth/* paths intercept. Other API paths still
forward to the upstream node."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
mock_collector.get_node_detail.return_value = {
"node_id": "node-a",
"server_url": "http://a:8080",
"reachable": True,
}
with patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"ok": True}),
) as proxy_mock:
resp = client.get("/node/node-a/v1/api/workstreams")
assert resp.status_code == 200
assert proxy_mock.await_count == 1
# ---------------------------------------------------------------------------
# Proxy URL rewriting unit tests (no HTTP needed)
@@ -1309,11 +1722,62 @@ class TestProxyRewriting:
assert "window.fetch" in _JS_PROXY_SHIM
assert "window.EventSource" in _JS_PROXY_SHIM
def test_console_banner_contains_placeholder(self):
from turnstone.console.server import _CONSOLE_BANNER_TEMPLATE
def test_js_shim_carries_node_id_placeholder(self):
"""The picker reads the current node_id from the shim's _nodeId
closure variable; the placeholder must be present and substitutable."""
from turnstone.console.server import _JS_PROXY_SHIM
assert "NODE_ID_PLACEHOLDER" in _CONSOLE_BANNER_TEMPLATE
assert "Console" in _CONSOLE_BANNER_TEMPLATE
assert "NODE_ID_PLACEHOLDER" in _JS_PROXY_SHIM
replaced = _JS_PROXY_SHIM.replace("NODE_ID_PLACEHOLDER", "node-a")
assert "node-a" in replaced
assert "NODE_ID_PLACEHOLDER" not in replaced
def test_js_shim_includes_picker_pieces(self):
"""Picker logic ships in the same IIFE as the prefix shim — verify
the moving parts are present so a future refactor doesn't silently
drop them. /v1/api/cluster/nodes is the lazy-fetch target;
#ui-header is the DOM anchor; console-node-pill is the trigger
class; ws-tab-dropdown is the menu shell we share with the
workstream chevron menu (style + behaviour parity); ArrowDown is
the keyboard-nav primitive that disambiguates this from a plain
click-only menu."""
from turnstone.console.server import _JS_PROXY_SHIM
# limit=1000 matches the collector's hard cap; without it the
# picker would silently drop nodes past the 100-default in
# clusters with >100 nodes.
assert "/v1/api/cluster/nodes?limit=1000" in _JS_PROXY_SHIM
assert "ui-header" in _JS_PROXY_SHIM
assert "console-node-pill" in _JS_PROXY_SHIM
assert "ws-tab-dropdown" in _JS_PROXY_SHIM
assert "ArrowDown" in _JS_PROXY_SHIM
assert "DOMContentLoaded" in _JS_PROXY_SHIM
def test_proxy_style_drops_banner_styles(self):
"""The legacy banner CSS classes (.console-banner, .ts-header-back-link
offsets, .dashboard-overlay top:32px hack) should be gone the new
picker lives inside #ui-header and doesn't need overlay offsets."""
from turnstone.console.server import _CONSOLE_PROXY_STYLE
assert ".console-banner" not in _CONSOLE_PROXY_STYLE
assert "dashboard-overlay" not in _CONSOLE_PROXY_STYLE
assert ".console-node-pill" in _CONSOLE_PROXY_STYLE
assert ".console-node-menu" in _CONSOLE_PROXY_STYLE
def test_proxy_style_uses_canonical_degraded_color(self):
"""Degraded health dot must use --accent (the canonical "needs
attention" token used by the cluster-overview node table at
console/static/style.css:548) and not --yellow. Yellow is reserved
for the dash-state attention dot, a stronger signal."""
from turnstone.console.server import _CONSOLE_PROXY_STYLE
assert "console-node-menu-item-dot--degraded" in _CONSOLE_PROXY_STYLE
# The degraded rule sits on its own line; assert it uses --accent
# by checking the CSS substring has --accent and not --yellow.
idx = _CONSOLE_PROXY_STYLE.find("console-node-menu-item-dot--degraded")
rule = _CONSOLE_PROXY_STYLE[idx : idx + 200]
assert "var(--accent)" in rule
assert "var(--yellow)" not in rule
def test_html_rewriting_changes_static_paths(self):
"""Simulate the proxy_index rewriting logic."""
@@ -1330,16 +1794,24 @@ class TestProxyRewriting:
assert 'href="/static/' not in rewritten
assert 'src="/static/' not in rewritten
def test_banner_injection_after_body(self):
"""Simulate the banner injection logic."""
from turnstone.console.server import _CONSOLE_BANNER_TEMPLATE
def test_shim_injection_after_body(self):
"""Simulate the proxy shim injection — the shim ships the node-id
and prefix as JS literals and renders the picker at runtime, so
we assert the substituted JS literals land in the page."""
from turnstone.console.server import _CONSOLE_PROXY_STYLE, _JS_PROXY_SHIM
sample_html = "<html><body><div>content</div></body></html>"
banner = _CONSOLE_BANNER_TEMPLATE.replace("NODE_ID_PLACEHOLDER", "node-a")
result = sample_html.replace("<body>", "<body>" + banner, 1)
assert "node-a" in result
assert "Console" in result
assert result.startswith("<html><body><div")
prefix = "/node/node-a"
shim_js = _JS_PROXY_SHIM.replace('"PREFIX_PLACEHOLDER"', json.dumps(prefix)).replace(
'"NODE_ID_PLACEHOLDER"', json.dumps("node-a")
)
injection = _CONSOLE_PROXY_STYLE + "<script>" + shim_js + "</script>"
result = sample_html.replace("<body>", "<body>" + injection, 1)
assert '"node-a"' in result
assert '"/node/node-a"' in result
assert "PREFIX_PLACEHOLDER" not in result
assert "NODE_ID_PLACEHOLDER" not in result
assert result.startswith("<html><body><style>")
# ---------------------------------------------------------------------------
@@ -1574,17 +2046,15 @@ class TestProxySharedStatic:
def test_proxy_shim_injected_in_html(self):
"""Verify shim is injected as inline script in proxied HTML."""
from turnstone.console.server import _CONSOLE_BANNER_TEMPLATE, _JS_PROXY_SHIM
from turnstone.console.server import _JS_PROXY_SHIM
sample_html = "<html><body><div>content</div></body></html>"
prefix = "/node/test-node"
banner = _CONSOLE_BANNER_TEMPLATE.replace("NODE_ID_PLACEHOLDER", "test-node")
shim = (
"<script>"
+ _JS_PROXY_SHIM.replace('"PREFIX_PLACEHOLDER"', json.dumps(prefix))
+ "</script>"
shim_js = _JS_PROXY_SHIM.replace('"PREFIX_PLACEHOLDER"', json.dumps(prefix)).replace(
'"NODE_ID_PLACEHOLDER"', json.dumps("test-node")
)
result = sample_html.replace("<body>", "<body>" + banner + shim, 1)
shim = "<script>" + shim_js + "</script>"
result = sample_html.replace("<body>", "<body>" + shim, 1)
assert "<script>" in result
assert "/node/test-node" in result
assert "window.fetch" in result
+345
View File
@@ -0,0 +1,345 @@
"""``GET /v1/api/models`` resolution-chain coverage.
The console handler resolves four defaults from settings + the enabled
model list:
* ``default_alias`` ``model.default_alias``
* ``channel_default_alias`` ``channels.default_model_alias``
* ``coordinator_default_alias`` ``coordinator.model_alias``, falling
back to ``default_alias`` when empty *or* pointing at a disabled /
removed alias (mirrors :mod:`turnstone.console.session_factory`).
* ``judge_default_alias`` ``judge.model``, falling back to the
resolved coordinator alias when empty *or* pointing at a value that
isn't an enabled alias. ``judge.model`` is alias-only — same
contract as the other model roles and
:class:`turnstone.core.judge.IntentJudge` silently inherits the
session model when an unknown value is configured, so the API
surfaces the resolved coordinator alias rather than echoing the
misconfigured string.
These tests pin each branch so the home composer's resolved-alias
placeholder stays correct as the precedence rules evolve.
"""
from __future__ import annotations
from typing import Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route
from starlette.testclient import TestClient
from tests._coord_test_helpers import _AuthMiddleware, _FakeConfigStore
from turnstone.console.server import list_available_models
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "available_models.db"))
def _seed_model(
storage: SQLiteBackend,
*,
definition_id: str,
alias: str,
model: str = "model-x",
enabled: bool = True,
) -> None:
storage.create_model_definition(
definition_id=definition_id,
alias=alias,
model=model,
provider="openai-compatible",
base_url="http://localhost:8000/v1",
api_key="sk-test",
context_window=8192,
capabilities="{}",
enabled=enabled,
created_by="admin",
)
class _StubRegistry:
"""Mimics the surface ``resolve_coordinator_alias`` reads from
``coord_registry``: ``.default`` and ``.has_alias()``.
Production wires this through ``ModelRegistry``, which in turn
pulls aliases from both DB rows and config.toml. The fixture
mirrors the storage's enabled-row set so ``has_alias()`` agrees
with what the placeholder's enabled-row filter would accept —
without that alignment the helper rejects every tier-2 candidate
and the placeholder goes blank in cases that production handles
fine."""
def __init__(self, *, default: str, known: set[str]) -> None:
self.default = default
self._known = known
def has_alias(self, alias: str) -> bool:
return alias in self._known
def _make_client(
storage: SQLiteBackend,
*,
settings: dict[str, str] | None = None,
registry_default: str = "",
config_store: bool = True,
) -> TestClient:
app = Starlette(
routes=[Route("/v1/api/models", list_available_models)],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.auth_storage = storage
if config_store:
app.state.config_store = _FakeConfigStore(dict(settings or {}))
# ``coord_registry`` is always set in production after lifespan
# startup; mirror that here. ``has_alias`` answers from the same
# enabled-rows set the handler filters against.
enabled = {r["alias"] for r in storage.list_model_definitions(enabled_only=True)}
app.state.coord_registry = _StubRegistry(default=registry_default, known=enabled)
client = TestClient(app)
client.headers.update({"X-Test-User": "admin", "X-Test-Perms": ""})
return client
def _get_models(client: TestClient) -> dict[str, Any]:
resp = client.get("/v1/api/models")
assert resp.status_code == 200, resp.text
return resp.json()
# ---------------------------------------------------------------------------
# Coordinator resolution
# ---------------------------------------------------------------------------
def test_no_settings_leaves_all_defaults_blank(storage: SQLiteBackend) -> None:
"""No model.default_alias, no per-role overrides → every default
field is empty and ``models`` is an empty list."""
body = _get_models(_make_client(storage))
assert body == {
"models": [],
"default_alias": "",
"channel_default_alias": "",
"coordinator_default_alias": "",
"judge_default_alias": "",
}
def test_coordinator_inherits_default_alias_when_unset(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, settings={"model.default_alias": "primary"}))
assert body["default_alias"] == "primary"
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_explicit_enabled_alias_passes_through(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "fast",
},
)
)
assert body["coordinator_default_alias"] == "fast"
def test_coordinator_set_to_disabled_alias_falls_back_to_default(
storage: SQLiteBackend,
) -> None:
"""Operator disabled the alias the coordinator was pinned to —
fall back to the registry default rather than advertising a model
that workstream creation would refuse to use."""
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="legacy", enabled=False)
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "legacy",
},
)
)
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_set_to_unknown_alias_falls_back_to_default(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "ghost",
},
)
)
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_falls_back_to_registry_default_when_config_store_empty(
storage: SQLiteBackend,
) -> None:
"""Match ``console/session_factory.py:109-110``: when both
``coordinator.model_alias`` and ``model.default_alias`` are unset, new
coordinator sessions run on ``registry.default`` (loaded from
config.toml ``[model].default``). The placeholder must report the
same alias rather than going blank otherwise the home composer
advertises "Default model" while sessions actually launch on a
concrete alias."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, registry_default="primary"))
assert body["default_alias"] == ""
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
def test_coordinator_skips_registry_default_when_alias_disabled(
storage: SQLiteBackend,
) -> None:
"""Registry default points at an alias that's been disabled in the DB
the placeholder stays blank rather than advertising a model that
workstream creation would refuse to use."""
_seed_model(storage, definition_id="m1", alias="legacy", enabled=False)
body = _get_models(_make_client(storage, registry_default="legacy"))
assert body["coordinator_default_alias"] == ""
def test_coordinator_falls_back_to_registry_default_when_config_store_missing(
storage: SQLiteBackend,
) -> None:
"""Edge case from PR #500 review: lifespan can leave
``app.state.config_store`` as None (e.g. a startup exception) while
``coord_registry`` still binds successfully. The placeholder must
still advertise ``registry.default`` (filtered against enabled rows)
rather than going blank otherwise the home composer is uselessly
empty in a degraded-but-recoverable state."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, registry_default="primary", config_store=False))
assert body["default_alias"] == ""
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
# ---------------------------------------------------------------------------
# Judge resolution
# ---------------------------------------------------------------------------
def test_judge_empty_inherits_resolved_coordinator_alias(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "fast",
},
)
)
assert body["coordinator_default_alias"] == "fast"
assert body["judge_default_alias"] == "fast"
def test_judge_explicit_enabled_alias_passes_through(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="judge-fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "judge-fast",
},
)
)
assert body["judge_default_alias"] == "judge-fast"
def test_judge_set_to_unknown_value_inherits_coordinator(
storage: SQLiteBackend,
) -> None:
"""``judge.model`` is alias-only — same contract as the other model
roles. An unknown value silently inherits the session model in
:class:`IntentJudge`, so the API surfaces the resolved coordinator
alias rather than echoing the misconfigured string."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "anthropic/claude-haiku-4-5", # raw, not an alias
},
)
)
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
def test_judge_set_to_disabled_alias_inherits_coordinator(
storage: SQLiteBackend,
) -> None:
"""Disabled-alias case is handled identically to the unknown-value
case both trip the alias-not-resolved path."""
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="judge-old", enabled=False)
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "judge-old",
},
)
)
assert body["judge_default_alias"] == "primary"
# ---------------------------------------------------------------------------
# Pre-existing fields stay correct under the new resolution code
# ---------------------------------------------------------------------------
def test_channel_default_alias_blanked_when_disabled(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary", enabled=False)
body = _get_models(
_make_client(
storage,
settings={"channels.default_model_alias": "primary"},
)
)
assert body["channel_default_alias"] == ""
def test_models_payload_strips_secret_fields(storage: SQLiteBackend) -> None:
"""Regression guard: only alias/model/provider land in the response,
never api_key / base_url / context_window / capabilities."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage))
assert body["models"] == [
{"alias": "primary", "model": "model-x", "provider": "openai-compatible"}
]
+306
View File
@@ -0,0 +1,306 @@
"""Tests for the console's coordinator idle-cleanup thread helper.
The helper itself is a tiny loop wrapping ``mgr.close_idle``; the heavy
lifting is in ``SessionManager.close_idle`` (covered in
``test_session_manager.py``) and ``bulk_close_stale_orphans`` (covered
in ``test_storage_sqlite.py``). These tests verify the glue:
- the helper runs an initial sweep BEFORE its first wait (cold-start
cleanup without blocking the lifespan),
- the helper swallows exceptions so a transient DB blip can't kill the
daemon thread,
- the helper exits cleanly when ``stop_event`` is set,
- the helper subscribes to ``mgr.subscribe_to_state`` and a state-change
event wakes the next sweep early (event-driven, not polling),
- the helper unsubscribes when the thread exits so the subscriber
doesn't leak past one cleanup-thread lifetime.
The ``stop_event`` parameter is exclusively for tests production
callers pass ``None`` and the daemon runs for process lifetime.
"""
from __future__ import annotations
import contextlib
import threading
import time
from typing import TYPE_CHECKING
from turnstone.console.server import _coord_idle_cleanup_thread
if TYPE_CHECKING:
from collections.abc import Callable
class _StubMgr:
"""Minimal SessionManager substitute exposing only what the cleanup
thread touches: ``close_idle``, ``subscribe_to_state``,
``unsubscribe_from_state``. Records call ordering for assertions
and lets the test fire state-change events manually via
:meth:`fire_state_change`.
"""
def __init__(
self, *, stop_event: threading.Event, expected_calls: int, raise_after: int = -1
) -> None:
self.calls: list[float] = []
self._stop_event = stop_event
self._expected = expected_calls
self._raise_after = raise_after
self._subscribers: list[Callable[[str, object], None]] = []
self._sub_lock = threading.Lock()
def close_idle(self, timeout_sec: float) -> list[str]:
self.calls.append(timeout_sec)
try:
if 0 <= self._raise_after < len(self.calls):
raise RuntimeError("simulated DB blip")
finally:
# Set stop after the helper has been exercised enough,
# regardless of whether this call raised.
if len(self.calls) >= self._expected:
self._stop_event.set()
return []
def subscribe_to_state(self, callback: Callable[[str, object], None]) -> None:
with self._sub_lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Callable[[str, object], None]) -> None:
with self._sub_lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
@property
def subscribers_count(self) -> int:
with self._sub_lock:
return len(self._subscribers)
def fire_state_change(self, ws_id: str = "ws-x", state: object = "idle") -> None:
with self._sub_lock:
snapshot = list(self._subscribers)
for cb in snapshot:
cb(ws_id, state)
def _run_until_done(mgr: _StubMgr, stop_event: threading.Event, timeout_sec: float) -> None:
# ``min_sweep_interval=0.0`` disables the production cadence floor
# (default 5 s) so tests can fire many close_idle calls back-to-back
# without waiting real time between them. The floor is exercised
# in its own dedicated test below.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, timeout_sec, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "helper failed to exit on stop_event"
def test_coord_idle_cleanup_runs_initial_sweep_before_wait() -> None:
"""The first close_idle call must happen BEFORE the first wait —
otherwise cold-start orphans wait one ``check_every`` interval (~30 min
on default 2h timeout) for the first reap. Crucial because the
lifespan no longer does a synchronous initial sweep.
Verified structurally: a single ``expected_calls=1`` run completes
in well under one ``check_every`` (here 0.04 s timeout 0.01 s
check_every), so the initial sweep must have happened before any
real wait could have blocked it.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
started = time.monotonic()
_run_until_done(mgr, stop_event, timeout_sec=0.04)
elapsed = time.monotonic() - started
assert len(mgr.calls) == 1
# check_every = min(300.0, 0.04/4) = 0.01 s. An initial sweep
# gated behind one full wait would have taken ~0.01+ s anyway, so
# the upper bound here is "much less than one check_every plus
# process noise" — the explicit 1.0 s gives generous CI headroom
# while still asserting the test is testing the right thing.
assert elapsed < 1.0
def test_coord_idle_cleanup_calls_close_idle_each_tick() -> None:
"""Heartbeat path: with no state-change events, close_idle fires
each ``check_every`` interval. Test uses a tiny timeout so the
test runs fast the contract under test is "the loop iterates",
not the production cadence.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert len(mgr.calls) == 3
assert all(t == 0.04 for t in mgr.calls)
def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
"""A transient DB error must not kill the daemon thread — the next
tick should still fire close_idle. Without the try/except, a single
blip would silently leak orphans forever."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=4, raise_after=1)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
# All four calls must have fired despite calls 2-4 raising.
assert len(mgr.calls) == 4
def test_coord_idle_cleanup_exits_cleanly_on_stop_event() -> None:
"""The stop_event mechanism is the test contract; verify the thread
actually exits when the event is set, without needing exceptions or
daemon-process termination."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert stop_event.is_set()
def test_state_change_wakes_close_idle_before_heartbeat() -> None:
"""The event-driven path is the whole point of the refactor: a
workstream state-change must wake the cleanup sweep without
waiting one ``check_every`` interval. Tested with a long
timeout_sec so the heartbeat would NOT have fired in the test
window the close_idle call past the initial sweep must come
from a state-change wake.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
# check_every = min(300.0, 120.0/4) = 30 s — well outside the test
# window. Any close_idle call past the initial sweep must come
# from a fire_state_change-driven wake-up.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
# Wait for the initial sweep to complete AND the thread to enter
# its first ``tick_now.wait`` (signalled here by the subscriber
# being registered + calls advancing to 1).
deadline = time.monotonic() + 1.0
while time.monotonic() < deadline:
if mgr.subscribers_count == 1 and len(mgr.calls) >= 1:
break
time.sleep(0.01)
assert mgr.subscribers_count == 1, "thread didn't subscribe to state"
assert len(mgr.calls) == 1, "initial sweep didn't fire"
# One state-change fire wakes the first ``wait`` → close_idle runs
# again → stop_event is set (expected_calls=2) → thread exits.
mgr.fire_state_change()
thread.join(timeout=2.0)
assert not thread.is_alive(), "thread didn't exit after state-change-driven sweep"
# 2 = initial + state-change-driven. If the state change weren't
# being honoured, close_idle would have stalled on the 30 s wait
# and the thread.join would have timed out.
assert len(mgr.calls) == 2
def test_subscriber_unregisters_when_thread_exits() -> None:
"""The cleanup thread's state-change subscriber must be removed
when the thread exits otherwise long-running processes that
restart their cleanup threads (admin model-CRUD path, tests) leak
subscribers and every state change fires N stale callbacks.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert mgr.subscribers_count == 0, "subscriber leaked past thread exit"
def test_state_change_during_close_idle_triggers_followup_sweep() -> None:
"""A state-change fired during the initial sweep (e.g. close_idle's
own ``close()`` calls firing subscribers) must wake the next
``tick_now.wait`` rather than being lost to the clear-before-sweep
ordering. The clear runs INSIDE the loop just before close_idle,
so a fire during the initial sweep which precedes the loop
arrives at an already-set event that the first wait sees set and
returns on immediately.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
real_close_idle = mgr.close_idle
# One-shot fire during the initial sweep, mirroring what
# close_idle's own close() calls do in production (set_state →
# state-change subscribers).
fired = [False]
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
result = real_close_idle(timeout_sec)
if not fired[0]:
fired[0] = True
mgr.fire_state_change()
return result
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "thread blocked on the next wait — mid-sweep wake was lost"
# 2 = initial sweep + state-change-driven follow-up. Without the
# event surviving the clear-before-sweep ordering, the thread
# would have blocked on the 30 s ``wait`` and the test would have
# timed out at thread.join.
assert len(mgr.calls) == 2
def test_min_sweep_interval_floors_close_idle_cadence_under_sustained_wakes() -> None:
"""Cadence floor: even when state-change events keep firing
``tick_now.set()``, ``close_idle`` must not run more often than
``min_sweep_interval`` otherwise the loop tight-spins close_idle
at the rate of its own DB latency, doing 600-1500x more DB work
than the pre-refactor fixed-30 s cadence.
Wires a state-change subscriber that fires another state change
from inside close_idle, so the bus would tick forever if not
floored. Asserts the elapsed-between-sweeps is at least
``min_sweep_interval`` modulo small wall-clock noise.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
real_close_idle = mgr.close_idle
sweep_times: list[float] = []
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
sweep_times.append(time.monotonic())
result = real_close_idle(timeout_sec)
# Always fire another state-change to simulate sustained
# activity (each turn fires thinking/running/attention/idle).
# If the floor were absent, the next wake would race the next
# close_idle immediately and ``sweep_times`` deltas would be
# bounded by close_idle latency (microseconds), not the floor.
mgr.fire_state_change()
return result
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
# 0.15 s floor keeps the test fast (~0.3 s total) while still
# representing a meaningful gap relative to close_idle's
# near-zero stub latency.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.15},
daemon=True,
)
thread.start()
thread.join(timeout=3.0)
assert not thread.is_alive(), "thread didn't exit"
assert len(sweep_times) >= 2, "fewer than two sweeps fired"
# Gap between sweep 1 (post-initial) and sweep 2 must respect
# the floor. Initial sweep at sweep_times[0] is unfloored
# (no prior sweep to compare against), so the meaningful
# assertion is on sweep_times[1] - sweep_times[0].
gap = sweep_times[1] - sweep_times[0]
assert gap >= 0.12, f"floor breached: gap {gap:.3f}s < min_sweep_interval 0.15s"
+219
View File
@@ -0,0 +1,219 @@
"""``console/session_factory.py`` alias-resolution coverage.
The console session factory resolves the coordinator alias through a
three-tier chain that must stay in lockstep with the placeholder logic
in ``console/server.py:list_available_models`` otherwise the home
composer advertises one alias while sessions launch on another.
Tier order (highest priority first):
1. Per-call ``model_alias`` arg, or the ``coordinator.model_alias``
ConfigStore setting (admin-pinned coordinator-specific override).
2. ``model.default_alias`` ConfigStore setting (admin-managed system
default surfaced in the Models tab).
3. ``registry.default`` (config.toml ``[model].default``, the boot-time
fallback).
These tests pin each branch by intercepting ``registry.resolve``
they short-circuit before ChatSession construction so the test never
has to satisfy ChatSession's full kwarg contract.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
import pytest
from tests._coord_test_helpers import _FakeConfigStore
from turnstone.console.session_factory import build_console_session_factory
class _StopBeforeChatSessionError(Exception):
"""Sentinel raised by the capturing registry to short-circuit
factory execution after alias resolution but before ChatSession is
built. The factory's outer code path is irrelevant to alias
resolution and would force the test to satisfy a long kwarg
contract for no extra coverage."""
class _CapturingRegistry:
"""Records the alias passed to ``resolve()`` and short-circuits.
``has_alias`` answers from the configured known set so the
``model.default_alias`` validation tier behaves realistically.
Mirrors the public surface ``ModelRegistry`` exposes to
session_factory: ``has_alias``, ``resolve``, and ``default``.
"""
def __init__(self, *, default: str, known: set[str]) -> None:
self.default = default
self._known = known
self.captured_alias: str | None = None
def has_alias(self, alias: str) -> bool:
return alias in self._known
def resolve(self, alias: str) -> Any:
self.captured_alias = alias
raise _StopBeforeChatSessionError()
def _build_factory(
*,
registry_default: str = "registry-default",
known_aliases: set[str] | None = None,
settings: dict[str, Any] | None = None,
) -> tuple[Any, _CapturingRegistry]:
"""Construct the factory with stub deps. Returns ``(factory_callable,
registry)`` so tests can read back ``registry.captured_alias``."""
registry = _CapturingRegistry(
default=registry_default,
known=known_aliases if known_aliases is not None else {registry_default},
)
config_store = _FakeConfigStore(dict(settings or {}))
factory = build_console_session_factory(
registry=registry, # type: ignore[arg-type]
config_store=config_store, # type: ignore[arg-type]
node_id="console",
coord_client_factory=lambda ws_id, uid: MagicMock(),
)
return factory, registry
def _invoke(factory: Any, **factory_kwargs: Any) -> None:
"""Call the factory with a stub UI and absorb the sentinel.
Forwards ``factory_kwargs`` to the factory so per-call overrides
(e.g. ``model_alias``) can flow through. Raises if any other
exception comes out the test should fail loudly when alias
resolution itself errors rather than swallowing it.
"""
ui = MagicMock()
ui._user_id = "" # skip storage-backed username lookup branch
with pytest.raises(_StopBeforeChatSessionError):
factory(ui, **factory_kwargs)
# ---------------------------------------------------------------------------
# Tier 1 — explicit pin (per-call arg or coordinator.model_alias)
# ---------------------------------------------------------------------------
def test_per_call_model_alias_arg_wins_over_everything() -> None:
"""The ``model_alias`` kwarg on the factory call (e.g. body field on
POST /workstreams/new) wins over both ConfigStore tiers and the
registry default."""
factory, registry = _build_factory(
known_aliases={"per-call", "coord-pin", "admin-default", "registry-default"},
settings={
"coordinator.model_alias": "coord-pin",
"model.default_alias": "admin-default",
},
)
_invoke(factory, model_alias="per-call")
assert registry.captured_alias == "per-call"
def test_coordinator_model_alias_wins_when_no_per_call_override() -> None:
factory, registry = _build_factory(
known_aliases={"coord-pin", "admin-default", "registry-default"},
settings={
"coordinator.model_alias": "coord-pin",
"model.default_alias": "admin-default",
},
)
_invoke(factory)
assert registry.captured_alias == "coord-pin"
def test_coordinator_model_alias_passed_through_unvalidated() -> None:
"""Tier 1 is an *explicit* operator pin — when it's stale or typoed
we deliberately pass it through to ``registry.resolve`` so the
request layer turns it into a 503 with the alias surfaced in the
error. Falling through silently would mask the misconfiguration."""
factory, registry = _build_factory(
known_aliases={"admin-default", "registry-default"},
settings={
"coordinator.model_alias": "ghost", # unknown
"model.default_alias": "admin-default",
},
)
_invoke(factory)
assert registry.captured_alias == "ghost"
def test_per_call_model_alias_arg_passed_through_unvalidated() -> None:
"""The per-call ``model_alias`` kwarg (POST body field — the more
common production trigger) is the same kind of explicit pin as the
ConfigStore setting, so a stale value passes through to
``registry.resolve`` rather than silently falling through to the
system default."""
factory, registry = _build_factory(
known_aliases={"registry-default"},
settings={"model.default_alias": "registry-default"},
)
_invoke(factory, model_alias="ghost")
assert registry.captured_alias == "ghost"
# ---------------------------------------------------------------------------
# Tier 2 — model.default_alias (admin-managed system default)
# ---------------------------------------------------------------------------
def test_model_default_alias_used_when_coordinator_unset() -> None:
"""Regression for the historical drift: admin sets the system
default in the Models tab, the home composer advertises it, and new
coordinator sessions must launch on the same alias rather than
silently falling through to ``registry.default``."""
factory, registry = _build_factory(
known_aliases={"admin-default", "registry-default"},
settings={"model.default_alias": "admin-default"},
)
_invoke(factory)
assert registry.captured_alias == "admin-default"
def test_unknown_model_default_alias_falls_through_to_registry_default() -> None:
"""Tier 2 is *not* an explicit pin — operators set
``model.default_alias`` once in the UI and forget about it; an alias
that's later disabled or typo'd should not 503 the coordinator,
since tier 3 (``registry.default``) is guaranteed to resolve."""
factory, registry = _build_factory(
known_aliases={"registry-default"}, # admin-default got removed
settings={"model.default_alias": "admin-default"},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
def test_blank_model_default_alias_falls_through_to_registry_default() -> None:
factory, registry = _build_factory(
settings={"model.default_alias": ""},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
# ---------------------------------------------------------------------------
# Tier 3 — registry.default (config.toml [model].default)
# ---------------------------------------------------------------------------
def test_no_settings_uses_registry_default() -> None:
factory, registry = _build_factory()
_invoke(factory)
assert registry.captured_alias == "registry-default"
def test_whitespace_only_coord_alias_falls_through() -> None:
"""``" "`` is not an explicit pin — ``.strip()`` reduces it to
"", which the chain should treat as unset."""
factory, registry = _build_factory(
settings={"coordinator.model_alias": " "},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
+145
View File
@@ -464,3 +464,148 @@ def test_coord_budget_override_survives_wildcard_allow_policy() -> None:
assert approved is True
types = [e.get("type") for e in captured_events]
assert "approve_request" in types, "Wildcard allow must not strip the budget-override prompt"
# ---------------------------------------------------------------------------
# Cluster-bus broadcast hooks — _broadcast_intent_verdict / _approval_resolved
# ---------------------------------------------------------------------------
class TestBroadcastIntentVerdict:
"""``ConsoleCoordinatorUI._broadcast_intent_verdict`` overrides the
no-op base hook to push the verdict onto the cluster bus via
``ClusterCollector.emit_console_ws_intent_verdict``. The far more
common path is the per-node ``WebUI`` override (covered in
test_webui_content.py); this lights up the rare coord-self path
(a coord that runs its own LLM judge).
"""
def test_calls_collector_emit_with_ws_id_and_verdict(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
ConsoleCoordinatorUI._collector = collector
try:
verdict = {
"call_id": "c1",
"risk_level": "high",
"confidence": 0.91,
}
ui._broadcast_intent_verdict(verdict)
collector.emit_console_ws_intent_verdict.assert_called_once_with(
"coord-a",
verdict,
)
finally:
ConsoleCoordinatorUI._collector = None
def test_no_op_when_collector_unset(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
ConsoleCoordinatorUI._collector = None
# Doesn't raise.
ui._broadcast_intent_verdict({"call_id": "c1"})
def test_collector_exception_swallowed(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
collector.emit_console_ws_intent_verdict.side_effect = RuntimeError("boom")
ConsoleCoordinatorUI._collector = collector
try:
# Doesn't raise — collector failures are observational only.
ui._broadcast_intent_verdict({"call_id": "c1"})
finally:
ConsoleCoordinatorUI._collector = None
class TestBroadcastApprovalResolved:
"""``ConsoleCoordinatorUI._broadcast_approval_resolved`` overrides
the base hook to push the resolution onto the cluster bus via
``ClusterCollector.emit_console_ws_approval_resolved``."""
def test_calls_collector_with_decision_fields(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
ConsoleCoordinatorUI._collector = collector
try:
ui._broadcast_approval_resolved(True, "lgtm", always=True)
collector.emit_console_ws_approval_resolved.assert_called_once_with(
"coord-a",
approved=True,
feedback="lgtm",
always=True,
)
finally:
ConsoleCoordinatorUI._collector = None
def test_normalises_none_feedback_to_empty_string(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
ConsoleCoordinatorUI._collector = collector
try:
ui._broadcast_approval_resolved(False, None)
collector.emit_console_ws_approval_resolved.assert_called_once_with(
"coord-a",
approved=False,
feedback="",
always=False,
)
finally:
ConsoleCoordinatorUI._collector = None
def test_no_op_when_collector_unset(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
ConsoleCoordinatorUI._collector = None
# Doesn't raise.
ui._broadcast_approval_resolved(True, None)
def test_collector_exception_swallowed(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
collector.emit_console_ws_approval_resolved.side_effect = RuntimeError("boom")
ConsoleCoordinatorUI._collector = collector
try:
# Doesn't raise.
ui._broadcast_approval_resolved(True, "ok")
finally:
ConsoleCoordinatorUI._collector = None
class TestBroadcastApproveRequest:
"""Coord-side override for the approve_request push. Same rationale
as the WebUI override the coord-self path is rare today, but
parity keeps the override symmetric with the rest of the broadcast
family."""
def test_calls_collector_emit_with_ws_id_and_detail(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
ConsoleCoordinatorUI._collector = collector
try:
detail = {
"type": "approve_request",
"items": [{"call_id": "c1", "header": "tool x"}],
"judge_pending": True,
}
ui._broadcast_approve_request(detail)
collector.emit_console_ws_approve_request.assert_called_once_with(
"coord-a",
detail,
)
finally:
ConsoleCoordinatorUI._collector = None
def test_no_op_when_collector_unset(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
ConsoleCoordinatorUI._collector = None
# Doesn't raise.
ui._broadcast_approve_request({"items": []})
def test_collector_exception_swallowed(self) -> None:
ui = ConsoleCoordinatorUI(ws_id="coord-a", user_id="u1")
collector = MagicMock()
collector.emit_console_ws_approve_request.side_effect = RuntimeError("boom")
ConsoleCoordinatorUI._collector = collector
try:
# Doesn't raise.
ui._broadcast_approve_request({"items": []})
finally:
ConsoleCoordinatorUI._collector = None
+283 -46
View File
@@ -378,13 +378,21 @@ class TestCoordinatorAdapterWorkerDispatch:
class TestCoordinatorAdapterChildrenRegistry:
def test_emit_created_seeds_empty_children_set(self) -> None:
"""Adapter-level integration with :class:`ChildrenRegistry`.
Pure-registry invariants (forward/reverse consistency, idempotent
merge, locking) live in ``test_children_registry.py``. These
tests cover the adapter's wiring: that ``emit_*`` paths drive the
registry correctly and that the snapshot-priming bridge between
a collector snapshot and the registry preserves merge semantics.
"""
def test_emit_created_installs_parent(self) -> None:
adapter, _ = _make_adapter()
ws = _make_ws()
adapter.emit_created(ws)
assert ws.id in adapter._children
assert adapter._children[ws.id] == set()
assert adapter._active_coords[ws.id] is ws.ui
assert adapter._registry.children_of(ws.id) == []
assert adapter._registry.ui_for(ws.id) is ws.ui
def test_emit_rehydrated_calls_rebuild(self) -> None:
adapter, _ = _make_adapter()
@@ -398,42 +406,36 @@ class TestCoordinatorAdapterChildrenRegistry:
adapter.emit_rehydrated(ws)
assert calls == [ws.id]
def test_emit_closed_clears_forward_and_reverse_indexes(self) -> None:
def test_emit_closed_uninstalls_parent_and_clears_children(self) -> None:
adapter, _ = _make_adapter()
with adapter._children_lock:
adapter._merge_child_ids_locked("coord-a", ["child-a1", "child-a2"])
adapter._merge_child_ids_locked("coord-b", ["child-b1"])
adapter._active_coords["coord-a"] = object()
adapter._active_coords["coord-b"] = object()
adapter._registry.install("coord-a", object())
adapter._registry.install("coord-b", object())
adapter._registry.merge_children("coord-a", ["child-a1", "child-a2"])
adapter._registry.merge_children("coord-b", ["child-b1"])
adapter.emit_closed("coord-a")
assert "coord-a" not in adapter._children
assert "coord-a" not in adapter._active_coords
assert "child-a1" not in adapter._child_to_coord
assert "child-a2" not in adapter._child_to_coord
assert adapter._registry.ui_for("coord-a") is None
assert adapter._registry.children_of("coord-a") == []
assert adapter._registry.parent_for("child-a1") is None
assert adapter._registry.parent_for("child-a2") is None
# coord-b untouched
assert adapter._child_to_coord["child-b1"] == "coord-b"
assert "coord-b" in adapter._children
def test_merge_child_ids_locked_is_idempotent(self) -> None:
adapter, _ = _make_adapter()
with adapter._children_lock:
adapter._merge_child_ids_locked("coord-a", ["child-1"])
adapter._merge_child_ids_locked("coord-a", ["child-1"])
assert adapter._children["coord-a"] == {"child-1"}
assert adapter._child_to_coord == {"child-1": "coord-a"}
assert adapter._registry.parent_for("child-b1") == "coord-b"
assert adapter._registry.ui_for("coord-b") is not None
def test_prime_children_from_snapshot_merges_without_overwriting(self) -> None:
# Snapshot priming now lives on ClusterChildSource (production
# path). The adapter no longer carries its own duplicate copy.
from turnstone.core.child_source import ClusterChildSource
adapter, _ = _make_adapter()
# Seed one in-memory coord + one existing child
coord_ws = _make_ws()
coord_ws.id = "coord-a"
mgr = MagicMock()
mgr.list_all.return_value = [coord_ws]
adapter.attach(mgr)
with adapter._children_lock:
adapter._merge_child_ids_locked("coord-a", ["child-a1"])
adapter._registry.merge_children("coord-a", ["child-a1"])
source = ClusterChildSource(
collector=MagicMock(),
registry=adapter._registry,
parents_provider=lambda: ["coord-a"],
)
snapshot = {
"nodes": [
@@ -448,10 +450,13 @@ class TestCoordinatorAdapterChildrenRegistry:
},
],
}
adapter._prime_children_from_snapshot(snapshot)
assert adapter._children["coord-a"] == {"child-a1", "child-a2"}
assert adapter._child_to_coord["child-a2"] == "coord-a"
assert "child-x" not in adapter._child_to_coord
source._prime_from_snapshot(snapshot)
assert set(adapter._registry.children_of("coord-a")) == {
"child-a1",
"child-a2",
}
assert adapter._registry.parent_for("child-a2") == "coord-a"
assert adapter._registry.parent_for("child-x") is None
# ---------------------------------------------------------------------------
@@ -478,9 +483,7 @@ class TestCoordinatorAdapterDispatchChildEvent:
coord_ws.id = coord_id
recorder = _UIRecorder()
coord_ws.ui = recorder # type: ignore[assignment]
with adapter._children_lock:
adapter._children.setdefault(coord_id, set())
adapter._active_coords[coord_id] = recorder
adapter._registry.install(coord_id, recorder)
adapter.attach(_StubManager(coord_ws)) # type: ignore[arg-type]
return adapter, recorder, coord_ws
@@ -510,12 +513,11 @@ class TestCoordinatorAdapterDispatchChildEvent:
assert payload["child_ws_id"] == "child-a1"
assert payload["parent_ws_id"] == "coord-a"
# Reverse index updated for subsequent cluster_state events.
assert adapter._child_to_coord["child-a1"] == "coord-a"
assert adapter._registry.parent_for("child-a1") == "coord-a"
def test_dispatch_cluster_state_routes_via_reverse_index(self) -> None:
adapter, recorder, _ = self._setup()
with adapter._children_lock:
adapter._merge_child_ids_locked("coord-a", ["child-a1"])
adapter._registry.merge_children("coord-a", ["child-a1"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
@@ -533,8 +535,7 @@ class TestCoordinatorAdapterDispatchChildEvent:
def test_dispatch_ws_closed_routes_to_parent_coord(self) -> None:
adapter, recorder, _ = self._setup()
with adapter._children_lock:
adapter._merge_child_ids_locked("coord-a", ["child-a1"])
adapter._registry.merge_children("coord-a", ["child-a1"])
adapter._dispatch_child_event(
{"type": "ws_closed", "ws_id": "child-a1", "reason": "evicted"}
)
@@ -548,8 +549,7 @@ class TestCoordinatorAdapterDispatchChildEvent:
"""perf-6: _enqueue_on_ui mutates the payload dict in place with
the coord's ws_id so the browser can discriminate child events."""
adapter, recorder, _ = self._setup()
with adapter._children_lock:
adapter._merge_child_ids_locked("coord-a", ["child-a1"])
adapter._registry.merge_children("coord-a", ["child-a1"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
@@ -558,3 +558,240 @@ class TestCoordinatorAdapterDispatchChildEvent:
}
)
assert recorder.enqueued[0]["ws_id"] == "coord-a"
def test_dispatch_cluster_state_does_not_carry_pending_approval_detail(
self,
) -> None:
"""Stage 3 cleanup — the ``pending_approval_detail`` piggyback
on ``cluster_state`` is gone. Approval items now arrive via
bulk fetch (triggered by ``activity_state="approval"`` in the
browser); verdicts via ``child_ws_intent_verdict``; resolution
via ``child_ws_approval_resolved``. The state event carries
only state + activity_state no detail field."""
adapter, recorder, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "running",
"activity_state": "approval",
}
)
assert len(recorder.enqueued) == 1
payload = recorder.enqueued[0]
assert payload["type"] == "child_ws_state"
assert payload["activity_state"] == "approval"
assert "pending_approval_detail" not in payload
def test_dispatch_intent_verdict_emits_child_ws_intent_verdict(self) -> None:
"""Stage 3 Step 6 — explicit verdict events are re-emitted as
child_ws_intent_verdict on the parent's SSE so the tree UI
renders the risk pill without polling."""
adapter, recorder, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
verdict = {
"call_id": "c1",
"risk_level": "low",
"confidence": 0.92,
"recommendation": "approve",
}
adapter._dispatch_child_event(
{
"type": "intent_verdict",
"ws_id": "child-a1",
"node_id": "node-1",
"verdict": verdict,
}
)
assert len(recorder.enqueued) == 1
payload = recorder.enqueued[0]
assert payload["type"] == "child_ws_intent_verdict"
assert payload["child_ws_id"] == "child-a1"
assert payload["parent_ws_id"] == "coord-a"
assert payload["node_id"] == "node-1"
assert payload["verdict"] == verdict
def test_dispatch_intent_verdict_unknown_child_drops(self) -> None:
adapter, recorder, _ = self._setup()
adapter._dispatch_child_event(
{
"type": "intent_verdict",
"ws_id": "ws-orphan",
"verdict": {"call_id": "c1"},
}
)
assert recorder.enqueued == []
def test_dispatch_approval_resolved_emits_child_ws_approval_resolved(
self,
) -> None:
"""Stage 3 Step 6 — paired with intent_verdict; clears the
pending-approval pill on the parent's tree UI in lockstep
with the actual decision."""
adapter, recorder, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
adapter._dispatch_child_event(
{
"type": "approval_resolved",
"ws_id": "child-a1",
"node_id": "node-1",
"approved": True,
"feedback": "lgtm",
"always": False,
}
)
assert len(recorder.enqueued) == 1
payload = recorder.enqueued[0]
assert payload["type"] == "child_ws_approval_resolved"
assert payload["child_ws_id"] == "child-a1"
assert payload["parent_ws_id"] == "coord-a"
assert payload["approved"] is True
assert payload["feedback"] == "lgtm"
assert payload["always"] is False
def test_dispatch_approval_resolved_coerces_missing_fields(self) -> None:
"""Older nodes mid-rolling-upgrade may omit approved / always /
feedback; dispatch coerces to safe defaults."""
adapter, recorder, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
adapter._dispatch_child_event({"type": "approval_resolved", "ws_id": "child-a1"})
assert len(recorder.enqueued) == 1
payload = recorder.enqueued[0]
assert payload["approved"] is False
assert payload["feedback"] == ""
assert payload["always"] is False
def test_dispatch_approval_resolved_unknown_child_drops(self) -> None:
"""Symmetric to the intent_verdict drop test — events for
ws_ids the registry doesn't know about silently drop instead
of fanning out to a parent that has no business seeing them."""
adapter, recorder, _ = self._setup()
adapter._dispatch_child_event(
{
"type": "approval_resolved",
"ws_id": "ws-orphan",
"approved": True,
},
)
assert recorder.enqueued == []
def test_dispatch_approve_request_emits_child_ws_approve_request(
self,
) -> None:
"""Push path for the initial approval items — eliminates the
bulk-fetch race that left the coord row stuck on a loading
placeholder when the bulk fetch landed in the gap between
_emit_state(ATTENTION) and approve_tools setting _pending_approval."""
adapter, recorder, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
detail = {
"type": "approve_request",
"items": [{"call_id": "c1", "header": "tool x"}],
"judge_pending": True,
}
adapter._dispatch_child_event(
{
"type": "approve_request",
"ws_id": "child-a1",
"node_id": "node-1",
"detail": detail,
},
)
assert len(recorder.enqueued) == 1
payload = recorder.enqueued[0]
assert payload["type"] == "child_ws_approve_request"
assert payload["child_ws_id"] == "child-a1"
assert payload["parent_ws_id"] == "coord-a"
assert payload["node_id"] == "node-1"
assert payload["detail"] == detail
def test_dispatch_approve_request_unknown_child_drops(self) -> None:
adapter, recorder, _ = self._setup()
adapter._dispatch_child_event(
{
"type": "approve_request",
"ws_id": "ws-orphan",
"detail": {"items": []},
},
)
assert recorder.enqueued == []
def test_dispatch_notifies_child_event_bus_on_state_event(self) -> None:
"""Every translated state-class event must call
``ChildEventBus.notify(ws_id)`` so a registered
``wait_for_workstream`` waiter wakes promptly. Notify fires
AFTER the UI enqueue so the SSE fan-out keeps priority the
order assertion here is structural (one notify call, matching
ws_id) since the bus side-effect lookup is what guards against
regressions, not the relative event ordering.
"""
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-a1"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "idle",
}
)
assert event.is_set(), "bus notify did not fire on cluster_state dispatch"
def test_dispatch_notifies_for_all_state_class_event_types(self) -> None:
"""The dispatch sink translates six event types into the
``child_ws_*`` SSE shape; all six must also fire the bus so
a wait on any of them wakes. ``ws_created`` is intentionally
NOT in this set waiters register against ws_ids they already
know exist (the wait tool takes a pre-known list)."""
for etype, extra in [
("cluster_state", {"state": "running"}),
("ws_closed", {"reason": "evicted"}),
("ws_rename", {"name": "renamed"}),
("intent_verdict", {"verdict": {"call_id": "c1"}}),
("approval_resolved", {"approved": True}),
("approve_request", {"detail": {}}),
]:
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-a1"])
adapter._dispatch_child_event(
{"type": etype, "ws_id": "child-a1", **extra},
)
assert event.is_set(), f"bus notify did not fire on {etype} dispatch"
def test_dispatch_does_not_notify_for_unrelated_ws_id(self) -> None:
"""Bus is keyed by ws_id — a dispatch for ws X must not wake a
waiter registered against ws Y, or every state change anywhere
in the system would shake every concurrent wait."""
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-other"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "idle",
}
)
assert not event.is_set(), "bus notify spuriously fired on unrelated ws_id"
def test_dispatch_does_not_notify_for_unknown_child(self) -> None:
"""Events whose ws_id isn't in any coord's registry are dropped
BEFORE the bus notify (early return at ``coord_id is None``).
Notify only fires for events the dispatch sink fully translated,
keeping the bus side-effect aligned with the UI enqueue."""
adapter, _, _ = self._setup()
bus = adapter.child_event_bus
event = bus.register_waiter(["ws-orphan"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "ws-orphan",
"state": "idle",
}
)
assert not event.is_set(), "bus notify fired for ws_id the dispatch dropped"
+378 -9
View File
@@ -9,6 +9,7 @@ storage-call path.
from __future__ import annotations
import json
import time
from typing import TYPE_CHECKING, Any
import httpx
@@ -20,6 +21,7 @@ from turnstone.console.coordinator_client import (
CoordinatorTokenManager,
)
from turnstone.core.auth import JWT_AUD_CONSOLE, validate_jwt
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
@@ -145,6 +147,7 @@ def _mock_client(
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
return client, captured
@@ -470,6 +473,7 @@ def _make_read_client(storage: SQLiteBackend) -> CoordinatorClient:
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
@@ -663,6 +667,7 @@ def _make_client_with_cluster_response(
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
@@ -949,6 +954,137 @@ def test_list_nodes_empty_on_no_matching_filters(storage_with_nodes):
assert result["truncated"] is False
def test_list_nodes_surfaces_healthy_model_aliases(tmp_path):
"""The node's heartbeat loop projects its registry into a ``models``
metadata entry shaped like ``[{alias, provider, healthy}, ...]``.
``list_nodes`` flattens that to the healthy-alias list at the top
level (under ``model_aliases``) so a coordinator can pass aliases
straight to ``spawn_workstream(model=)`` without having to
introspect the metadata blob. The provider-side model identifier
(``cfg.model``) is intentionally NOT in the payload coords kept
reaching for it when they should pass the local alias."""
st = SQLiteBackend(str(tmp_path / "nodes.db"))
_set_meta(
st,
"node-x",
[
("arch", "x86_64", "auto"),
(
"models",
[
{"alias": "gpt5", "provider": "openai", "healthy": True},
{"alias": "claude-opus-47", "provider": "anthropic", "healthy": True},
{"alias": "broken", "provider": "openai", "healthy": False},
],
"auto",
),
],
)
_register_service(st, "node-x")
client = _make_read_client(st)
result = client.list_nodes()
node = result["nodes"][0]
assert node["model_aliases"] == ["gpt5", "claude-opus-47"]
# Full per-alias info still available under metadata for callers
# that want provider / healthy detail (e.g. surfacing degraded
# aliases in a UI).
full = node["metadata"]["models"]["value"]
assert {row["alias"] for row in full} == {"gpt5", "claude-opus-47", "broken"}
# ``model`` (the provider-side identifier) is intentionally absent
# — keep the payload to the three values a coord actually uses.
for row in full:
assert "model" not in row
def test_list_nodes_model_aliases_distinct_from_metadata_models(tmp_path):
"""Pin the naming distinction explicitly: the top-level shortlist
(``model_aliases``, list of strings) and the rich metadata blob
(``metadata.models.value``, list of dicts) live under different
keys so a caller that confuses them gets a clear KeyError rather
than a silent shape mismatch."""
st = SQLiteBackend(str(tmp_path / "nodes.db"))
_set_meta(
st,
"node-x",
[
(
"models",
[{"alias": "a", "provider": "openai", "healthy": True}],
"auto",
),
],
)
_register_service(st, "node-x")
client = _make_read_client(st)
node = client.list_nodes()["nodes"][0]
# No top-level ``models`` field — only ``model_aliases``.
assert "models" not in node
assert node["model_aliases"] == ["a"]
# Rich shape stays under metadata.
assert isinstance(node["metadata"]["models"]["value"], list)
assert isinstance(node["metadata"]["models"]["value"][0], dict)
def test_list_nodes_model_aliases_empty_when_node_has_not_published(tmp_path):
"""Nodes from older builds — or a node mid-startup before its first
metadata write won't have a ``models`` entry. The top-level
``model_aliases`` field defaults to ``[]`` rather than being
omitted so coordinators can rely on the key being present."""
st = SQLiteBackend(str(tmp_path / "nodes.db"))
_set_meta(st, "node-y", [("arch", "x86_64", "auto")])
_register_service(st, "node-y")
client = _make_read_client(st)
result = client.list_nodes()
assert result["nodes"][0]["model_aliases"] == []
def test_list_nodes_models_tolerates_malformed_entries(tmp_path):
"""If a node ever stores a malformed ``models`` entry (wrong outer
type, missing alias, non-bool healthy), the projection drops the
bad rows rather than raising the rest of the response should
still be useful."""
st = SQLiteBackend(str(tmp_path / "nodes.db"))
_set_meta(
st,
"node-z",
[
(
"models",
[
{"alias": "ok", "provider": "p", "healthy": True},
"not-a-dict",
{"provider": "p", "healthy": True}, # missing alias
{"alias": "", "healthy": True}, # empty alias
{"alias": "degraded", "healthy": False},
{"alias": 42, "healthy": True}, # non-string alias
],
"auto",
),
],
)
_register_service(st, "node-z")
client = _make_read_client(st)
result = client.list_nodes()
assert result["nodes"][0]["model_aliases"] == ["ok"]
def test_list_nodes_models_handles_non_list_payload(tmp_path):
"""A node with a corrupted models entry (dict, scalar, null) shouldn't
blow up the whole list_nodes call. ``model_aliases`` falls back to ``[]``."""
st = SQLiteBackend(str(tmp_path / "nodes.db"))
_set_meta(
st,
"node-w",
[
("models", {"oops": "not a list"}, "auto"),
],
)
_register_service(st, "node-w")
client = _make_read_client(st)
result = client.list_nodes()
assert result["nodes"][0]["model_aliases"] == []
# ---------------------------------------------------------------------------
# list_skills
# ---------------------------------------------------------------------------
@@ -1312,6 +1448,23 @@ def test_wait_for_workstream_denies_foreign_ws_id(populated_storage):
assert result["elapsed"] < 1.0
def test_wait_for_workstream_denies_cross_tenant_child(populated_storage):
"""Defense-in-depth (Copilot #506): a row whose ``parent_ws_id``
matches the coordinator but whose ``user_id`` belongs to a
different tenant must collapse to ``denied`` otherwise a
forged / migration-era / pre-tenant-gate row would let a
coordinator's LLM observe foreign-tenant state through
``wait_for_workstream``. The ``populated_storage`` fixture's
``cross-tenant-child`` row has exactly this shape
(parent_ws_id="coord-1", user_id="user-2").
"""
client = _make_read_client(populated_storage)
result = client.wait_for_workstream(["cross-tenant-child"], timeout=5, mode="any")
assert result["results"]["cross-tenant-child"]["state"] == "denied"
assert result["complete"] is False
assert result["elapsed"] < 1.0
def test_wait_for_workstream_missing_ws_id_indistinguishable_from_denied(populated_storage):
"""A ws_id that doesn't exist collapses into the same 'denied'
shape as a foreign ws_id so wait can't be used as an existence
@@ -1400,10 +1553,22 @@ def test_wait_for_workstream_dedupes_ws_ids(populated_storage):
assert list(result["results"].keys()) == ["child-a"]
def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monkeypatch):
"""Per-tick polling must issue batched storage calls — at the
documented cap (32 ws_ids over a 600s wait) the naive per-id
shape produced ~38k row reads. Guard against regression."""
def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
populated_storage, monkeypatch
):
"""All storage reads issued by ``wait_for_workstream`` must go
through the batched paths. At the documented cap (32 ws_ids over
a 600 s wait) the naive per-id shape produced ~38k row reads, so
a regression to per-id is the meaningful failure mode this test
guards against.
The primary safety net is the ``pytest.fail`` mock on the per-id
``get_workstream`` / ``sum_workstream_tokens`` paths any call
there blows up loudly with the regression message. The
additional ``batch_calls`` / ``sum_calls`` assertions cover the
subtler regression where the call IS batched but only covers a
subset of ws_ids (e.g. one ws_id per call in a loop).
"""
client = _make_read_client(populated_storage)
batch_calls: list[list[str]] = []
sum_calls: list[list[str]] = []
@@ -1434,11 +1599,16 @@ def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monke
result = client.wait_for_workstream(["child-a", "child-b"], timeout=5, mode="any")
assert result["complete"] is True
# One tick is enough since child-a is already idle (terminal).
assert len(batch_calls) == 1
assert len(sum_calls) == 1
assert set(batch_calls[0]) == {"child-a", "child-b"}
assert set(sum_calls[0]) == {"child-a", "child-b"}
# Every batched call carried the full ws_id set. The exact count
# (currently 2: one pre-loop ownership filter + one snapshot tick)
# is incidental; if either gains another batched read it stays
# batched, which is the property under test.
assert batch_calls, "no batched get_workstreams_batch call observed"
assert sum_calls, "no batched sum_workstream_tokens_batch call observed"
first_batch = set(batch_calls[0])
first_sum = set(sum_calls[0])
assert first_batch == {"child-a", "child-b"}
assert first_sum == {"child-a", "child-b"}
def test_wait_for_workstream_handles_non_string_mode(populated_storage):
@@ -1450,6 +1620,205 @@ def test_wait_for_workstream_handles_non_string_mode(populated_storage):
assert "invalid mode" in result["error"]
# ---------------------------------------------------------------------------
# wait_for_workstream — event-driven (ChildEventBus wired in)
# ---------------------------------------------------------------------------
#
# When the coord adapter wires its ``child_event_bus`` into the client,
# the wait loop blocks on a per-call ``threading.Event`` keyed by ws_id
# and only re-snapshots storage on state-change wakes or the heartbeat
# cap. The legacy ``time.sleep`` poll path remains intact for tests
# that don't wire the bus (above), so this section adds focused
# coverage of the bus-driven behaviour without re-running the full
# matrix of mode / since / cross-tenant cases.
def _make_read_client_with_bus(storage, bus) -> CoordinatorClient:
"""Like ``_make_read_client`` but wires a real ``ChildEventBus``.
Caller owns the bus so the test can call ``bus.notify(ws_id)`` to
simulate the dispatch-sink wake-up.
"""
transport = httpx.MockTransport(lambda r: httpx.Response(200))
http = httpx.Client(transport=transport)
return CoordinatorClient(
console_base_url="http://x",
storage=storage,
token_factory=lambda: "t",
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=bus,
)
def test_wait_with_bus_returns_immediately_when_already_terminal(populated_storage):
"""Subscribe-after-terminal race: the wait registers its waiter
BEFORE the first snapshot, then re-snapshots an already-terminal
child must return at once without spinning the heartbeat cap.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
result = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
assert result["complete"] is True
assert result["results"]["child-a"]["state"] == "idle"
assert result["elapsed"] < 1.0
# Waiter must be unregistered on exit so a long-lived bus doesn't
# accumulate dead keys across many waits.
assert "child-a" not in bus._waiters
def test_wait_with_bus_wakes_on_notify(populated_storage):
"""The core property of the refactor: a state-change ``notify``
must wake the wait promptly well under the legacy 0.5 s poll
cadence AND the 2 s heartbeat cap. Test fires a state update
+ notify after a short delay and asserts the wait returns quickly.
"""
import threading as _t
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# child-b starts running; flip to idle + notify after the wait
# blocks. 100 ms is enough that the wait is parked in event.wait()
# but short enough that the test runs fast.
timer = _t.Timer(
0.1,
lambda: (
populated_storage.update_workstream_state("child-b", "idle"),
bus.notify("child-b"),
),
)
timer.start()
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=5.0, mode="any")
elapsed = time.monotonic() - start
assert result["complete"] is True
assert result["results"]["child-b"]["state"] == "idle"
# Bus-driven wake should fire well under 1 s; legacy poll would
# take ~0.5 s but bus-driven should be ~0.1 s (the timer delay)
# plus a few ms. Generous 0.6 s budget for CI noise.
assert elapsed < 0.6, f"wake-up too slow: {elapsed}s"
def test_wait_with_bus_unrelated_notify_does_not_wake(populated_storage):
"""A notify on a ws_id the wait isn't watching must NOT wake it —
otherwise every state change anywhere on the system would shake
every concurrent wait into a redundant storage snapshot.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# child-b is running indefinitely; mode='all' will time out unless
# a relevant notify fires. Fire only unrelated notifies — wait
# should still hit the full timeout.
import threading as _t
def _fire_unrelated() -> None:
for _ in range(5):
bus.notify("ws-unrelated-1")
bus.notify("ws-unrelated-2")
time.sleep(0.05)
t = _t.Thread(target=_fire_unrelated, daemon=True)
t.start()
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=0.5, mode="all")
elapsed = time.monotonic() - start
assert result["complete"] is False, "unrelated notify falsely satisfied wait"
# Wait should burn its full timeout (give or take heartbeat
# granularity). The bus path doesn't have a 0.5 s poll, so the
# bound is "approximately timeout".
assert elapsed >= 0.5
t.join(timeout=1.0)
def test_wait_with_bus_heartbeat_still_progresses_without_notify(populated_storage):
"""Without any notify, the wait must still progress through ticks
via the heartbeat cap so ``progress_callback`` keeps firing for
the sidebar UI. Verified by counting callback firings over an
interval longer than the heartbeat.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# Shrink the heartbeat for test speed via the ClassVar seam —
# instance attribute shadows the class-level default. Production
# stays at 2.0 s; the test exercises the heartbeat-fires-without-
# notify property in well under 1 s.
client._WAIT_HEARTBEAT_INTERVAL = 0.1 # type: ignore[misc]
snapshots: list[dict[str, dict[str, object]]] = []
def _cb(snap: dict[str, dict[str, object]], _elapsed: float) -> None:
snapshots.append(snap)
# child-b is running indefinitely; wait will time out at 0.4 s.
# With heartbeat = 0.1 s, we expect ~3-5 callback firings
# (initial tick + ~3-4 heartbeats). Loose lower bound to avoid
# CI flakiness.
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=0.4, mode="all", progress_callback=_cb)
elapsed = time.monotonic() - start
assert result["complete"] is False
assert elapsed >= 0.4
# At least 2 callback firings: the initial snapshot plus at least
# one heartbeat-driven re-tick. Tight upper bound would be
# ~ceil(0.4/0.1) + 1 = 5 firings.
assert len(snapshots) >= 2, f"heartbeat didn't fire: {len(snapshots)} snapshots"
def test_wait_with_bus_unregisters_waiter_on_exit(populated_storage):
"""Both the success path and the timeout path must unregister the
waiter otherwise a long-lived bus accumulates dead
``threading.Event`` instances forever.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# Success path (already-terminal child).
client.wait_for_workstream(["child-a"], timeout=5, mode="any")
assert bus._waiters == {}, "success path leaked waiter"
# Timeout path (running child, mode='all' that times out).
client.wait_for_workstream(["child-a", "child-b"], timeout=0.3, mode="all")
assert bus._waiters == {}, "timeout path leaked waiter"
def test_wait_with_bus_multi_waiter_independence(populated_storage):
"""Two concurrent waits on the same ws_id must be independent —
one wait completing must not affect the other's wake-up state.
Smoke-tests the multi-Event-per-bucket bus behaviour against the
real wait-loop.
"""
import threading as _t
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
results: dict[str, dict[str, object]] = {}
def _do_wait(label: str) -> None:
results[label] = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
threads = [_t.Thread(target=_do_wait, args=(f"t{i}",), daemon=True) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5.0)
for label in ("t0", "t1", "t2"):
assert results[label]["complete"] is True
assert results[label]["results"]["child-a"]["state"] == "idle"
# All waiters must be unregistered after exit.
assert bus._waiters == {}
# ---------------------------------------------------------------------------
# wait_for_workstream — last-message bundling
# ---------------------------------------------------------------------------
+3
View File
@@ -40,6 +40,7 @@ from turnstone.console.server import (
_require_coord_mgr,
)
from turnstone.core.auth import AuthResult
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.session_manager import SessionManager
from turnstone.core.session_routes import (
SessionEndpointConfig,
@@ -286,6 +287,7 @@ def test_coordinator_client_spawn_close_delete(tmp_path):
coord_ws_id="coord-42",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
# spawn ---------------------------------------------------------------
@@ -387,6 +389,7 @@ def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
coord_ws_id="coord-root",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
+489
View File
@@ -0,0 +1,489 @@
"""Unit tests for :class:`CoordinatorIdleObserver`.
Drives a fake :class:`SessionManager` that mirrors the real one's
``subscribe_to_state`` / ``get`` contract, plus a fake storage with the
``list_workstreams`` slice the observer queries.
"""
from __future__ import annotations
import contextlib
import threading
from typing import Any
from unittest.mock import MagicMock
import pytest
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
from turnstone.core.nudge_queue import NudgeQueue
from turnstone.core.workstream import WorkstreamKind, WorkstreamState
class _FakeRow:
"""SQLAlchemy-Row-like wrapper exposing ``_mapping``."""
def __init__(self, **kwargs: Any) -> None:
self._mapping = kwargs
class _FakeStorage:
def __init__(self) -> None:
self.children: list[dict[str, Any]] = []
self.list_calls: list[dict[str, Any]] = []
self.count_calls: list[dict[str, Any]] = []
self.list_raises: bool = False
self.count_raises: bool = False
def list_workstreams(
self,
node_id: str | None = None,
limit: int = 100,
*,
parent_ws_id: str | None = None,
kind: WorkstreamKind | str | None = None,
user_id: str | None = None,
) -> list[Any]:
self.list_calls.append(
{
"limit": limit,
"parent_ws_id": parent_ws_id,
"kind": kind,
"user_id": user_id,
}
)
if self.list_raises:
raise RuntimeError("storage forced failure")
return [_FakeRow(**c) for c in self.children]
def count_workstreams_by_state(
self,
*,
parent_ws_id: str | None = None,
user_id: str | None = None,
) -> dict[str, int]:
self.count_calls.append({"parent_ws_id": parent_ws_id, "user_id": user_id})
if self.count_raises:
raise RuntimeError("count forced failure")
counts: dict[str, int] = {}
for c in self.children:
counts[c["state"]] = counts.get(c["state"], 0) + 1
return counts
class _FakeSession:
def __init__(self) -> None:
self._nudge_queue = NudgeQueue()
self.messages: list[dict[str, Any]] = []
self._wake_source_tag: str = ""
self._metacog_state: dict[str, float] = {}
self._mem_cfg = MagicMock(nudge_cooldown=300)
def _visible_memory_count(self) -> int:
return 0
class _FakeWorkstream:
def __init__(
self,
ws_id: str = "ws-coord",
kind: WorkstreamKind = WorkstreamKind.COORDINATOR,
user_id: str = "u1",
) -> None:
self.id = ws_id
self.kind = kind
self.user_id = user_id
self.session: _FakeSession | None = _FakeSession()
class _FakeManager:
def __init__(self) -> None:
self._workstreams: dict[str, _FakeWorkstream] = {}
self._subscribers: list[Any] = []
self._lock = threading.Lock()
def add_ws(self, ws: _FakeWorkstream) -> None:
self._workstreams[ws.id] = ws
def remove_ws(self, ws_id: str) -> None:
self._workstreams.pop(ws_id, None)
def get(self, ws_id: str) -> _FakeWorkstream | None:
return self._workstreams.get(ws_id)
def subscribe_to_state(self, callback: Any) -> None:
with self._lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Any) -> None:
with self._lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
def fire_state(self, ws_id: str, state: WorkstreamState) -> None:
with self._lock:
subs = list(self._subscribers)
for cb in subs:
with contextlib.suppress(Exception):
cb(ws_id, state)
@pytest.fixture
def coord_setup() -> tuple[_FakeManager, _FakeStorage, _FakeWorkstream]:
mgr = _FakeManager()
storage = _FakeStorage()
ws = _FakeWorkstream()
mgr.add_ws(ws)
return mgr, storage, ws
def _add_active_child(storage: _FakeStorage, **overrides: Any) -> None:
storage.children.append(
{
"ws_id": overrides.get("ws_id", "child-1"),
"name": overrides.get("name", "research"),
"state": overrides.get("state", "running"),
}
)
class TestEnqueueOnIdle:
def test_idle_with_active_children_enqueues(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage, ws_id="child-a", state="running")
_add_active_child(storage, ws_id="child-b", state="thinking")
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
snap = ws.session._nudge_queue.pending("any")
assert len(snap) == 1
nudge_type, text = snap[0]
assert nudge_type == "idle_children"
assert "child-a" in text
assert "child-b" in text
def test_idle_with_no_active_children_no_enqueue(self, coord_setup):
mgr, storage, ws = coord_setup
# storage.children is empty
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
def test_idle_only_idle_state_children_no_enqueue(self, coord_setup):
mgr, storage, ws = coord_setup
# All children "idle" — terminal-from-coord-perspective; not active.
_add_active_child(storage, state="idle")
_add_active_child(storage, state="closed")
_add_active_child(storage, state="error")
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
def test_non_idle_state_no_enqueue(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
for state in (
WorkstreamState.RUNNING,
WorkstreamState.THINKING,
WorkstreamState.ATTENTION,
WorkstreamState.ERROR,
):
mgr.fire_state(ws.id, state)
assert len(ws.session._nudge_queue) == 0
class TestKindFilter:
def test_interactive_workstream_skipped(self):
mgr = _FakeManager()
storage = _FakeStorage()
_add_active_child(storage)
ws = _FakeWorkstream(kind=WorkstreamKind.INTERACTIVE)
mgr.add_ws(ws)
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Observer ignored the non-coord workstream entirely.
assert len(ws.session._nudge_queue) == 0
# Storage was NOT queried — kind check happens before list_workstreams.
assert storage.list_calls == []
class TestWaitForWorkstreamSkip:
def test_skips_when_last_assistant_used_wait(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
ws.session.messages = [
{"role": "user", "content": "kick off"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call-1",
"function": {"name": "wait_for_workstream", "arguments": "{}"},
}
],
},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Don't pile on — model is already using the right tool.
assert len(ws.session._nudge_queue) == 0
def test_fires_when_last_assistant_used_different_tool(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
ws.session.messages = [
{"role": "user", "content": "go"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{"id": "call-1", "function": {"name": "spawn_workstream", "arguments": "{}"}}
],
},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 1
class TestHardCap:
def test_hard_cap_blocks_after_n_fires(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Bypass cooldown for this test: each call burns a per-type slot
# in ``_metacog_state`` so we need to clear it between fires.
for _ in range(3):
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Cap = 3 fires. Even with cooldown bypassed, the 4th doesn't fire.
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# We enqueued 3 entries total; cap blocked the 4th.
snap = ws.session._nudge_queue.pending("any")
assert len(snap) == 3
def test_cap_resets_when_state_leaves_idle_without_wake(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Burn the cap.
for _ in range(3):
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 3
# Drain the queue (simulate the watcher delivering them).
ws.session._nudge_queue.drain({"any"})
# Real (non-wake) leave-IDLE: tag is empty. Cap resets.
ws.session._wake_source_tag = ""
mgr.fire_state(ws.id, WorkstreamState.RUNNING)
# New IDLE — cap is fresh, fires again.
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 1
def test_cap_does_not_reset_during_wake_driven_exit(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Burn the cap.
for _ in range(3):
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
ws.session._nudge_queue.drain({"any"})
# Wake-driven leave-IDLE: tag is set during the wake send.
ws.session._wake_source_tag = "system_nudge"
mgr.fire_state(ws.id, WorkstreamState.RUNNING)
ws.session._wake_source_tag = "" # tag cleared at end of wake send
# Cap should NOT have reset — re-IDLE shouldn't fire.
ws.session._metacog_state.clear()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 0
class TestCooldown:
def test_cooldown_blocks_within_window(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 1
# Drain so the queue isn't the gate.
ws.session._nudge_queue.drain({"any"})
# Second fire within the cooldown window → should_nudge returns False.
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue.pending("any")) == 0
class TestStorageFailure:
def test_storage_exception_is_swallowed(self, coord_setup):
mgr, storage, ws = coord_setup
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
storage.list_raises = True
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
# Must not raise / propagate.
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
class TestValidUntilPredicate:
def test_predicate_drops_when_children_finish_before_drain(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage, ws_id="child-a", state="running")
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 1
# Children now complete (storage shows none active).
storage.children.clear()
# Drain at the user seam — predicate re-queries, finds 0 active,
# drops the entry without delivering.
from turnstone.core.nudge_queue import USER_DRAIN
delivered = ws.session._nudge_queue.drain(USER_DRAIN)
assert delivered == []
assert len(ws.session._nudge_queue) == 0
def test_predicate_delivers_when_children_still_active(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage, ws_id="child-a", state="running")
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Children still active → predicate returns True → entry delivers.
from turnstone.core.nudge_queue import USER_DRAIN
delivered = ws.session._nudge_queue.drain(USER_DRAIN)
assert len(delivered) == 1
assert delivered[0][0] == "idle_children"
def test_predicate_drops_on_storage_failure(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Storage failure at drain time. Predicate treats raises as
# "no longer valid" (drop) — see NudgeQueue.drain's predicate
# exception handling.
storage.count_raises = True
from turnstone.core.nudge_queue import USER_DRAIN
delivered = ws.session._nudge_queue.drain(USER_DRAIN)
assert delivered == []
class TestLifecycle:
def test_start_idempotent(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
observer.start() # no-op
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Double-subscribe would have produced 2 entries.
assert len(ws.session._nudge_queue.pending("any")) == 1
def test_shutdown_unsubscribes(self, coord_setup):
mgr, storage, ws = coord_setup
_add_active_child(storage)
# ≥2 messages so should_nudge's message_count > 1 gate clears.
ws.session.messages = [
{"role": "user", "content": "go"},
{"role": "assistant", "content": "ok"},
]
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
observer.shutdown()
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert len(ws.session._nudge_queue) == 0
def test_shutdown_idempotent(self, coord_setup):
mgr, _storage, _ws = coord_setup
observer = CoordinatorIdleObserver(mgr, _storage)
observer.start()
observer.shutdown()
observer.shutdown() # no error
+206 -16
View File
@@ -79,9 +79,10 @@ def test_coordinator_js_exposes_inline_approval_helpers():
assert "function submitChildApproval" in body or "submitChildApproval(" in body
# The shared approve POST helper (parameterized for child ws_ids)
assert "function approveWorkstream" in body or "approveWorkstream(" in body
# The urgent live-bulk fetch option that fires on activity_state
# transitions in/out of "approval"
assert "{ urgent: true }" in body or "urgent: true" in body
# The 409 stale-call_id retry path uses invalidateLiveBadge +
# scheduleLiveFetch (Stage 3 cleanup removed the urgent flag —
# cache invalidation makes the TTL gate fall through naturally).
assert "invalidateLiveBadge(targetWsId)" in body
# Server-side payload field — drift here means the JS reads stale keys
assert "pending_approval_detail" in body
# Reconnect parity (chunk 4): the SSE re-open handler must drop
@@ -90,10 +91,10 @@ def test_coordinator_js_exposes_inline_approval_helpers():
# can't render zombie approve/deny buttons on a row whose
# approval was resolved during the gap. The implementation
# iterates the cache and deletes only !permanent entries —
# asserting the literal Map iteration form keeps a refactor
# back to liveBadgeCache.clear() (which would re-pay 403s on
# every reconnect for denied ids) from sneaking in.
assert "liveBadgeCache.delete" in body
# asserting the literal helper call keeps a refactor back to
# _liveBadgeCacheClear() (which would re-pay 403s on every
# reconnect for denied ids) from sneaking in.
assert "_liveBadgeCacheDelete" in body
# Edge-case matrix sentinel labels — POLICY-BLOCKED renders when
# an item has error set + needs_approval=False (server-side
# tool policy already blocked the call); "(judge unavailable)"
@@ -113,15 +114,19 @@ def test_coordinator_js_exposes_inline_approval_helpers():
# coord-self ws_id (the coord lives on the console process).
# Children live on cluster nodes and 404 without the prefix.
assert "/v1/api/route/workstreams/" in body
# Late-judge polling — the LLM judge runs async on the child
# node and never pushes a signal that reaches the coord, so
# the row's pending_approval_detail with judge_pending=true
# would freeze on heuristic verdicts forever without this
# poll loop. The poller is GLOBAL (not per-row) so off-screen
# rows still refresh — a per-row poller's scheduleLiveFetch
# call short-circuits on non-visible rows, leaving them stuck.
assert "_maybeStartJudgePoll" in body
assert "_judgePollTick" in body
# Late-arriving LLM judge verdicts — Stage 3 Step 5 promoted
# ``intent_verdict`` and ``approval_resolved`` to first-class
# cluster-bus event types, so the coord adapter dispatches them
# as ``child_ws_intent_verdict`` / ``child_ws_approval_resolved``
# on the parent's SSE stream. The browser handlers write
# directly to liveBadgeCache (bypassing scheduleLiveFetch's
# visibility gate cleanly) so off-screen rows pick up verdicts
# without polling. Replaced the old ``_judgePollTick`` 90-second
# global poll loop and its visibility-gate-bypass workaround.
assert "handleChildIntentVerdict" in body
assert "handleChildApprovalResolved" in body
assert "child_ws_intent_verdict" in body
assert "child_ws_approval_resolved" in body
# Reload parity for the coord-self approval gate: init() must
# consume the authoritative GET /workstreams snapshot's
# pending_approval_detail so a freshly opened tab can render
@@ -152,3 +157,188 @@ def test_coordinator_js_exposes_inline_approval_helpers():
# any prior denial. bug-1 / bug-3 from the second /review pass.
assert "Denied by user" in body
assert "callOutcomes" in body
# User-message attachment pills — both live send (coordSend) and
# history replay route through appendUserMessageWithAttachments.
# Renaming or dropping the helper would silently regress the
# attachment affordance to the pre-fix plain-text bubble, which
# would only surface in manual testing of an attached-file flow.
# The CSS class is the visual anchor (coordinator.css) — keeping
# both literals in the smoke layer covers JS↔CSS drift in either
# direction.
assert "function appendUserMessageWithAttachments" in body
assert "msg-user-attach" in body
# PR #487 — whitespace-only assistant content (Qwen3 with vLLM
# ``--reasoning-parser`` strips ``<think>…</think>`` and emits only
# ``"\n\n"`` as content before a tool call) must be skipped on
# history replay or the empty ``.msg.assistant`` card surfaces as
# a phantom row. The literal substring ``content.trim()`` is the
# single-line guard the rendering branch uses; a refactor that
# drops the trim() (e.g. simplifies to ``if (!content)``) silently
# regresses the phantom-card fix on the multi-node coord path.
# Mirrors ``test_app_js.py``'s same-shape pin on ``app.js``.
assert "content.trim()" in body
# PR #487 — coord history replay must render the assistant content
# card BEFORE the tool batch, not after, so DOM order matches the
# chronological order the model emitted (text → dispatch → results).
# Pre-fix the tool_calls branch sat at the role-agnostic top of the
# loop and rendered ahead of the assistant text that announced the
# batch, putting parallel fan-outs visually above their narrating
# message. The fix hoisted the synthesis into ``renderAssistantToolBatch``
# called from inside the assistant branch AFTER the content card —
# asserting the helper name lets a refactor that re-inlines or
# renames it surface here instead of via manual reload testing.
assert "function renderAssistantToolBatch" in body
assert "renderAssistantToolBatch(m)" in body
def test_coordinator_js_handle_child_state_no_longer_reads_sse_pending_approval_detail():
"""Stage 3 cleanup — ``pending_approval_detail`` is no longer
piggybacked on child_ws_state events. Approval items now arrive
via bulk fetch on the activity_state="approval" transition;
verdicts via the explicit ``child_ws_intent_verdict`` event class;
resolution via ``child_ws_approval_resolved``. A refactor that
re-introduces the piggyback would silently re-open the
duplicate-path race the dedicated event classes were added to
eliminate.
Structural assertions (regex against multi-line source) symbol-
presence alone wouldn't catch a guard that keeps the names but
inverts the comparison or drops the ``prev.live`` check. This
codebase has no JS test framework, so locking the guard's shape
here is the next-best thing to a behavioral test."""
import re
from pathlib import Path
coord_js = Path(__file__).resolve().parent.parent / (
"turnstone/console/static/coordinator/coordinator.js"
)
body = coord_js.read_text(encoding="utf-8")
# The piggyback read is gone from handleChildState. (The string
# may still appear elsewhere — e.g. handleChildIntentVerdict
# reading from cache, or comments — but never as ``ev.pending_approval_detail``.)
assert "ev.pending_approval_detail" not in body
# The pre-fix urgent-fetch on activity_state transitions is gone.
assert "enteredApproval" not in body
assert "leftApproval" not in body
# ``pendingApproval`` flag derivation must check BOTH state and
# activity_state. The worker thread can fire the state transition
# to "attention" before approve_tools updates activity_state, so
# checking only activity_state misses children that legitimately
# need approval. Pin the disjunction so the regression doesn't
# silently re-introduce.
assert re.search(
r'existing\.state\s*===\s*"attention"\s*\|\|\s*'
r'existing\.activity_state\s*===\s*"approval"',
body,
), (
"handleChildState must derive pendingApproval from "
"(state==='attention' || activity_state==='approval')"
)
# SSE-authoritative window constant is defined and used.
assert re.search(r"\bconst\s+SSE_AUTHORITATIVE_MS\s*=\s*\d+", body), (
"SSE_AUTHORITATIVE_MS constant must be defined as a numeric literal"
)
# SSE writers tag entries with sseUpdatedAt: Date.now() so the
# merge guard in flushLiveFetches preserves them against stale
# bulk-fetch responses. handleChildState only stamps when it
# AUTHORITATIVELY clears the detail (off-approval transition);
# writers that stamp unconditionally are intent_verdict (verdict
# stamp), approval_resolved (clear), and the optimistic-clear
# path in submitChildApproval. Pinning the literal Date.now()
# call keeps a refactor that drops the SSE-source tag entirely
# from sneaking in.
assert re.search(
r"sseUpdatedAt:\s*Date\.now\(\)",
body,
), "Critical SSE writers must stamp sseUpdatedAt: Date.now()"
# flushLiveFetches' merge guard structure: SSE-set pending_approval
# / _detail wins over a stale bulk-poll snapshot when (live) AND
# (prev exists) AND (prev.sseUpdatedAt set) AND (within window)
# AND (prev.live exists). Inverting the comparison or dropping
# any of these guards reopens the clobber bug.
merge_guard = re.search(
r"if\s*\(\s*live\s*&&\s*prev\s*&&\s*prev\.sseUpdatedAt\s*&&\s*"
r"now\s*-\s*prev\.sseUpdatedAt\s*<\s*SSE_AUTHORITATIVE_MS\s*&&\s*"
r"prev\.live\s*\)",
body,
)
assert merge_guard is not None, (
"flushLiveFetches merge guard must be the conjunction "
"(live && prev && prev.sseUpdatedAt && now - prev.sseUpdatedAt < "
"SSE_AUTHORITATIVE_MS && prev.live). An inverted comparison or "
"missing prev.live check would let a stale bulk-poll clobber a "
"fresh SSE-set approval."
)
# The merge body must preserve BOTH pending_approval and
# pending_approval_detail from prev — preserving only one would
# render a row with a phantom badge but no buttons (or vice versa).
merge_body = re.search(
r"mergedLive\s*=\s*Object\.assign\(\s*\{\}\s*,\s*live\s*,\s*\{"
r"[^}]*pending_approval:\s*prev\.live\.pending_approval[^}]*"
r"pending_approval_detail:\s*prev\.live\.pending_approval_detail",
body,
)
assert merge_body is not None, (
"Merge body must preserve both pending_approval AND "
"pending_approval_detail from prev.live — preserving only one "
"creates a half-rendered approval row."
)
# flushLiveFetches must forward sseUpdatedAt onto the new cache
# entry so the SSE-source tag survives the bulk-poll write back —
# without this, every bulk-poll resets the window and the next
# late-arriving poll silently clobbers.
assert re.search(
r"sseUpdatedAt:\s*prev\s*\?\s*prev\.sseUpdatedAt",
body,
), (
"flushLiveFetches must forward prev.sseUpdatedAt onto the new "
"cache entry (preserving the SSE-source window across bulk-poll "
"cycles) — without this, the second bulk-poll after an SSE "
"transition silently clobbers."
)
def test_coord_history_renders_user_interjection_advisory_after_tool_block():
"""Queued user messages spliced into the last tool-result envelope
of a batch (Seam 1) persist on the tool DB row as a wrapped
``<tool_output>`` envelope. ``decorate_history_messages`` extracts
the advisory back out and the wire layer projects it onto
``m.advisories``; the coord history loop must invoke the shared
``replayAdvisoriesAfterTool`` helper (defined in
``shared/utils.js``) so each ``user_interjection`` renders through
``appendUserMessageWithAttachments`` and the bubble looks identical
to a Seam 2/3 user row.
This test pins the call site so a refactor that drops the helper
invocation regresses the queued-during-batch replay shape silently.
Mirrors ``test_app_js.py``'s same-shape pin on interactive's
``replayHistory``."""
import re
from pathlib import Path
coord_js = Path(__file__).resolve().parent.parent / (
"turnstone/console/static/coordinator/coordinator.js"
)
body = coord_js.read_text(encoding="utf-8")
assert "replayAdvisoriesAfterTool(m.advisories" in body, (
"Coord history loop must invoke replayAdvisoriesAfterTool with "
"m.advisories so queued messages spliced into the tool envelope "
"render as user bubbles after the tool block."
)
# The renderer callback routes through appendUserMessageWithAttachments
# so the bubble matches a normal user-row replay.
assert re.search(
r"appendUserMessageWithAttachments\(\s*text",
body,
), (
"Coord history loop's renderer callback must route the extracted "
"advisory text through appendUserMessageWithAttachments so the "
"rendered bubble matches a normal user-row replay."
)
+10
View File
@@ -15,6 +15,16 @@ class TestIsSecret:
assert _is_secret("TURNSTONE_JWT_SECRET") is True
assert _is_secret("AWS_SECRET_ACCESS_KEY") is True
def test_tool_config_paths_scrubbed(self):
"""Tool-config env vars whose target files load executable
directives must be scrubbed even though they don't match a
secret-suffix pattern. Defence-in-depth alongside on-CLI
``--no-config`` for ripgrep and friends."""
assert _is_secret("RIPGREP_CONFIG_PATH") is True
assert _is_secret("GIT_CONFIG") is True
assert _is_secret("GIT_CONFIG_GLOBAL") is True
assert _is_secret("GIT_CONFIG_SYSTEM") is True
def test_suffix_matching(self):
assert _is_secret("MY_CUSTOM_API_KEY") is True
assert _is_secret("DB_PASSWORD") is True
+8
View File
@@ -8,6 +8,7 @@ from __future__ import annotations
from datetime import UTC, datetime
import pytest
import sqlalchemy as sa
# ---------------------------------------------------------------------------
@@ -317,6 +318,13 @@ class TestPromptTemplateCRUD:
def test_get_prompt_template_nonexistent(self, db):
assert db.get_prompt_template("missing") is None
def test_create_prompt_template_duplicate_id_raises_conflict(self, db):
from turnstone.core.storage._protocol import StorageConflictError
db.create_prompt_template("dup", "first", "general", "A")
with pytest.raises(StorageConflictError, match="prompt_template conflict"):
db.create_prompt_template("dup", "second", "general", "B")
def test_list_prompt_templates_ordered_by_name(self, db):
db.create_prompt_template("t2", "beta", "general", "B")
db.create_prompt_template("t1", "alpha", "general", "A")
+677
View File
@@ -0,0 +1,677 @@
"""Unit tests for ``turnstone.core.history_decoration``.
The decoration helpers are shared between two surfaces interactive's
SSE replay (``_build_history``) and the lifted ``/history`` REST
endpoint (``make_history_handler``, used by both interactive and
coord). Pinning the wire shape here lets a future schema/projection
change land in one file rather than spread across the two surfaces.
"""
from __future__ import annotations
from turnstone.core.history_decoration import (
build_output_assessment_payload,
build_verdict_payload,
decorate_history_messages,
decorate_tool_call,
)
class TestBuildVerdictPayload:
"""The wire-shape projection that's the single source of truth for
what intent_verdict fields ship to the client."""
def test_skips_unflagged_baseline(self) -> None:
"""``risk_level`` "none" is the unflagged-tool baseline; the
client filters those anyway, so projecting None at the wire
layer keeps the payload tight on long workstreams."""
row = {"risk_level": "none", "recommendation": "approve", "tier": "heuristic"}
assert build_verdict_payload(row) is None
def test_drops_call_id_and_func_name(self) -> None:
"""The client already has these on ``tc.id`` / ``tc.name``;
re-shipping them per-tool_call would balloon long replays."""
row = {
"call_id": "call_abc",
"func_name": "bash",
"risk_level": "medium",
"recommendation": "review",
"confidence": 0.8,
"intent_summary": "summary",
"tier": "heuristic",
}
out = build_verdict_payload(row)
assert out is not None
assert "call_id" not in out
assert "func_name" not in out
# Sanity — the kept fields are the ones renderVerdictBadge reads.
assert out["risk_level"] == "medium"
assert out["recommendation"] == "review"
assert out["confidence"] == 0.8
assert out["intent_summary"] == "summary"
assert out["tier"] == "heuristic"
def test_includes_reasoning_for_either_tier_when_present(self) -> None:
"""Heuristic verdicts in this project emit structured
rationales (one per matched pattern) e.g.
``policy.py`` writes a reasoning string per heuristic hit.
Ship the field for either tier when it has content; only
omit when the row didn't write one."""
for tier in ("heuristic", "llm"):
row = {
"risk_level": "high",
"tier": tier,
"reasoning": "The command exfiltrates ~/.ssh/id_rsa over an external connection.",
}
out = build_verdict_payload(row)
assert out is not None
assert "id_rsa" in out["reasoning"]
def test_omits_reasoning_when_empty(self) -> None:
"""An absent / empty reasoning string shouldn't ship as
``reasoning: ""`` the rationale ``<details>`` block on the
client renders an empty disclosure when the field is present
but empty."""
row = {"risk_level": "high", "tier": "heuristic", "reasoning": ""}
out = build_verdict_payload(row)
assert out is not None
assert "reasoning" not in out
def test_includes_judge_model_when_present(self) -> None:
"""``judge_model`` rides through so the batch tier badge can
render `` llm:claude-haiku-4`` on history-only replays
rather than the bare `` llm`` label."""
row = {"risk_level": "high", "tier": "llm", "judge_model": "claude-haiku-4"}
out = build_verdict_payload(row)
assert out is not None
assert out["judge_model"] == "claude-haiku-4"
def test_omits_judge_model_when_empty(self) -> None:
row = {"risk_level": "medium", "tier": "heuristic", "judge_model": ""}
out = build_verdict_payload(row)
assert out is not None
assert "judge_model" not in out
class TestBuildOutputAssessmentPayload:
"""Output-guard wire shape — flags decoded from JSON string at
this layer so the client never has to parse twice."""
def test_skips_unflagged_baseline(self) -> None:
row = {"risk_level": "none", "flags": "[]"}
assert build_output_assessment_payload(row) is None
def test_decodes_flags_from_json(self) -> None:
row = {"risk_level": "high", "flags": '["api_key","email"]', "redacted": 1}
out = build_output_assessment_payload(row)
assert out is not None
assert out["flags"] == ["api_key", "email"]
assert out["redacted"] is True
assert out["risk_level"] == "high"
def test_handles_malformed_flags_json(self) -> None:
"""Bad JSON in ``flags`` must not block the rest of the
assessment from rendering degrade to empty list."""
row = {"risk_level": "medium", "flags": "not-json", "redacted": 0}
out = build_output_assessment_payload(row)
assert out is not None
assert out["flags"] == []
assert out["redacted"] is False
class TestDecorateToolCall:
"""In-place mutation of either OpenAI-format or flattened tool_call
entries both shapes carry ``id`` at the top level."""
def test_attaches_verdict_when_present(self) -> None:
tc: dict[str, object] = {"id": "call_1", "function": {"name": "bash", "arguments": "{}"}}
verdicts = {
"call_1": {
"risk_level": "medium",
"recommendation": "review",
"confidence": 0.7,
"intent_summary": "summary",
"tier": "heuristic",
}
}
decorate_tool_call(tc, verdicts, {})
assert "verdict" in tc
assert tc["verdict"]["risk_level"] == "medium" # type: ignore[index]
def test_skips_when_no_call_id_match(self) -> None:
tc: dict[str, object] = {"id": "call_other", "name": "bash"}
verdicts = {
"call_1": {"risk_level": "medium", "tier": "heuristic"},
}
decorate_tool_call(tc, verdicts, {})
assert "verdict" not in tc
def test_skips_unflagged_verdict(self) -> None:
"""``build_verdict_payload`` returns None for unflagged rows;
decorate_tool_call must not stamp ``verdict`` in that case."""
tc: dict[str, object] = {"id": "call_1", "name": "bash"}
verdicts = {"call_1": {"risk_level": "none", "tier": "heuristic"}}
decorate_tool_call(tc, verdicts, {})
assert "verdict" not in tc
def test_handles_empty_id(self) -> None:
"""A tool_call with no id can't be paired against the lookup
table must not raise (or stamp the wrong row's verdict)."""
tc: dict[str, object] = {"id": "", "name": "bash"}
verdicts = {"call_1": {"risk_level": "high", "tier": "heuristic"}}
decorate_tool_call(tc, verdicts, {})
assert "verdict" not in tc
class TestDecorateHistoryMessages:
"""End-to-end mutation of a /history-shaped message list — covers
the full transform applied by ``make_history_handler``."""
def test_decorates_tool_calls_with_verdict_and_assessment(self) -> None:
verdicts = {
"call_a": {
"risk_level": "high",
"recommendation": "deny",
"confidence": 0.95,
"intent_summary": "exfil",
"tier": "llm",
"reasoning": "ssh key access",
}
}
assessments = {
"call_a": {"risk_level": "high", "flags": '["secret"]', "redacted": 1},
}
messages: list[dict[str, object]] = [
{"role": "user", "content": "hi"},
{
"role": "assistant",
"content": "running",
"tool_calls": [
{
"id": "call_a",
"function": {"name": "bash", "arguments": "{}"},
}
],
},
{"role": "tool", "tool_call_id": "call_a", "content": "long output"},
{"role": "tool", "tool_call_id": "call_b", "content": "short"},
]
decorate_history_messages(messages, verdicts, assessments)
# Assistant tool_calls got both decorations.
tc = messages[1]["tool_calls"][0] # type: ignore[index]
assert tc["verdict"]["risk_level"] == "high"
assert tc["verdict"]["tier"] == "llm"
assert "reasoning" in tc["verdict"]
assert tc["output_assessment"]["flags"] == ["secret"]
assert tc["output_assessment"]["redacted"] is True
# Plain tool content (no envelope) is left intact and no
# advisories key is set.
assert messages[2]["content"] == "long output"
assert "advisories" not in messages[2]
assert messages[3]["content"] == "short"
assert "advisories" not in messages[3]
def test_no_op_on_empty_indexes(self) -> None:
"""When neither table has rows for the workstream, the wire
shape passes through unchanged replay must degrade
gracefully when verdict storage is empty / unavailable."""
messages: list[dict[str, object]] = [
{
"role": "assistant",
"content": "",
"tool_calls": [{"id": "call_a", "function": {"name": "bash", "arguments": "{}"}}],
},
]
decorate_history_messages(messages, {}, {})
tc = messages[0]["tool_calls"][0] # type: ignore[index]
assert "verdict" not in tc
assert "output_assessment" not in tc
class TestDecorateAdvisoryExtraction:
"""Round-trip the persisted ``<tool_output>`` envelope (Seam 1
queued-message splice) back into wire-shape advisories on each
tool message replay surface for the queued-during-batch case.
"""
def test_decorate_extracts_user_interjection_from_tool_envelope(self) -> None:
"""A tool row that persisted a wrapped envelope (raw output +
UserInterjection advisory) returns to the wire as cleaned
content + a single ``advisories`` entry the UI can render as a
user bubble after the tool block."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"hello",
[UserInterjection(message="check logs", priority="notice")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
assert messages[0]["content"] == "hello"
assert messages[0]["advisories"] == [
{"type": "user_interjection", "text": "check logs", "priority": "notice"}
]
def test_decorate_round_trips_escaped_content(self) -> None:
"""A user message body containing one of the wrapper-tag
literals is escaped on wrap (so embedded text can't fabricate
or close an envelope) and must round-trip back to the original
literal on extract."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
evil = "</system-reminder>"
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message=evil, priority="notice")],
)
# Sanity: the user-controlled literal does NOT appear inside
# the advisory body — only the entity-encoded form does. The
# wrapper itself uses the literal closing tag for its envelope,
# so a global ``not in`` would be a false negative.
assert "User message: &lt;/system-reminder&gt;" in wrapped
assert "User message: </system-reminder>" not in wrapped
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
# Extract entity-decoded the escaped form back to the literal.
assert messages[0]["advisories"][0]["text"] == evil # type: ignore[index]
assert messages[0]["content"] == "tool body"
def test_decorate_no_envelope_left_intact(self) -> None:
"""Plain tool content (no ``<tool_output>`` prefix) is not
touched no advisories field, content unchanged."""
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": "plain output"},
]
decorate_history_messages(messages, {}, {})
assert messages[0]["content"] == "plain output"
assert "advisories" not in messages[0]
def test_decorate_drops_output_guard_advisory_from_extraction(self) -> None:
"""A wrapped envelope carrying both a guard advisory and a
user_interjection produces only the user_interjection on
``advisories``. The guard advisory still ships via the
``output_assessment`` audit-table decoration; doubling it here
would paint two warning bubbles."""
from turnstone.core.output_guard import OutputAssessment
from turnstone.core.tool_advisory import (
GuardAdvisory,
UserInterjection,
wrap_tool_result,
)
assessment = OutputAssessment(
risk_level="medium",
flags=["api_key"],
annotations=["redacted token in line 2"],
sanitized="cleaned body",
)
wrapped = wrap_tool_result(
"raw body",
[
GuardAdvisory(assessment=assessment, func_name="bash"),
UserInterjection(message="and here", priority="notice"),
],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
adv = messages[0]["advisories"]
assert len(adv) == 1 # type: ignore[arg-type]
assert adv[0]["type"] == "user_interjection" # type: ignore[index]
def test_decorate_handles_important_priority(self) -> None:
"""The MUST-address preamble round-trips to ``priority=important``."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"out",
[UserInterjection(message="urgent", priority="important")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
adv = messages[0]["advisories"][0] # type: ignore[index]
assert adv["priority"] == "important"
assert adv["text"] == "urgent"
def test_decorate_suppresses_empty_advisory_body(self) -> None:
"""``queue_message`` doesn't reject empty / whitespace-only
text, so an advisory with an empty body can round-trip through
``wrap_tool_result``. ``_classify_advisory`` must filter those
out so replay doesn't paint a featureless empty user bubble.
Removing the ``if not body.strip(): return None`` guard in
``_classify_advisory`` breaks this test."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message="", priority="notice")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
# Envelope is still stripped from content (the cleaning side
# of decoration runs unconditionally), but no advisories
# surface — the empty body is filtered.
assert messages[0]["content"] == "tool body"
assert "advisories" not in messages[0]
def test_decorate_suppresses_whitespace_only_advisory_body(self) -> None:
"""Whitespace-only bodies are similarly suppressed — same
reasoning as the empty-body case."""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message=" \n\t ", priority="notice")],
)
messages: list[dict[str, object]] = [
{"role": "tool", "tool_call_id": "call_a", "content": wrapped},
]
decorate_history_messages(messages, {}, {})
assert messages[0]["content"] == "tool body"
assert "advisories" not in messages[0]
def test_wrap_extract_round_trips_preexisting_entities(self) -> None:
"""A user message body containing literal HTML-entity references
matching the wrapper-escape forms must round-trip identically
through ``wrap_tool_result + extract_advisories_from_tool_envelope``.
Without escaping ``&`` first in the encode step, encodedecode
would produce the bare wrapper tag, fabricating an envelope the
wrapper layer never produced.
"""
from turnstone.core.history_decoration import (
extract_advisories_from_tool_envelope,
)
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
tricky = "I describe XML tags like &lt;tool_output&gt; in my docs."
wrapped = wrap_tool_result(
"tool body",
[UserInterjection(message=tricky, priority="notice")],
)
result = extract_advisories_from_tool_envelope(wrapped)
assert result is not None
cleaned, advisories = result
assert cleaned == "tool body"
assert len(advisories) == 1
# The original literal entity-reference text round-trips
# identically — the parser does not silently turn it into a
# bare wrapper tag.
assert advisories[0]["text"] == tricky
def test_save_load_decorate_round_trips_envelope(self, backend) -> None:
"""End-to-end round-trip pinning the persisted-envelope
contract. Persists a wrapped tool-output envelope via
``save_message``, loads via ``load_messages``, runs
``decorate_history_messages``, asserts the wire shape carries
the extracted advisory + cleaned content. Pins the contract
every component in the chain participates in (persistence
layer in-memory replay wire projection) so a schema drift,
an envelope-format change, or a parser regression surfaces
here rather than only in production.
"""
from turnstone.core.tool_advisory import UserInterjection, wrap_tool_result
wrapped = wrap_tool_result(
"command output",
[UserInterjection(message="check the logs", priority="notice")],
)
backend.register_workstream("ws_rt_1")
backend.save_message("ws_rt_1", "user", "go")
backend.save_message(
"ws_rt_1",
"assistant",
None,
tool_calls='[{"id":"call_a","type":"function","function":{"name":"bash","arguments":"{}"}}]',
)
backend.save_message(
"ws_rt_1",
"tool",
wrapped,
tool_call_id="call_a",
)
msgs = backend.load_messages("ws_rt_1")
# Persisted shape — content survives the storage layer
# untouched. Symmetry with in-memory ``self.messages[i]['content']``
# is what makes envelope extraction lossless on replay.
tool_msg = next(m for m in msgs if m["role"] == "tool")
assert tool_msg["content"] == wrapped
# Decorate (the /history shared transform) — extracts the
# advisory and strips the envelope.
decorate_history_messages(msgs, {}, {})
tool_msg = next(m for m in msgs if m["role"] == "tool")
assert tool_msg["content"] == "command output"
assert tool_msg["advisories"] == [
{"type": "user_interjection", "text": "check the logs", "priority": "notice"}
]
class TestExtractReasoningForHistory:
"""``extract_reasoning_for_history`` — Phase 1 surfaces stored
Anthropic thinking blocks on assistant messages and strips
``_provider_content`` from the wire payload.
Drives through the real ``AnthropicProvider.extract_reasoning_text``
(no mock-of-extractor) the helper test and the provider unit
test (``tests/test_provider_anthropic_reasoning.py``) together
catch a regression at either layer distinctly.
"""
def _anthropic_thinking_msg(self, text: str = "let me think") -> dict[str, object]:
return {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": text, "signature": "sig"},
{"type": "text", "text": "Final answer."},
],
}
def test_extract_thinking_surfaces_reasoning_field(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("let me think")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "let me think"
def test_strips_provider_content_after_extraction(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("anything")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "_provider_content" not in messages[0]
def test_strips_provider_content_when_flag_false(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("anything")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=False)
# Strip is unconditional; reasoning is the conditional bit.
assert "_provider_content" not in messages[0]
assert "reasoning" not in messages[0]
def test_first_block_thinking_dispatches_to_anthropic(self) -> None:
# Even when text and tool_use blocks follow, the first-block-type
# discriminator routes thinking-prefixed payloads correctly.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "first", "signature": "s"},
{"type": "text", "text": "spoken"},
{"type": "tool_use", "id": "t1", "name": "f", "input": {}},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "first"
def test_first_block_reasoning_dispatches_to_openai_responses(self) -> None:
# Phase 3: dispatcher routes type=="reasoning" to the
# OpenAI Responses extractor, which now returns the
# summary[*].text concatenation. Pre-Phase-3 this asserted
# "" (the stub); the assertion was tightened once the wire
# path landed.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "s"}]}
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "s"
assert "_provider_content" not in messages[0]
def test_unknown_first_block_type_no_op(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [{"type": "text", "text": "no reasoning here"}],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "_provider_content" not in messages[0]
def test_skips_messages_without_provider_content(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [{"role": "assistant", "content": "plain"}]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert messages[0]["content"] == "plain"
def test_user_and_tool_messages_untouched(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages: list[dict[str, object]] = [
{"role": "user", "content": "hi"},
{"role": "tool", "tool_call_id": "c1", "content": "out"},
self._anthropic_thinking_msg("only this one"),
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "reasoning" not in messages[1]
assert messages[2]["reasoning"] == "only this one"
def test_empty_provider_content_no_extraction(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [{"role": "assistant", "content": "x", "_provider_content": []}]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
# Empty-list provider_content is still stripped from the wire.
assert "_provider_content" not in messages[0]
def test_first_block_not_a_dict_skipped(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages: list[dict[str, object]] = [
{
"role": "assistant",
"content": "x",
"_provider_content": ["bogus"],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "_provider_content" not in messages[0]
def test_first_block_reasoning_text_dispatches_to_openai_chat(self) -> None:
# Phase 3 path 3: synthetic ``reasoning_text`` blocks (stamped
# by ChatSession._maybe_synth_reasoning_block for vLLM /
# llama.cpp / Gemini-compat conversations) dispatch to
# OpenAIChatCompletionsProvider.extract_reasoning_text.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "synth thought"
assert "_provider_content" not in messages[0]
def test_dispatcher_scans_past_unrecognized_first_blocks(self) -> None:
# Regression for Copilot finding: dispatcher used to inspect
# only provider_content[0]['type']. OpenAI Responses captures
# EVERY output_item.done event into provider_blocks (not just
# reasoning), so a hypothetical [message, reasoning, ...]
# ordering would have silently dropped the reasoning. Now
# walks the list for the first recognised reasoning-bearing
# type and dispatches the whole list to that provider.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
# First block is a non-reasoning OpenAI Responses item.
{"type": "message", "role": "assistant", "content": "answer"},
# Reasoning sits later in the list.
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "deferred"}],
},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "deferred"
assert "_provider_content" not in messages[0]
def test_first_block_redacted_thinking_dispatches_to_anthropic(self) -> None:
# Anthropic's extended-thinking API documents that
# ``redacted_thinking`` blocks (sealed by the safety system)
# can appear before, after, or interleaved with regular
# ``thinking`` blocks. When the redacted block lands first,
# the dispatcher must still route to AnthropicProvider so the
# surrounding real thinking text surfaces — without this the
# reasoning bubble silently disappears on history rehydration.
# Pinned by registering "redacted_thinking" as a second key
# in _BLOCK_TYPE_PROVIDER_FACTORY pointing at the Anthropic
# factory; Anthropic's extractor's type=="thinking" filter
# already correctly skips the redacted block.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
{"type": "redacted_thinking", "data": "sealed-blob"},
{"type": "thinking", "thinking": "real thought", "signature": "s"},
{"type": "text", "text": "answer"},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "real thought"
assert "_provider_content" not in messages[0]
+391
View File
@@ -0,0 +1,391 @@
"""Boundary-crossing integration test for the wake trigger pipeline.
Drives a *real* :class:`SessionManager` + a *real* :class:`ChatSession`
+ a *real* :class:`IdleNudgeWatcher` end-to-end. The only stub is the
LLM provider (patched ``_create_stream_with_retry``); every other layer
is production code:
* ``SessionManager.set_state`` snapshotting + iterating subscribers
* ``IdleNudgeWatcher._on_state`` peeking the queue
* ``session_worker.send`` atomic-spawn + daemon thread
* ``ChatSession.deliver_wake_nudge_from_queue`` opening / closing
``_wake_source_tag``
* ``ChatSession.send`` chat loop short-circuiting metacog detection
* ``_append_user_turn`` stamping ``_source = "system_nudge"``
* ``_attach_pending_user_reminders`` draining ``USER_DRAIN``
* ``_apply_reminders_for_provider`` splicing the rendered envelope
onto empty content
Per ``feedback_tests_through_boundaries.md``: direct injection tests
that bypass these boundaries silently mask wiring bugs. This test is
the structural integration gate.
"""
from __future__ import annotations
import time
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests.test_session_manager import FakeStorage
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core.session import ChatSession
from turnstone.core.session_manager import SessionManager
from turnstone.core.workstream import Workstream, WorkstreamKind, WorkstreamState
# ---------------------------------------------------------------------------
# Minimal fake adapter / UI for this integration test. Storage reuses
# the canonical FakeStorage from test_session_manager.py to avoid the
# drift risk of a parallel fake.
# ---------------------------------------------------------------------------
class _FakeUI:
"""Minimal UI surface for ChatSession + SessionManager.cleanup_ui."""
def __init__(self) -> None:
self.events: list[tuple[str, Any]] = []
def _unblock(self) -> None: # SessionManager.close calls this
pass
def broadcast_ws_closed(self) -> None:
pass
# ChatSession callbacks (no-op for this test)
def on_turn_start(self) -> None:
pass
def on_turn_committed(self) -> None:
pass
def on_thinking_start(self) -> None:
pass
def on_thinking_end(self) -> None:
pass
def on_state_change(self, state: str) -> None:
self.events.append(("state", state))
def on_user_reminder(self, reminders: Any, source: str | None = None) -> None:
self.events.append(("user_reminder", reminders, source))
def on_error(self, message: str) -> None:
pass
def on_rename(self, name: str) -> None:
pass
def on_output_warning(self, call_id: Any, assessment: Any) -> None:
pass
def __getattr__(self, name: str) -> Any:
# Catch-all for any UI hook not enumerated above so the chat
# loop's ``self.ui.<something>()`` call doesn't blow up.
return MagicMock()
class _BuildRealSessionAdapter:
"""Adapter that returns a real :class:`ChatSession` instead of a stub.
Tracks emit_* events the integration test asserts on. Mirrors the
``SessionKindAdapter`` + ``SessionEventEmitter`` Protocol surface
that production ``WebUI`` / coord adapters expose.
"""
def __init__(self, kind: WorkstreamKind = WorkstreamKind.INTERACTIVE) -> None:
self.kind = kind
self.events: list[str] = []
self.cleaned_up: list[str] = []
def emit_created(self, ws: Workstream) -> None:
self.events.append(f"created:{ws.id}")
def emit_rehydrated(self, ws: Workstream) -> None:
self.events.append(f"rehydrated:{ws.id}")
def emit_state(self, ws: Workstream, state: WorkstreamState) -> None:
self.events.append(f"state:{ws.id}:{state.value}")
def emit_closed(self, ws_id: str, *, reason: str = "closed", name: str = "") -> None:
self.events.append(f"closed:{ws_id}")
def cleanup_ui(self, ws: Workstream) -> None:
# Real production cleanup_ui calls ws.session.cancel() + close().
# We don't need that here — the test exits cleanly via pytest
# teardown without exercising the cleanup path. Just record
# the call for any test that wants to assert on it.
self.cleaned_up.append(ws.id)
def build_ui(self, ws: Workstream) -> Any:
return _FakeUI()
def build_session(
self,
ws: Workstream,
*,
skill: Any = None,
model: Any = None,
client_type: Any = None,
**extra: Any,
) -> Any:
# Mirror SessionManager.create's keyword set so config-threading
# bugs surface here rather than being silently swallowed by
# **kwargs. ``model`` flows to the real ChatSession; the rest
# are accepted but not used by this test.
client = MagicMock()
return ChatSession(
client=client,
model=str(model) if model else "test-model",
ui=ws.ui,
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
# ---------------------------------------------------------------------------
# Test
# ---------------------------------------------------------------------------
@pytest.fixture
def real_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter]:
"""Real SessionManager wired to an adapter that builds real ChatSessions.
No StateWriter is wired so ``set_state`` writes directly to storage
on the calling thread (we want subscriber dispatch to fire in the
same thread the test invokes ``set_state`` on).
"""
adapter = _BuildRealSessionAdapter()
storage = FakeStorage()
mgr = SessionManager(
adapter,
storage=storage,
max_active=5,
event_emitter=adapter,
)
return mgr, adapter
def _wait_for_worker_done(ws: Workstream, timeout: float = 5.0) -> None:
"""Poll ``ws._worker_running`` until it clears or timeout elapses."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
with ws._lock:
if not ws._worker_running:
return
time.sleep(0.01)
raise AssertionError(f"worker thread for ws={ws.id[:8]} didn't exit within {timeout}s")
def test_idle_event_through_real_session_manager_drives_wake_send(real_mgr, tmp_db):
"""The full wake pipeline, no direct-injection shortcuts.
Boundary path under test:
enqueue mgr.set_state(IDLE)
SessionManager._state_subscribers iteration (real)
IdleNudgeWatcher._on_state (real)
session_worker.send (real)
real daemon thread
ChatSession.deliver_wake_nudge_from_queue (real)
ChatSession.send("") (real, with patched LLM stream)
_append_user_turn stamps ``_source``
_attach_pending_user_reminders drains ``{"user","any"}``
_apply_reminders_for_provider splices envelope onto empty content
"""
mgr, _adapter = real_mgr
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
ws = mgr.create(user_id="u1", name="wake-int", skill=None)
assert ws.session is not None
# Patch the LLM-facing surface so send() runs the chat loop end-to-end
# without any real provider. We patch on the just-built ChatSession;
# the patches are reverted by the `with` block.
with (
patch.object(ws.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
ws.session,
"_stream_response",
return_value={"role": "assistant", "content": "ok"},
),
patch.object(ws.session, "_update_token_table"),
patch.object(ws.session, "_print_status_line"),
patch.object(ws.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
# Suppress the auto-title side-thread; orthogonal to wake.
ws.session._title_generated = True
# Enqueue an any-channel nudge — the future ``idle_children`` shape.
ws.session._nudge_queue.enqueue("idle_children", "your kids", "any")
assert len(ws.session._nudge_queue) == 1
# Trigger IDLE. This runs subscriber dispatch synchronously on
# the calling thread → IdleNudgeWatcher._on_state → session_worker.send
# → spawn daemon thread → deliver_wake_nudge_from_queue.
mgr.set_state(ws.id, WorkstreamState.IDLE)
# Wait for the daemon thread to clear ``_worker_running`` so the
# post-conditions are stable.
_wait_for_worker_done(ws)
# Queue fully drained by the wake.
assert len(ws.session._nudge_queue) == 0
# The synthesized empty user message landed in history with the
# ``_source`` audit tag and the reminder side-channel populated.
user_msgs = [m for m in ws.session.messages if m.get("role") == "user"]
assert user_msgs, "expected a synthesized user message from the wake"
wake_msg = user_msgs[-1]
assert wake_msg["content"] == ""
assert wake_msg.get("_source") == "system_nudge"
assert wake_msg.get("_reminders") == [{"type": "idle_children", "text": "your kids"}]
# The wake-source tag is reset post-send so subsequent activity
# behaves normally.
assert ws.session._wake_source_tag == ""
finally:
watcher.shutdown()
def test_idle_event_with_empty_queue_does_not_dispatch_wake(real_mgr, tmp_db):
"""Non-empty queue is the gate. An IDLE event on a workstream with
nothing queued must NOT call ``session_worker.send``.
Patches the dispatch primitive directly rather than racing a
``time.sleep`` against an erroneous spawn the question is
whether the watcher's gate fired, which is a deterministic
decision the patch captures.
"""
mgr, _adapter = real_mgr
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
ws = mgr.create(user_id="u1", name="empty-int", skill=None)
# No enqueue.
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.set_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0, "wake must not dispatch for an empty queue"
finally:
watcher.shutdown()
@pytest.fixture
def coord_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter, FakeStorage]:
"""Real coord-side SessionManager with the adapter's kind set to
COORDINATOR. Same shape as ``real_mgr`` but for the coord half of
the lifespan. No StateWriter wired so subscriber dispatch fires
synchronously on the test thread.
"""
adapter = _BuildRealSessionAdapter(kind=WorkstreamKind.COORDINATOR)
storage = FakeStorage()
mgr = SessionManager(
adapter,
storage=storage,
max_active=5,
event_emitter=adapter,
)
return mgr, adapter, storage
def test_coord_idle_with_active_children_emits_envelope_via_real_managers(coord_mgr, tmp_db):
"""Full coord-path integration test (matches design doc §7.4).
Drives the production install order ``CoordinatorIdleObserver``
registered FIRST, then ``IdleNudgeWatcher`` and asserts the
full chain: observer enqueues on IDLE watcher peeks wake
spawns a worker ``deliver_wake_nudge_from_queue`` drains and
runs the synthetic empty-user turn reminder envelope reaches
the synthesized user message via the side-channel.
The boundary-crossing path tested here mirrors what
``console/server.py``'s lifespan does at production startup; if
the install order is ever reversed, this test fails.
"""
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
from turnstone.core.workstream import WorkstreamKind as _Kind
mgr, adapter, storage = coord_mgr
# Observer FIRST, then watcher. Same order as
# ``console/server.py:4435-4443`` — production correctness depends
# on subscribers firing in registration order on the same IDLE.
observer = CoordinatorIdleObserver(mgr, storage)
observer.start()
watcher = IdleNudgeWatcher(mgr)
watcher.start()
try:
coord = mgr.create(user_id="u1", name="parent-coord", skill=None)
assert coord.session is not None
# Two interactive children of the coord, both running. Use
# the storage's register_workstream API so the rows match
# production shape (the observer queries via list_workstreams).
storage.register_workstream(
"child-a",
user_id="u1",
name="research-pricing",
kind=_Kind.INTERACTIVE,
parent_ws_id=coord.id,
state="running",
)
storage.register_workstream(
"child-b",
user_id="u1",
name="draft-rfc",
kind=_Kind.INTERACTIVE,
parent_ws_id=coord.id,
state="thinking",
)
# Pretend the coord has already had a real conversation so
# ``should_nudge``'s message_count > 1 gate passes.
coord.session.messages.append({"role": "user", "content": "spawn 2"})
coord.session.messages.append({"role": "assistant", "content": "ok"})
with (
patch.object(coord.session, "_create_stream_with_retry", return_value=iter([])),
patch.object(
coord.session,
"_stream_response",
return_value={"role": "assistant", "content": "ack"},
),
patch.object(coord.session, "_full_messages", return_value=[]),
patch.object(coord.session, "_update_token_table"),
patch.object(coord.session, "_print_status_line"),
patch.object(coord.session, "_visible_memory_count", return_value=0),
patch("turnstone.core.session.save_message"),
):
coord.session._title_generated = True
mgr.set_state(coord.id, WorkstreamState.IDLE)
_wait_for_worker_done(coord)
# Queue drained — the wake delivered the observer's enqueue.
assert len(coord.session._nudge_queue) == 0
# The synthetic empty-user turn landed with a reminder containing
# both children.
user_msgs = [m for m in coord.session.messages if m.get("role") == "user"]
# Two real msgs (user + assistant context above) plus the wake.
wake_msg = user_msgs[-1]
assert wake_msg["content"] == ""
assert wake_msg.get("_source") == "system_nudge"
reminders = wake_msg.get("_reminders") or []
assert len(reminders) == 1
assert reminders[0]["type"] == "idle_children"
text = reminders[0]["text"]
assert "research-pricing" in text
assert "draft-rfc" in text
assert "child-a" in text
assert "child-b" in text
assert "wait_for_workstream" in text
finally:
watcher.shutdown()
observer.shutdown()
+165
View File
@@ -0,0 +1,165 @@
"""Unit tests for :class:`IdleNudgeWatcher`.
Drives a fake :class:`SessionManager` that mimics the real one's
``subscribe_to_state`` / ``get`` contract. The watcher itself
dispatches via ``turnstone.core.session_worker.send``; we patch that
module-level function to capture calls without spawning real threads.
"""
from __future__ import annotations
import contextlib
import threading
from typing import Any
from unittest.mock import patch
import pytest
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
from turnstone.core.nudge_queue import NudgeQueue
from turnstone.core.workstream import WorkstreamState
class _FakeSession:
def __init__(self) -> None:
self._nudge_queue = NudgeQueue()
self.deliver_wake_nudge_from_queue_called = 0
def deliver_wake_nudge_from_queue(self) -> None:
self.deliver_wake_nudge_from_queue_called += 1
class _FakeWorkstream:
def __init__(self, ws_id: str = "ws-test") -> None:
self.id = ws_id
self.session: _FakeSession | None = _FakeSession()
self._lock = threading.Lock()
self._worker_running = False
self._closed = False
self.worker_thread: Any = None
class _FakeManager:
"""Mimics SessionManager's subscribe-to-state surface without a DB."""
def __init__(self) -> None:
self._workstreams: dict[str, _FakeWorkstream] = {}
self._subscribers: list[Any] = []
self._subscribers_lock = threading.Lock()
def add_ws(self, ws: _FakeWorkstream) -> None:
self._workstreams[ws.id] = ws
def get(self, ws_id: str) -> _FakeWorkstream | None:
return self._workstreams.get(ws_id)
def subscribe_to_state(self, callback: Any) -> None:
with self._subscribers_lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Any) -> None:
with self._subscribers_lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
def fire_state(self, ws_id: str, state: WorkstreamState) -> None:
"""Mirror SessionManager.set_state's subscriber-fan-out behaviour."""
with self._subscribers_lock:
subs = list(self._subscribers)
for cb in subs:
# Match contextlib.suppress(Exception) in real SessionManager.
with contextlib.suppress(Exception):
cb(ws_id, state)
@pytest.fixture
def fake_mgr_and_ws() -> tuple[_FakeManager, _FakeWorkstream]:
mgr = _FakeManager()
ws = _FakeWorkstream()
mgr.add_ws(ws)
return mgr, ws
class TestIdleNudgeWatcher:
def test_idle_event_with_empty_queue_no_op(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_idle_event_with_pending_nudge_dispatches(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("idle_children", "your kids", "any")
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 1
kwargs = mock_send.call_args.kwargs
# `enqueue=lambda: None` — verify by calling and checking no-op.
assert kwargs["enqueue"]() is None
# `run` should call deliver_wake_nudge_from_queue when invoked.
kwargs["run"]()
assert ws.session.deliver_wake_nudge_from_queue_called == 1
assert kwargs["thread_name"].startswith("wake-nudge-")
def test_non_idle_state_ignored(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("foo", "bar", "any")
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
for state in (
WorkstreamState.RUNNING,
WorkstreamState.THINKING,
WorkstreamState.ATTENTION,
WorkstreamState.ERROR,
):
mgr.fire_state(ws.id, state)
assert mock_send.call_count == 0
def test_unknown_ws_ignored(self, fake_mgr_and_ws):
mgr, _ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state("ghost", WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_session_none_ignored(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session = None # workstream loaded but session not yet built
watcher = IdleNudgeWatcher(mgr)
watcher.start()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_start_is_idempotent(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
watcher.start() # no-op
ws.session._nudge_queue.enqueue("foo", "bar", "any")
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
# Only one subscriber was registered despite the double-start.
assert mock_send.call_count == 1
def test_shutdown_unsubscribes(self, fake_mgr_and_ws):
mgr, ws = fake_mgr_and_ws
ws.session._nudge_queue.enqueue("foo", "bar", "any")
watcher = IdleNudgeWatcher(mgr)
watcher.start()
watcher.shutdown()
with patch("turnstone.core.session_worker.send") as mock_send:
mgr.fire_state(ws.id, WorkstreamState.IDLE)
assert mock_send.call_count == 0
def test_shutdown_is_idempotent(self, fake_mgr_and_ws):
mgr, _ws = fake_mgr_and_ws
watcher = IdleNudgeWatcher(mgr)
watcher.start()
watcher.shutdown()
watcher.shutdown() # no error
+52
View File
@@ -777,6 +777,58 @@ class TestModelAliasResolution:
assert judge._client_factory_args["api_key"] == "alias-key"
assert judge._client_factory_args["provider_name"] == "openai"
def test_unknown_alias_inherits_session_model(self):
"""``judge.model`` is alias-only. A value that doesn't resolve
through the registry inherits the session model (same path as
an empty config.model) rather than getting pinned onto the
session provider as a raw model id that legacy behavior
silently broke whenever the session provider didn't speak the
configured model id (Anthropic session, ``judge.model =
"gpt-5-mini"`` every verdict came back as ``llm_fallback``)."""
session_provider = _make_mock_provider()
session_provider.provider_name = "anthropic"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
registry = MagicMock()
registry.has_alias.return_value = False # judge.model isn't an alias
config = JudgeConfig(enabled=True, model="gpt-5-mini")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
model_registry=registry,
)
assert judge._provider is session_provider
assert judge._model == "session-default-model"
# Context window mirrors the session, not the (uncalled) caps lookup.
assert judge._judge_context_window == 100_000
def test_empty_model_inherits_session_model(self):
"""Empty ``config.model`` is the documented self-consistency path."""
session_provider = _make_mock_provider()
session_provider.provider_name = "openai"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
config = JudgeConfig(enabled=True, model="")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
)
assert judge._provider is session_provider
assert judge._model == "session-default-model"
def test_coordinator_tool_call_returns_llm_verdict_not_fallback(self):
"""Happy-path regression for coordinator tool calls: with a properly
resolved provider, the verdict tier must be ``llm`` the
+3 -2
View File
@@ -420,8 +420,9 @@ class TestSkillCatalogDisclosure:
session.system_messages = []
session._agent_system_messages = []
session.reasoning_effort = "medium"
session._pending_tool_advisories = []
session._pending_user_advisories = []
from turnstone.core.nudge_queue import NudgeQueue
session._nudge_queue = NudgeQueue()
session._tool_search = None
session._mcp_client = None
session._notify_on_complete = "{}"
File diff suppressed because it is too large Load Diff
+211
View File
@@ -0,0 +1,211 @@
"""Integration tests for the Phase 9 admin bulk-revoke endpoint.
POST /v1/api/admin/mcp-servers/{name}/bulk-revoke clears every user's
OAuth token for a server (admin-side counterpart to the per-user
DELETE /v1/api/mcp/oauth/connections/{server_name} that shipped in
Phase 8).
Coverage:
- requires ``admin.mcp`` permission (401/403 without).
- 404 when the named server is missing.
- 400 when the server's ``auth_type`` is not ``oauth_user``.
- 200 + ``rows_deleted`` + ``consented_users_before`` on success.
- Audit row written with
``upstream_revoke_outcome="bulk_admin_no_upstream"``.
- Token rows are gone from ``mcp_user_tokens`` post-call.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.console.server import admin_mcp_bulk_revoke
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
class _InjectAdminMcp(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="admin-user",
scopes=frozenset({"approve"}),
token_source="config",
permissions=frozenset({"read", "write", "approve", "admin.mcp"}),
)
return await call_next(request)
class _InjectNoAdminMcp(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="regular-user",
scopes=frozenset({"approve"}),
token_source="jwt",
permissions=frozenset({"read", "write", "approve"}),
)
return await call_next(request)
def _build_app(storage: SQLiteBackend, *, with_admin_mcp: bool = True) -> Starlette:
mw = _InjectAdminMcp if with_admin_mcp else _InjectNoAdminMcp
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route(
"/api/admin/mcp-servers/{name}/bulk-revoke",
admin_mcp_bulk_revoke,
methods=["POST"],
),
],
),
],
middleware=[Middleware(mw)],
)
app.state.auth_storage = storage
return app
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
backend: SQLiteBackend,
*,
name: str = "srv-oauth",
server_id: str = "srv-oauth-id",
) -> None:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="oauth_user",
)
def _seed_static_server(
backend: SQLiteBackend,
*,
name: str = "srv-static",
server_id: str = "srv-static-id",
) -> None:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="static",
)
def _seed_user_tokens(backend: SQLiteBackend, server_name: str, users: int) -> None:
for i in range(users):
backend.create_mcp_user_token(
f"user-{i}",
server_name,
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=None,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
def test_requires_admin_mcp_permission(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage, with_admin_mcp=False))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 403
def test_404_on_missing_server(storage: SQLiteBackend) -> None:
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/never-existed/bulk-revoke")
assert resp.status_code == 404
assert resp.json() == {"error": "No such server"}
def test_400_on_static_server(storage: SQLiteBackend) -> None:
_seed_static_server(storage)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-static/bulk-revoke")
assert resp.status_code == 400
body = resp.json()
assert "oauth_user" in body["error"]
def test_400_on_invalid_server_name(storage: SQLiteBackend) -> None:
# double-underscore is reserved for the prefixed-tool-name encoding.
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/bad__name/bulk-revoke")
assert resp.status_code == 400
def test_200_on_success_with_no_consented_users(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
assert body["rows_deleted"] == 0
assert body["consented_users_before"] == 0
def test_200_clears_all_user_tokens(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_user_tokens(storage, "srv-oauth", users=3)
# Token for another server must survive the bulk-revoke.
_seed_oauth_server(storage, name="srv-other", server_id="srv-other-id")
_seed_user_tokens(storage, "srv-other", users=2)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
assert body["rows_deleted"] == 3
assert body["consented_users_before"] == 3
# Target server's tokens are gone; bystander's tokens survive.
assert storage.count_mcp_consented_users_by_server("srv-oauth") == 0
assert storage.count_mcp_consented_users_by_server("srv-other") == 2
def test_audits_with_bulk_admin_no_upstream(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_user_tokens(storage, "srv-oauth", users=2)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
# Pull the most-recent audit row for the bulk_revoked action and
# verify it carries the deferral marker.
events = storage.list_audit_events(limit=10)
bulk_rows = [e for e in events if e.get("action") == "mcp_server.oauth.bulk_revoked"]
assert len(bulk_rows) == 1
detail = bulk_rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("upstream_revoke_outcome") == "bulk_admin_no_upstream"
assert detail.get("rows_deleted") == 2
assert detail.get("consented_users_before") == 2
assert detail.get("name") == "srv-oauth"
+931 -148
View File
File diff suppressed because it is too large Load Diff
+117
View File
@@ -0,0 +1,117 @@
"""Structural gate against the Phase 7b sibling-bug pattern.
Phase 7b's bug-1 was a single ``f"MCP X error: {e}"`` site dropping a
structured-error JSON. Phase 8 introduces the ``consent_url`` field on
the same JSON envelope: every ``_structured_error(...)`` invocation
that emits ``mcp_consent_required`` or ``mcp_insufficient_scope`` MUST
also pass a ``consent_url=`` kwarg, otherwise the dashboard renderer
can't surface a re-consent button.
This test is purely structural it scans the source of
:mod:`turnstone.core.mcp_client` and asserts every consent-required /
insufficient-scope ``_structured_error`` call carries
``consent_url=``. It catches future regressions where a new exec path
adds a fourth call site and forgets the kwarg.
"""
from __future__ import annotations
import re
from pathlib import Path
import turnstone.core.mcp_client as _mcp_client_module
_USER_ACTIONABLE_CODES = ("mcp_consent_required", "mcp_insufficient_scope")
def _read_source() -> str:
path = Path(_mcp_client_module.__file__)
return path.read_text(encoding="utf-8")
def _find_structured_error_blocks(source: str) -> list[tuple[int, str]]:
"""Return ``(line_no, block)`` pairs for every ``_structured_error(...)``.
Each block is the call's argument list expanded across however many
lines the formatter chose. Uses a paren-counting walk so multi-line
kwargs and nested expressions are captured correctly.
"""
blocks: list[tuple[int, str]] = []
needle = "_structured_error("
idx = 0
while True:
loc = source.find(needle, idx)
if loc < 0:
break
# Skip the function definition itself.
if source[loc - 4 : loc] == "def ":
idx = loc + len(needle)
continue
line_no = source.count("\n", 0, loc) + 1
depth = 1
end = loc + len(needle)
while end < len(source) and depth > 0:
ch = source[end]
if ch == "(":
depth += 1
elif ch == ")":
depth -= 1
end += 1
blocks.append((line_no, source[loc:end]))
idx = end
return blocks
def test_every_user_actionable_structured_error_passes_consent_url() -> None:
source = _read_source()
blocks = _find_structured_error_blocks(source)
user_actionable_blocks = [
(ln, blk)
for ln, blk in blocks
if any(f'code="{code}"' in blk for code in _USER_ACTIONABLE_CODES)
]
# Sanity check: ensure we actually scanned the file the audit cares
# about (a stale path or import would otherwise silently pass with
# zero matches).
assert user_actionable_blocks, (
"No mcp_consent_required / mcp_insufficient_scope _structured_error "
"call sites found — has the audit been pointed at the wrong file?"
)
missing: list[tuple[int, str]] = []
for ln, blk in user_actionable_blocks:
if "consent_url=" not in blk:
# Strip whitespace and truncate so the failure message is
# readable in CI.
collapsed = re.sub(r"\s+", " ", blk).strip()
missing.append((ln, collapsed[:200]))
assert not missing, (
"Sibling-bug regression: the following consent-required / "
"insufficient-scope _structured_error sites are missing the "
"consent_url= kwarg.\n" + "\n".join(f" line {ln}: {snippet}" for ln, snippet in missing)
)
def test_audit_finds_all_known_user_actionable_sites() -> None:
"""Lock the count so accidental deletions are caught.
There are 13 user-actionable ``_structured_error`` call sites today
(4 each in the tool / resource / prompt token-classify branches +
3 in the post-retry-failed branches + 1 in ``_handle_auth_403``'s
insufficient-scope branch). If a new exec path is added the count
can rise; if a branch is removed the count can fall both are
fine, but require an intentional bump of this number to confirm
the change went through review.
"""
source = _read_source()
blocks = _find_structured_error_blocks(source)
user_actionable_count = sum(
1 for _, blk in blocks if any(f'code="{code}"' in blk for code in _USER_ACTIONABLE_CODES)
)
assert user_actionable_count == 13, (
f"Expected 13 user-actionable _structured_error sites, got "
f"{user_actionable_count}. If this is intentional, bump the "
f"expected count and document why in the commit message."
)
+235
View File
@@ -0,0 +1,235 @@
"""Tests for ``turnstone.core.mcp_crypto`` cipher + config loading.
Covers token-at-rest encryption for OAuth-MCP.
"""
from __future__ import annotations
import base64
import pytest
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenDecryptError,
MCPTokenKeyConfigError,
_key_fingerprint,
_validate_key,
load_mcp_token_cipher_config,
)
def _new_raw_key() -> bytes:
"""Return a fresh 32-byte Fernet key as raw bytes (post-base64-decode)."""
return base64.urlsafe_b64decode(Fernet.generate_key())
# ---------------------------------------------------------------------------
# Cipher round-trip
# ---------------------------------------------------------------------------
class TestCipherRoundTrip:
def test_round_trip_single_key(self) -> None:
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
plaintext = b"access_token_12345"
ct = cipher.encrypt(plaintext)
assert ct != plaintext
assert cipher.decrypt(ct) == plaintext
def test_round_trip_unicode_token(self) -> None:
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
# Tokens may legitimately carry UTF-8 bytes (e.g. JWT with
# non-ASCII claim values). Round-trip a multi-byte sequence.
plaintext = "tok_é中💯".encode()
ct = cipher.encrypt(plaintext)
assert cipher.decrypt(ct) == plaintext
def test_wrong_key_raises_decrypt_error(self) -> None:
cipher_a = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
cipher_b = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
ct = cipher_a.encrypt(b"secret")
with pytest.raises(MCPTokenDecryptError) as exc_info:
cipher_b.decrypt(ct)
# Audit-trail correlation: error must carry the fingerprints of
# the keys actually attempted, not a placeholder.
assert exc_info.value.key_fingerprints_attempted
assert exc_info.value.key_fingerprints_attempted == cipher_b.key_fingerprints
# ---------------------------------------------------------------------------
# Rotation (MultiFernet behavior)
# ---------------------------------------------------------------------------
class TestRotation:
def test_rotation_forward(self) -> None:
"""Encrypt with a new-only cipher, decrypt with a [v2, v1] cluster.
Mirrors the operational situation where a node already has the
rotated key list installed and a peer just wrote a row under v2.
"""
v1 = _new_raw_key()
v2 = _new_raw_key()
new_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2,)))
cluster = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
ct = new_only.encrypt(b"hello")
assert cluster.decrypt(ct) == b"hello"
def test_rotation_backward_keeps_old_decryptable(self) -> None:
"""A row written under the OLD key (v1) must still decrypt after
rotation places v2 first and keeps v1 as fallback."""
v1 = _new_raw_key()
v2 = _new_raw_key()
old_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1,)))
rotated = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
ct = old_only.encrypt(b"legacy")
assert rotated.decrypt(ct) == b"legacy"
# ---------------------------------------------------------------------------
# Config loader
# ---------------------------------------------------------------------------
def _patch_load_config(monkeypatch: pytest.MonkeyPatch, payload: dict) -> None:
"""Override ``turnstone.core.config.load_config`` to return ``payload``
when the ``"security"`` section is requested."""
def fake(section: str | None = None) -> dict:
if section == "security":
return payload
return {}
import turnstone.core.config as cfg_mod
monkeypatch.setattr(cfg_mod, "load_config", fake)
class TestLoadConfig:
def test_load_singular_key(self, monkeypatch: pytest.MonkeyPatch) -> None:
key = Fernet.generate_key().decode()
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": key})
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 1
def test_load_plural_overrides_singular(self, monkeypatch: pytest.MonkeyPatch) -> None:
plural = [Fernet.generate_key().decode(), Fernet.generate_key().decode()]
_patch_load_config(
monkeypatch,
{
"mcp_token_encryption_keys": plural,
"mcp_token_encryption_key": Fernet.generate_key().decode(),
},
)
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 2 # plural wins, singular ignored
def test_load_returns_none_when_absent(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {})
assert load_mcp_token_cipher_config() is None
def test_load_empty_plural_falls_through_to_singular(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Operator wrote ``mcp_token_encryption_keys = []`` AND set a
singular value: empty plural is treated as absent."""
key = Fernet.generate_key().decode()
_patch_load_config(
monkeypatch,
{"mcp_token_encryption_keys": [], "mcp_token_encryption_key": key},
)
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 1
def test_load_invalid_base64_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": "###not-base64###"})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
# Operator-facing hint is part of every error message.
assert "regenerate with:" in str(exc_info.value)
def test_load_wrong_length_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
# 24 raw bytes → 32 base64 chars; not 32 raw bytes after decode.
short_key = base64.urlsafe_b64encode(b"\x00" * 24).decode()
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": short_key})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
assert "32 bytes" in str(exc_info.value)
def test_load_non_list_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": "single-string-not-list"})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
assert "list" in str(exc_info.value).lower()
def test_load_non_string_in_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": [12345]})
with pytest.raises(MCPTokenKeyConfigError):
load_mcp_token_cipher_config()
# ---------------------------------------------------------------------------
# Fingerprint stability
# ---------------------------------------------------------------------------
class TestFingerprint:
def test_key_fingerprint_stable_and_short(self) -> None:
key = _new_raw_key()
fp1 = _key_fingerprint(key)
fp2 = _key_fingerprint(key)
assert fp1 == fp2
# 8 bytes -> 16 hex characters.
assert len(fp1) == 16
assert all(c in "0123456789abcdef" for c in fp1)
def test_different_keys_have_different_fingerprints(self) -> None:
fp1 = _key_fingerprint(_new_raw_key())
fp2 = _key_fingerprint(_new_raw_key())
assert fp1 != fp2
def test_cipher_fingerprints_match_keys(self) -> None:
v1 = _new_raw_key()
v2 = _new_raw_key()
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1, v2)))
assert cipher.key_fingerprints == (
_key_fingerprint(v1),
_key_fingerprint(v2),
)
# ---------------------------------------------------------------------------
# Direct ``_validate_key`` — exercises edge cases not reachable via loader
# ---------------------------------------------------------------------------
class TestValidateKey:
def test_empty_string_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
_validate_key("", label="x")
def test_whitespace_only_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
_validate_key(" ", label="x")
def test_label_propagated_in_error(self) -> None:
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
_validate_key("###", label="my_label_42")
assert "my_label_42" in str(exc_info.value)
# ---------------------------------------------------------------------------
# MCPTokenCipher constructor guard
# ---------------------------------------------------------------------------
class TestCipherConstructorGuard:
def test_empty_keys_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
MCPTokenCipher(MCPTokenCipherConfig(keys=()))
+28 -27
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
from typing import Any
from tests.conftest import _seed_static_state
from turnstone.core.mcp_client import MCPClientManager
# ---------------------------------------------------------------------------
@@ -105,14 +106,18 @@ class TestRemoveServerSync:
"""remove_server_sync cleans up all per-server state dicts."""
mgr = MCPClientManager({"test": {"command": "echo"}})
# Simulate state as if the server was connected
mgr._per_server_tools["test"] = [_fake_openai_tool()]
mgr._per_server_resources["test"] = [_fake_resource_dict()]
mgr._per_server_prompts["test"] = [_fake_prompt_dict()]
mgr._supports_list_changed["test"] = True
mgr._supports_resources["test"] = True
mgr._supports_resource_list_changed["test"] = True
mgr._supports_prompts["test"] = True
mgr._supports_prompt_list_changed["test"] = True
_seed_static_state(
mgr,
"test",
tools=[_fake_openai_tool()],
resources=[_fake_resource_dict()],
prompts=[_fake_prompt_dict()],
supports_list_changed=True,
supports_resources=True,
supports_resource_list_changed=True,
supports_prompts=True,
supports_prompt_list_changed=True,
)
mgr._rebuild_tools()
mgr._rebuild_resources()
mgr._rebuild_prompts()
@@ -127,14 +132,7 @@ class TestRemoveServerSync:
assert len(mgr.get_tools()) == 0
assert mgr.resource_count == 0
assert mgr.prompt_count == 0
assert "test" not in mgr._per_server_tools
assert "test" not in mgr._per_server_resources
assert "test" not in mgr._per_server_prompts
assert "test" not in mgr._supports_list_changed
assert "test" not in mgr._supports_resources
assert "test" not in mgr._supports_resource_list_changed
assert "test" not in mgr._supports_prompts
assert "test" not in mgr._supports_prompt_list_changed
assert "test" not in mgr._static_servers
def test_removes_config_to_prevent_reconnect(self) -> None:
"""remove_server_sync removes from _server_configs to prevent reconnect."""
@@ -146,8 +144,8 @@ class TestRemoveServerSync:
def test_preserves_other_servers(self) -> None:
"""Removing one server does not affect another server's state."""
mgr = MCPClientManager({"srv_a": {}, "srv_b": {}})
mgr._per_server_tools["srv_a"] = [_fake_openai_tool("mcp__srv_a__foo")]
mgr._per_server_tools["srv_b"] = [_fake_openai_tool("mcp__srv_b__bar")]
_seed_static_state(mgr, "srv_a", tools=[_fake_openai_tool("mcp__srv_a__foo")])
_seed_static_state(mgr, "srv_b", tools=[_fake_openai_tool("mcp__srv_b__bar")])
mgr._rebuild_tools()
assert len(mgr.get_tools()) == 2
@@ -179,13 +177,17 @@ class TestGetServerStatus:
"""Status of a connected server reports correct tool/resource/prompt counts."""
mgr = MCPClientManager({"test": {}})
# Simulate connected state
mgr._sessions["test"] = object() # any truthy value
mgr._per_server_tools["test"] = [
_fake_openai_tool("mcp__test__a"),
_fake_openai_tool("mcp__test__b"),
]
mgr._per_server_resources["test"] = [_fake_resource_dict()]
mgr._per_server_prompts["test"] = [_fake_prompt_dict()]
_seed_static_state(
mgr,
"test",
session=object(), # any truthy value
tools=[
_fake_openai_tool("mcp__test__a"),
_fake_openai_tool("mcp__test__b"),
],
resources=[_fake_resource_dict()],
prompts=[_fake_prompt_dict()],
)
status = mgr.get_server_status("test")
assert status["connected"] is True
@@ -225,8 +227,7 @@ class TestGetAllServerStatus:
def test_mixed_connected_and_disconnected(self) -> None:
"""Status correctly reflects a mix of connected and disconnected servers."""
mgr = MCPClientManager({"up": {}, "down": {}})
mgr._sessions["up"] = object()
mgr._per_server_tools["up"] = [_fake_openai_tool("mcp__up__x")]
_seed_static_state(mgr, "up", session=object(), tools=[_fake_openai_tool("mcp__up__x")])
statuses = mgr.get_all_server_status()
assert statuses["up"]["connected"] is True
+242
View File
@@ -0,0 +1,242 @@
"""Unit tests for ``turnstone.core.mcp_http_parsers``.
The parser replaces the prior hand-rolled scanners that used
``header.lower().find("scope")`` to locate parameter names that approach
misparsed ``scope`` embedded inside other tokens (``xscope``) or inside
quoted-string values of preceding params. Each adversarial case below
asserts the new tokenizer respects RFC 7235 ``challenge auth-param``
boundaries; the docstrings document the equivalent input that broke the
naive parser. Negative-test verification: temporarily reverting
``parse_www_authenticate_scope`` to delegate to ``header.lower().find("scope")``
makes ``test_scope_inside_realm_value`` and ``test_scope_inside_xscope`` fail.
"""
from __future__ import annotations
import time
import pytest
from turnstone.core.mcp_http_parsers import (
parse_www_authenticate_bearer,
parse_www_authenticate_error,
parse_www_authenticate_scope,
)
class TestParseScope:
def test_basic_scope(self) -> None:
header = 'Bearer error="insufficient_scope", scope="files:read mail:send"'
assert parse_www_authenticate_scope(header) == ("files:read", "mail:send")
def test_no_scope_param(self) -> None:
assert parse_www_authenticate_scope('Bearer error="invalid_token"') == ()
def test_unterminated_quoted_string_returns_empty(self) -> None:
assert parse_www_authenticate_scope('Bearer scope="files:read') == ()
def test_escaped_chars_in_value_drops_invalid_scope_token(self) -> None:
# RFC 7230 §3.2.6 backslash escapes decode the literal scope to
# ``files:read "weird"``. RFC 6749 §3.3 ``scope-token`` forbids
# ``"``, so ``"weird"`` is dropped and only ``files:read``
# survives the post-split validation.
header = r'Bearer scope="files:read \"weird\""'
assert parse_www_authenticate_scope(header) == ("files:read",)
def test_empty_string(self) -> None:
assert parse_www_authenticate_scope("") == ()
def test_unquoted_scope_value(self) -> None:
# Unquoted single token.
assert parse_www_authenticate_scope("Bearer scope=files:read") == ("files:read",)
# --- the four headline misparse cases ---
def test_scope_inside_xscope(self) -> None:
"""``Bearer xscope="value"`` must NOT be read as ``scope``.
The naive ``find("scope")`` matched at position 7 inside
``xscope`` and returned ``("value",)``.
"""
assert parse_www_authenticate_scope('Bearer xscope="value"') == ()
def test_scope_inside_realm_value(self) -> None:
"""``Bearer realm="my scope=fake", scope="real"`` must return ``("real",)``.
The naive parser found ``scope=`` inside the quoted ``realm``
value first and returned ``("fake",)``.
"""
header = 'Bearer realm="my scope=fake", scope="real"'
assert parse_www_authenticate_scope(header) == ("real",)
def test_scope_inside_quoted_realm_with_escaped_quotes(self) -> None:
"""``Bearer realm="foo scope=\\"admin:write\\" bar"`` returns ``()``.
The inner ``scope=`` is wholly inside the quoted-string value of
``realm`` there is no top-level ``scope`` auth-param, so the
result is empty.
"""
header = r'Bearer realm="foo scope=\"admin:write\" bar"'
assert parse_www_authenticate_scope(header) == ()
def test_scope_token_validation_drops_control_bytes(self) -> None:
"""Tokens containing CR / LF / tab / DEL / quote are dropped.
RFC 6749 §3.3 restricts ``scope-token`` to visible ASCII
excluding ``"`` and ``\\``. The splitter applies that
validation so a malicious AS cannot smuggle CRLF (or the like)
through a future log / notification path that prints the scope
list verbatim. ``"a\\rb"`` and ``"\\nc"`` fail validation;
``"d"`` survives. The legitimate space separator splits ``d``
into its own token.
"""
# Build via concatenation so the assertion stays intelligible.
header = 'Bearer scope="a\rb \nc d"'
assert parse_www_authenticate_scope(header) == ("d",)
class TestParseError:
def test_basic_quoted_error(self) -> None:
assert (
parse_www_authenticate_error('Bearer error="insufficient_scope"')
== "insufficient_scope"
)
def test_other_quoted_error_tokens(self) -> None:
assert parse_www_authenticate_error('Bearer error="invalid_token"') == "invalid_token"
assert parse_www_authenticate_error('Bearer error="invalid_request"') == "invalid_request"
def test_no_error_param(self) -> None:
assert parse_www_authenticate_error("Bearer realm=foo") is None
def test_error_description_does_not_match_error(self) -> None:
"""``error_description`` is its own auth-param key, not ``error``.
The tokenizer reads ``_`` as part of the token (RFC 7230 ``tchar``),
so ``error_description`` becomes one key, ``error`` another.
"""
assert parse_www_authenticate_error('Bearer error_description="bad"') is None
def test_unquoted_error(self) -> None:
# Some ASes don't quote the error token.
assert (
parse_www_authenticate_error("Bearer error=insufficient_scope") == "insufficient_scope"
)
def test_empty_string(self) -> None:
assert parse_www_authenticate_error("") is None
def test_error_inside_realm_value(self) -> None:
"""``Bearer realm="my error=fake", error="real"`` must return ``"real"``.
Naive parser grabbed ``fake`` from inside the ``realm`` quoted
value.
"""
header = 'Bearer realm="my error=fake", error="real"'
assert parse_www_authenticate_error(header) == "real"
class TestBearerDict:
def test_returns_lowercased_keys(self) -> None:
header = 'Bearer Realm="x", Error="y", Scope="a b"'
params = parse_www_authenticate_bearer(header)
assert params == {"realm": "x", "error": "y", "scope": "a b"}
def test_non_bearer_scheme_returns_empty(self) -> None:
assert parse_www_authenticate_bearer('Basic realm="x"') == {}
def test_no_scheme(self) -> None:
assert parse_www_authenticate_bearer('realm="x"') == {}
def test_bearer_only_no_params(self) -> None:
assert parse_www_authenticate_bearer("Bearer ") == {}
def test_bearer_with_no_space_returns_empty(self) -> None:
# ``BearerToken`` is not a Bearer challenge (no separator).
assert parse_www_authenticate_bearer("BearerToken") == {}
def test_first_value_wins_on_duplicate(self) -> None:
# If a malformed AS sends two ``scope=`` params we keep the first.
# The earlier ``find()``-based scanner would have returned the
# last; either choice is legal for malformed input but we need
# to be consistent.
header = 'Bearer scope="first", scope="second"'
assert parse_www_authenticate_bearer(header) == {"scope": "first"}
def test_trailing_comma(self) -> None:
header = 'Bearer error="x",'
assert parse_www_authenticate_bearer(header) == {"error": "x"}
def test_multiple_commas(self) -> None:
header = 'Bearer ,, error="x",,, scope="y"'
assert parse_www_authenticate_bearer(header) == {"error": "x", "scope": "y"}
def test_embedded_escaped_quote(self) -> None:
header = r'Bearer realm="he said \"hi\""'
assert parse_www_authenticate_bearer(header) == {"realm": 'he said "hi"'}
def test_param_without_value_skipped(self) -> None:
header = 'Bearer realm, error="x"'
# ``realm`` without ``=`` is dropped; ``error`` survives.
assert parse_www_authenticate_bearer(header) == {"error": "x"}
@pytest.mark.parametrize(
"header,expected",
[
("", {}),
("Bearer", {}),
('Bearer realm=""', {"realm": ""}),
('Bearer realm="", scope=""', {"realm": "", "scope": ""}),
],
)
def test_edge_cases(self, header: str, expected: dict[str, str]) -> None:
assert parse_www_authenticate_bearer(header) == expected
class TestPathologicalInput:
def test_oversized_pathological_input_rejected_under_50ms(self) -> None:
"""Headers longer than the defensive cap return ``{}`` immediately.
The cap is set to 4096 bytes real ASes emit a few hundred bytes
at most. This guards both ``parse_www_authenticate_bearer``
callers against pathological input from a misbehaving server.
The previous ``header.lower().find("scope", i)`` loop was
O(N**2) a 100 KB header with no ``=`` took ~330 ms because
each ``find`` rescanned the entire suffix. The single-pass
tokenizer (capped at 4 KB) reduces this to a one-shot length
check that returns ``{}`` in microseconds, so the budget is
generous regardless of which side of the cap was hit.
"""
big = "Bearer scope=" + "a" * 10_000
start = time.perf_counter()
result = parse_www_authenticate_scope(big)
elapsed = time.perf_counter() - start
assert result == ()
assert elapsed < 0.05, f"oversized-header reject took {elapsed * 1000:.1f}ms"
def test_within_cap_long_header_under_50ms(self) -> None:
"""A 4 KB header with thousands of ``find`` candidates still parses fast.
Stays under the cap so the tokenizer actually runs end to end
the goal is to prove the inner loop is O(N), not just that the
cap rejects oversized input.
"""
# Pack the header right up to the cap with non-matching
# auth-params, then put the real ``scope`` at the end.
filler_parts = []
size = len("Bearer ")
i = 0
while size < 3900:
part = f'xscope{i}="ignore", '
if size + len(part) > 3900:
break
filler_parts.append(part)
size += len(part)
i += 1
header = "Bearer " + "".join(filler_parts) + 'scope="real"'
assert len(header) <= 4096
start = time.perf_counter()
result = parse_www_authenticate_scope(header)
elapsed = time.perf_counter() - start
assert result == ("real",)
assert elapsed < 0.05, f"4kb tokenize took {elapsed * 1000:.1f}ms"
+81 -66
View File
@@ -14,6 +14,7 @@ from unittest.mock import AsyncMock, MagicMock
import pytest
from tests.conftest import _seed_static_state
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.storage._sqlite import SQLiteBackend
@@ -109,13 +110,15 @@ class TestFullLifecycleResourcesPrompts:
def test_rebuild_resources_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_resources merges per-server resources into a unified list."""
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
_make_resource("file:///b.txt", "b", "alpha"),
]
mgr._per_server_resources["beta"] = [
_make_resource("file:///c.txt", "c", "beta"),
]
_seed_static_state(
mgr,
"alpha",
resources=[
_make_resource("file:///a.txt", "a", "alpha"),
_make_resource("file:///b.txt", "b", "alpha"),
],
)
_seed_static_state(mgr, "beta", resources=[_make_resource("file:///c.txt", "c", "beta")])
mgr._rebuild_resources()
@@ -130,13 +133,19 @@ class TestFullLifecycleResourcesPrompts:
def test_rebuild_prompts_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_prompts merges per-server prompts into a unified list."""
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._per_server_prompts["beta"] = [
_make_prompt("mcp__beta__summarize", "summarize", "beta", "Summarize text"),
_make_prompt("mcp__beta__translate", "translate", "beta", "Translate text"),
]
_seed_static_state(
mgr,
"alpha",
prompts=[_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello")],
)
_seed_static_state(
mgr,
"beta",
prompts=[
_make_prompt("mcp__beta__summarize", "summarize", "beta", "Summarize text"),
_make_prompt("mcp__beta__translate", "translate", "beta", "Translate text"),
],
)
mgr._rebuild_prompts()
@@ -164,10 +173,12 @@ class TestFullLifecycleResourcesPrompts:
try:
# Populate session and resource map
session = _make_mock_session()
mgr._sessions["alpha"] = session
mgr._per_server_resources["alpha"] = [
_make_resource("file:///readme.md", "readme", "alpha"),
]
_seed_static_state(
mgr,
"alpha",
session=session,
resources=[_make_resource("file:///readme.md", "readme", "alpha")],
)
mgr._rebuild_resources()
result = mgr.read_resource_sync("file:///readme.md", timeout=5)
@@ -194,18 +205,22 @@ class TestFullLifecycleResourcesPrompts:
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
# Register a template resource (no concrete resources)
mgr._per_server_resources["alpha"] = [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "Fetch a row",
"mimeType": "application/json",
"server": "alpha",
"template": True,
},
]
_seed_static_state(
mgr,
"alpha",
session=session,
resources=[
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "Fetch a row",
"mimeType": "application/json",
"server": "alpha",
"template": True,
},
],
)
mgr._rebuild_resources()
# Template should not be in _resource_map
@@ -230,10 +245,12 @@ class TestFullLifecycleResourcesPrompts:
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
_seed_static_state(
mgr,
"alpha",
session=session,
prompts=[_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello")],
)
mgr._rebuild_prompts()
messages = mgr.get_prompt_sync(
@@ -314,40 +331,42 @@ class TestFullLifecycleResourcesPrompts:
def test_shutdown_clears_all_state(self, mgr: MCPClientManager) -> None:
"""shutdown() clears sessions, tools, resources, prompts, and listeners."""
# Populate state
mgr._sessions["alpha"] = MagicMock()
mgr._per_server_tools["alpha"] = [
{
"type": "function",
"function": {
"name": "mcp__alpha__search",
"description": "Search",
"parameters": {},
_seed_static_state(
mgr,
"alpha",
session=MagicMock(),
tools=[
{
"type": "function",
"function": {
"name": "mcp__alpha__search",
"description": "Search",
"parameters": {},
},
}
],
resources=[
_make_resource("file:///a.txt", "a", "alpha"),
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "alpha",
"template": True,
},
}
]
],
prompts=[_make_prompt("mcp__alpha__greet", "greet", "alpha")],
)
mgr._rebuild_tools()
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha"),
]
mgr._rebuild_prompts()
mgr._listeners.append(lambda: None)
mgr._resource_listeners.append(lambda: None)
mgr._prompt_listeners.append(lambda: None)
# Verify populated
assert len(mgr._sessions) == 1
assert len(mgr._static_servers) == 1
assert len(mgr._tools) == 1
assert len(mgr._resources) == 2 # 1 concrete + 1 template
assert len(mgr._template_prefixes) == 1
@@ -355,7 +374,7 @@ class TestFullLifecycleResourcesPrompts:
mgr.shutdown()
assert len(mgr._sessions) == 0
assert len(mgr._static_servers) == 0
assert len(mgr._tools) == 0
assert len(mgr._tool_map) == 0
assert len(mgr._resources) == 0
@@ -376,19 +395,15 @@ class TestFullLifecycleResourcesPrompts:
mgr.add_resource_listener(lambda: resource_fired.append(1))
mgr.add_prompt_listener(lambda: prompt_fired.append(1))
mgr._per_server_tools["alpha"] = []
_seed_static_state(mgr, "alpha", tools=[])
mgr._rebuild_tools()
assert len(tool_fired) == 1
mgr._per_server_resources["alpha"] = [
_make_resource("file:///x.txt", "x", "alpha"),
]
_seed_static_state(mgr, "alpha", resources=[_make_resource("file:///x.txt", "x", "alpha")])
mgr._rebuild_resources()
assert len(resource_fired) == 1
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__p1", "p1", "alpha"),
]
_seed_static_state(mgr, "alpha", prompts=[_make_prompt("mcp__alpha__p1", "p1", "alpha")])
mgr._rebuild_prompts()
assert len(prompt_fired) == 1
+780
View File
@@ -0,0 +1,780 @@
"""Integration tests for the MCP OAuth ``/connections`` endpoints.
Covers the list and revoke handlers that surface user-owned MCP server
consents to the settings UI:
* ``GET /v1/api/mcp/oauth/connections`` non-secret projection only.
* ``DELETE /v1/api/mcp/oauth/connections/{server_name}`` best-effort
upstream revoke (RFC 7009) followed by the authoritative local
delete; cross-user attempts return 404 with the exact same body
shape as a never-existed row to avoid leaking tenant existence.
"""
from __future__ import annotations
import asyncio
from typing import TYPE_CHECKING, Any
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from tests.conftest import make_mcp_token_cipher
from turnstone.core.auth import AuthResult
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import (
handle_mcp_oauth_list_connections,
handle_mcp_oauth_revoke_connection,
)
from turnstone.core.oidc import OIDCConfig
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
# ---------------------------------------------------------------------------
# Fixtures + helpers (mirror tests/test_mcp_oauth_handlers.py)
# ---------------------------------------------------------------------------
class _InjectAuthMiddleware(BaseHTTPMiddleware):
"""Stamp a fixed authenticated user on every request."""
def __init__(self, app: Any, user_id: str = "user-1") -> None:
super().__init__(app)
self._user_id = user_id
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id=self._user_id,
scopes=frozenset({"write"}),
token_source="config",
permissions=frozenset({"read", "write"}),
)
return await call_next(request)
class _NoAuthMiddleware(BaseHTTPMiddleware):
"""Leave ``request.state.auth_result`` unset so handlers see anon."""
async def dispatch(self, request: Request, call_next: Any) -> Response:
return await call_next(request)
async def _list_handler(request: Request) -> Response:
return await handle_mcp_oauth_list_connections(request)
async def _revoke_handler(request: Request) -> Response:
return await handle_mcp_oauth_revoke_connection(request)
def _build_app(
*,
storage: SQLiteBackend,
http_client: httpx.AsyncClient | MagicMock,
token_store: MCPTokenStore | None,
user_id: str = "user-1",
mcp_client: Any = None,
authenticated: bool = True,
) -> Starlette:
middleware: list[Middleware]
if authenticated:
middleware = [Middleware(_InjectAuthMiddleware, user_id=user_id)]
else:
middleware = [Middleware(_NoAuthMiddleware)]
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/mcp/oauth/connections", _list_handler),
Route(
"/api/mcp/oauth/connections/{server_name}",
_revoke_handler,
methods=["DELETE"],
),
],
),
],
middleware=middleware,
)
app.state.auth_storage = storage
app.state.mcp_token_store = token_store
app.state.mcp_oauth_http_client = http_client
app.state.mcp_oauth_refresh_locks = {}
app.state.mcp_oauth_dcr_locks = {}
app.state.mcp_oauth_metadata_cache = {}
app.state.mcp_oauth_last_cleanup_monotonic = 0.0
app.state.oidc_config = OIDCConfig(enabled=False, redirect_base="https://testserver")
if mcp_client is not None:
app.state.mcp_client = mcp_client
return app
def _make_token_store(backend: SQLiteBackend) -> MCPTokenStore:
return MCPTokenStore(backend, make_mcp_token_cipher(), node_id="test")
def _seed_oauth_user_server(
backend: SQLiteBackend,
*,
name: str = "srv-oauth",
server_id: str = "srv-id-1",
cached_issuer: str | None = "https://as.example.com",
) -> str:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid profile",
oauth_audience="https://mcp.example.com",
oauth_authorization_server_url=None,
)
if cached_issuer is not None:
backend.update_mcp_server(server_id, oauth_as_issuer_cached=cached_issuer)
return server_id
def _seed_user_token(
token_store: MCPTokenStore,
*,
user_id: str = "user-1",
server_name: str = "srv-oauth",
refresh_token: str | None = "refresh-secret",
) -> None:
token_store.create_user_token(
user_id,
server_name,
access_token="access-secret",
refresh_token=refresh_token,
expires_at="2099-12-31T00:00:00",
scopes="openid profile",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _good_as_metadata_doc(
*, revocation_endpoint: str | None = "https://as.example.com/revoke"
) -> dict[str, Any]:
doc: dict[str, Any] = {
"issuer": "https://as.example.com",
"authorization_endpoint": "https://as.example.com/authorize",
"token_endpoint": "https://as.example.com/token",
"registration_endpoint": "https://as.example.com/register",
"jwks_uri": "https://as.example.com/jwks",
"code_challenge_methods_supported": ["S256"],
"token_endpoint_auth_methods_supported": ["none", "client_secret_basic"],
}
if revocation_endpoint is not None:
doc["revocation_endpoint"] = revocation_endpoint
return doc
def _mk_response(
status_code: int = 200,
json_body: Any = None,
headers: dict[str, str] | None = None,
) -> MagicMock:
import json as _json
resp = MagicMock(spec=httpx.Response)
resp.status_code = status_code
resp.headers = headers or {}
body_str = _json.dumps(json_body) if json_body is not None else ""
resp.content = body_str.encode("utf-8")
if json_body is not None:
resp.json.return_value = json_body
else:
resp.json.side_effect = ValueError("no body")
resp.text = body_str
return resp
def _public_addr_patch():
return patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("93.184.216.34", 0))])
def _drain_revoke_upstream_tasks(client: TestClient, timeout: float = 2.0) -> None:
"""Block until all in-flight upstream-revoke tasks complete.
Phase 8 perf-1 made the RFC 7009 AS round-trip a fire-and-forget
task so the user-visible 204 isn't gated on the AS. The tasks were
scheduled on the TestClient's portal loop; we re-enter that loop
via :attr:`TestClient.portal` to await them. Tests that assert
against the upstream POST must call this helper before the
assertion.
"""
from turnstone.core.mcp_oauth import _revoke_upstream_tasks
portal = getattr(client, "portal", None)
if portal is None:
return
async def _drain() -> None:
pending = list(_revoke_upstream_tasks)
if pending:
async with asyncio.timeout(timeout):
await asyncio.gather(*pending, return_exceptions=True)
portal.call(_drain)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
backend = SQLiteBackend(str(tmp_path / "test.db"))
backend.create_user("user-1", "user1", "User One", "hash")
backend.create_user("user-2", "user2", "User Two", "hash")
return backend
@pytest.fixture
def http_client_mock() -> MagicMock:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock()
client.post = AsyncMock()
return client
# ---------------------------------------------------------------------------
# GET /connections
# ---------------------------------------------------------------------------
class TestListConnections:
def test_list_connections_unauthenticated_401(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
token_store = _make_token_store(storage)
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
authenticated=False,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 401
assert resp.json() == {"error": "Authentication required"}
def test_list_connections_no_token_store_503(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
app = _build_app(storage=storage, http_client=http_client_mock, token_store=None)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 503
def test_list_connections_empty_user_returns_empty_list(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
token_store = _make_token_store(storage)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 200
assert resp.json() == {"connections": []}
def test_list_connections_returns_users_consents(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage, name="srv-a", server_id="srv-id-a")
_seed_oauth_user_server(storage, name="srv-b", server_id="srv-id-b")
token_store = _make_token_store(storage)
_seed_user_token(token_store, server_name="srv-a")
_seed_user_token(token_store, server_name="srv-b")
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
assert resp.status_code == 200
body = resp.json()
assert "connections" in body
servers = sorted(row["server_name"] for row in body["connections"])
assert servers == ["srv-a", "srv-b"]
def test_list_connections_isolates_by_user(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, user_id="user-1", server_name="srv-oauth")
_seed_user_token(token_store, user_id="user-2", server_name="srv-oauth")
# User-1 sees only user-1's row.
app = _build_app(
storage=storage, http_client=http_client_mock, token_store=token_store, user_id="user-1"
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
rows = resp.json()["connections"]
assert all(row["user_id"] == "user-1" for row in rows)
assert len(rows) == 1
# User-2 sees only user-2's row.
app2 = _build_app(
storage=storage, http_client=http_client_mock, token_store=token_store, user_id="user-2"
)
client2 = TestClient(app2, raise_server_exceptions=False)
resp2 = client2.get("/v1/api/mcp/oauth/connections")
rows2 = resp2.json()["connections"]
assert all(row["user_id"] == "user-2" for row in rows2)
assert len(rows2) == 1
def test_list_connections_does_not_leak_secret_fields(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.get("/v1/api/mcp/oauth/connections")
rows = resp.json()["connections"]
assert rows
for row in rows:
for forbidden in (
"access_token",
"refresh_token",
"access_token_ct",
"refresh_token_ct",
):
assert forbidden not in row, f"secret field {forbidden!r} leaked in {row!r}"
# ---------------------------------------------------------------------------
# DELETE /connections/{server_name}
# ---------------------------------------------------------------------------
class TestRevokeConnection:
def test_revoke_connection_unauthenticated_401(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store)
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
authenticated=False,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 401
def test_revoke_connection_missing_row_404(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
token_store = _make_token_store(storage)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-nonexistent")
assert resp.status_code == 404
assert resp.json() == {"error": "No such connection"}
def test_revoke_connection_local_delete_succeeds_204(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
# No refresh token → upstream revoke is skipped entirely.
_seed_user_token(token_store, refresh_token=None)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row is gone.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# Upstream not contacted.
http_client_mock.post.assert_not_called()
def test_revoke_connection_with_revocation_endpoint_calls_upstream(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
http_client_mock.post.return_value = _mk_response(200)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
# ``with TestClient(...)`` keeps a persistent portal so the
# fire-and-forget upstream-revoke task isn't cancelled when
# the request handler returns. See ``_drain_revoke_upstream_tasks``.
# The SSRF-validator's ``socket.getaddrinfo`` patch must wrap
# the drain too — the discovery call now runs on the background
# task and resolves the AS hostname after the request returns.
with (
TestClient(app, raise_server_exceptions=False) as client,
_public_addr_patch(),
):
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row is gone.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# The upstream RFC 7009 POST is fire-and-forget post-Phase-8 perf-1
# so the test must drain the in-flight task set before asserting.
_drain_revoke_upstream_tasks(client)
# Upstream POSTed to revocation_endpoint with refresh-token grant.
assert http_client_mock.post.await_count == 1
call = http_client_mock.post.await_args
assert call.args[0] == "https://as.example.com/revoke"
data = call.kwargs.get("data") or {}
assert data.get("token") == "refresh-secret"
assert data.get("token_type_hint") == "refresh_token"
assert data.get("client_id") == "client-abc"
def test_revoke_connection_without_revocation_endpoint_skips_upstream(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(
200, _good_as_metadata_doc(revocation_endpoint=None)
)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
# ``with TestClient(...)`` keeps the portal alive for the
# background task drain.
with (
TestClient(app, raise_server_exceptions=False) as client,
_public_addr_patch(),
):
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row gone, upstream POST never made.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# Drain the fire-and-forget discovery task before asserting on
# the AS POST — the task runs ``discover_authorization_server``
# but does NOT proceed to POST because revocation_endpoint is
# absent.
_drain_revoke_upstream_tasks(client)
http_client_mock.post.assert_not_called()
def test_revoke_connection_upstream_failure_still_204(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
# AS returns 500 — local delete must still succeed.
http_client_mock.post.return_value = _mk_response(500)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
with _public_addr_patch():
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
assert token_store.get_user_token("user-1", "srv-oauth") is None
def test_revoke_connection_audit_event_emitted_with_user_revoked_reason(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token=None)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Audit row was written via the storage API (tests don't poke at
# the SQLite schema directly — the table name is an internal
# detail).
events = storage.list_audit_events(action="mcp_server.oauth.token_revoked")
assert len(events) == 1
ev = events[0]
assert ev["user_id"] == "user-1"
# resource_id is the immutable server_id PK, not the name.
assert ev["resource_id"] == "srv-id-1"
import json as _json
detail = _json.loads(ev["detail"]) if isinstance(ev["detail"], str) else ev["detail"]
assert detail["reason"] == "user_revoked"
assert detail["upstream_revoke_outcome"] == "no_refresh_token"
assert detail["server_name"] == "srv-oauth"
def test_revoke_connection_cross_user_attempt_404(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
# Owned by user-2, not user-1.
_seed_user_token(token_store, user_id="user-2", server_name="srv-oauth")
app = _build_app(
storage=storage, http_client=http_client_mock, token_store=token_store, user_id="user-1"
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
# Cross-user attempt MUST surface as a generic 404, byte-identical
# body to the never-existed case (no tenant existence leak).
assert resp.status_code == 404
assert resp.json() == {"error": "No such connection"}
# Drain pending tasks defensively, then confirm the upstream
# endpoint was NEVER contacted on the 404-cross-user path. A
# bug that scheduled the AS round-trip before the cross-user
# check would leak existence via the AS-side 200/4xx response.
_drain_revoke_upstream_tasks(client)
http_client_mock.post.assert_not_called()
# User-2's row is untouched.
assert token_store.get_user_token("user-2", "srv-oauth") is not None
def test_revoke_connection_evicts_pool_session(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token=None)
mcp_client_mock = MagicMock()
# ``evict_user_session`` is the public sync surface on
# MCPClientManager; mirror its signature here so the handler's
# ``hasattr`` gate triggers.
mcp_client_mock.evict_user_session = MagicMock(return_value=None)
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
mcp_client=mcp_client_mock,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
mcp_client_mock.evict_user_session.assert_called_once_with("user-1", "srv-oauth")
def test_revoke_connection_pool_eviction_failure_does_not_block_204(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token=None)
mcp_client_mock = MagicMock()
mcp_client_mock.evict_user_session = MagicMock(side_effect=RuntimeError("loop closed"))
app = _build_app(
storage=storage,
http_client=http_client_mock,
token_store=token_store,
mcp_client=mcp_client_mock,
)
client = TestClient(app, raise_server_exceptions=False)
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local delete still happened.
assert token_store.get_user_token("user-1", "srv-oauth") is None
def test_revoke_connection_204_not_gated_on_slow_upstream(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
"""The user-visible 204 must return promptly even when the
upstream AS round-trip is slow / hanging. Pre-perf-1 the
handler awaited ``revoke_token_at_as`` synchronously, so a
stuck AS could block the user's revoke confirmation. The
fire-and-forget refactor moves the call onto a background task
so the 204 returns in well under 1s regardless of AS latency.
Bound is conservative for CI runner jitter.
"""
import time
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
async def _slow_post(*_args: Any, **_kwargs: Any) -> Any:
# Simulate a slow / unreachable AS — must NOT gate the
# user-visible 204 on this round-trip.
await asyncio.sleep(5.0)
return _mk_response(200)
http_client_mock.post = AsyncMock(side_effect=_slow_post)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
with _public_addr_patch():
start = time.monotonic()
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
elapsed = time.monotonic() - start
assert resp.status_code == 204
# 1s ceiling — the 204 must return on the local-delete path
# without waiting on the AS POST (which sleeps 5s above). Bound
# is intentionally generous for CI runner jitter; the actual
# path is on the order of milliseconds.
assert elapsed < 1.0, (
f"204 returned in {elapsed:.3f}s — should be <1s; the "
"fire-and-forget upstream revoke isn't decoupled from the "
"response."
)
# The local row IS gone — the authoritative delete ran before
# the 204 returned, even though the AS round-trip is still
# in flight.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# Cancel any in-flight tasks so the test client can exit cleanly.
from turnstone.core.mcp_oauth import _revoke_upstream_tasks
portal = getattr(client, "portal", None)
if portal is not None:
for task in list(_revoke_upstream_tasks):
portal.call(task.cancel)
def test_revoke_connection_sheds_upstream_when_task_set_full(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
"""Round-2 q-2 regression: the soft cap on ``_revoke_upstream_tasks``
is the only protection against unbounded background-task pile-up
under a coordinated mass-revoke. When the set is full, the local
delete still runs but no upstream task is scheduled; the audit
detail records ``upstream_revoke_outcome="shed_by_cap"`` and
the AS endpoint is never contacted.
"""
from turnstone.core.mcp_oauth import (
_REVOKE_UPSTREAM_TASKS_MAX,
_revoke_upstream_tasks,
)
_seed_oauth_user_server(storage)
token_store = _make_token_store(storage)
_seed_user_token(token_store, refresh_token="refresh-secret")
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
sentinel_event_holder: dict[str, asyncio.Event] = {}
# Use ``with TestClient(...)`` so the portal stays alive — we
# need to schedule sentinel tasks on the portal's loop and the
# tasks must outlive the request to actually fill the set.
with (
TestClient(app, raise_server_exceptions=False) as client,
_public_addr_patch(),
):
portal = client.portal
assert portal is not None
async def _create_sentinel_event() -> asyncio.Event:
event = asyncio.Event()
sentinel_event_holder["event"] = event
return event
sentinel_event = portal.call(_create_sentinel_event)
async def _wait_on_event() -> None:
await sentinel_event.wait()
async def _fill_task_set() -> list[asyncio.Task[None]]:
tasks: list[asyncio.Task[None]] = []
for _ in range(_REVOKE_UPSTREAM_TASKS_MAX):
t = asyncio.create_task(_wait_on_event())
_revoke_upstream_tasks.add(t)
tasks.append(t)
return tasks
sentinels = portal.call(_fill_task_set)
assert len(_revoke_upstream_tasks) >= _REVOKE_UPSTREAM_TASKS_MAX
try:
resp = client.delete("/v1/api/mcp/oauth/connections/srv-oauth")
assert resp.status_code == 204
# Local row is still gone — authoritative delete ran.
assert token_store.get_user_token("user-1", "srv-oauth") is None
# AS endpoint MUST NOT have been contacted.
http_client_mock.post.assert_not_called()
# Audit detail records the categorical shed outcome.
events = storage.list_audit_events(action="mcp_server.oauth.token_revoked")
assert len(events) == 1
detail = events[0]["detail"]
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail["upstream_revoke_outcome"] == "shed_by_cap"
finally:
# Release sentinels so the portal can shut down cleanly.
async def _release() -> None:
sentinel_event.set()
for t in sentinels:
t.cancel()
await asyncio.gather(*sentinels, return_exceptions=True)
portal.call(_release)
# ---------------------------------------------------------------------------
# evict_user_session helper sanity checks
# ---------------------------------------------------------------------------
class TestEvictUserSession:
def test_evict_user_session_no_loop_is_silent_noop(self) -> None:
from turnstone.core.mcp_client import MCPClientManager
mgr = MCPClientManager.__new__(MCPClientManager)
mgr._loop = None # type: ignore[attr-defined]
# Must not raise.
mgr.evict_user_session("user-1", "srv-oauth")
def test_evict_user_session_dispatches_to_loop(self) -> None:
from turnstone.core.mcp_client import MCPClientManager
mgr = MCPClientManager.__new__(MCPClientManager)
loop = asyncio.new_event_loop()
try:
mgr._loop = loop # type: ignore[attr-defined]
mgr._user_pool_entries = {} # type: ignore[attr-defined]
mgr._last_pool_notification_refresh = {} # type: ignore[attr-defined]
evicted: list[tuple[str, str]] = []
def _fake_evict(key: tuple[str, str]) -> None:
evicted.append(key)
mgr._evict_session = _fake_evict # type: ignore[method-assign]
# Run the dispatch on a separate thread so the loop can drain.
import threading
done = threading.Event()
def _run_loop() -> None:
loop.call_later(0.05, loop.stop)
loop.run_forever()
done.set()
t = threading.Thread(target=_run_loop, daemon=True)
t.start()
mgr.evict_user_session("user-1", "srv-oauth")
done.wait(timeout=1.0)
assert evicted == [("user-1", "srv-oauth")]
finally:
if not loop.is_closed():
loop.close()
+626
View File
@@ -0,0 +1,626 @@
"""Discovery tests for the per-(user, server) MCP OAuth flow.
Covers PRM (RFC 9728) and AS metadata (RFC 8414) discovery, including:
- override URL takes precedence
- PRM happy path: server URL -> .well-known/oauth-protected-resource
-> ``authorization_servers[0]``
- PRM 401 + ``WWW-Authenticate: Bearer resource_metadata="..."`` follows
the URL.
- AS metadata without S256 -> :class:`MCPOAuthDiscoveryError`.
- SSRF rejection on AS issuer URL.
- In-memory cache hit/miss + persistent cache write to
``mcp_servers.oauth_as_issuer_cached``.
"""
from __future__ import annotations
import asyncio
import time
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
from turnstone.core.mcp_oauth import (
ASMetadata,
MCPOAuthDiscoveryError,
_parse_prm_url_from_www_authenticate,
discover_authorization_server,
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _mk_response(
status_code: int = 200,
json_body: Any = None,
headers: dict[str, str] | None = None,
) -> MagicMock:
"""Build a MagicMock that quacks like ``httpx.Response``."""
resp = MagicMock(spec=httpx.Response)
resp.status_code = status_code
resp.headers = headers or {}
resp.content = (str(json_body) if json_body is not None else "").encode("utf-8")
if json_body is not None:
resp.json.return_value = json_body
else:
resp.json.side_effect = ValueError("no body")
resp.text = str(json_body) if json_body is not None else ""
return resp
def _good_as_metadata_doc() -> dict[str, Any]:
return {
"issuer": "https://as.example.com",
"authorization_endpoint": "https://as.example.com/authorize",
"token_endpoint": "https://as.example.com/token",
"jwks_uri": "https://as.example.com/jwks",
"code_challenge_methods_supported": ["S256"],
"token_endpoint_auth_methods_supported": ["none", "client_secret_basic"],
"registration_endpoint": "https://as.example.com/register",
}
def _public_addr_patch():
return patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("93.184.216.34", 0))])
def _mk_storage_mock(server_id: str = "srv-id") -> MagicMock:
storage = MagicMock()
storage.update_mcp_server.return_value = True
return storage
# ---------------------------------------------------------------------------
# PRM parser
# ---------------------------------------------------------------------------
class TestParsePRMUrl:
def test_extracts_resource_metadata_url(self) -> None:
header = (
'Bearer error="invalid_token", '
'resource_metadata="https://srv.example.com/.well-known/oauth-protected-resource"'
)
url = _parse_prm_url_from_www_authenticate(header)
assert url == "https://srv.example.com/.well-known/oauth-protected-resource"
def test_returns_none_when_absent(self) -> None:
assert _parse_prm_url_from_www_authenticate('Bearer realm="x"') is None
def test_handles_empty_header(self) -> None:
assert _parse_prm_url_from_www_authenticate("") is None
def test_handles_escaped_quote_in_value(self) -> None:
"""RFC 7230 quoted-string allows ``\\"`` — naive ``[^"]+`` truncates.
A malicious or buggy resource server could send an embedded
escaped quote; the parser must yield the unescaped value, not
the prefix up to the escaped quote.
"""
header = 'Bearer resource_metadata="https://srv.example.com/with\\"quote"'
url = _parse_prm_url_from_www_authenticate(header)
assert url == 'https://srv.example.com/with"quote'
def test_handles_escaped_backslash(self) -> None:
header = 'Bearer resource_metadata="https://srv.example.com/back\\\\slash"'
url = _parse_prm_url_from_www_authenticate(header)
assert url == "https://srv.example.com/back\\slash"
def test_unterminated_quoted_string_returns_none(self) -> None:
# Closing quote missing — naive regex would still match, but
# the proper parser should reject malformed input.
header = 'Bearer resource_metadata="https://srv.example.com/no-close'
assert _parse_prm_url_from_www_authenticate(header) is None
# ---------------------------------------------------------------------------
# discover_authorization_server happy paths
# ---------------------------------------------------------------------------
class TestDiscoveryOverride:
def test_override_url_skips_prm(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert isinstance(meta, ASMetadata)
assert meta.token_endpoint == "https://as.example.com/token"
# Only the AS metadata URL was hit, not PRM.
called_urls = [c.args[0] for c in client.get.call_args_list]
assert all("oauth-authorization-server" in u for u in called_urls)
class TestDiscoveryPRM:
def test_prm_happy_path(self) -> None:
async def _get(url, *args, **kwargs):
if url.endswith("/oauth-protected-resource"):
return _mk_response(
200,
{
"resource": "https://mcp.example.com",
"authorization_servers": ["https://as.example.com"],
},
)
if url.endswith("/oauth-authorization-server"):
return _mk_response(200, _good_as_metadata_doc())
raise AssertionError(f"unexpected URL: {url}")
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.issuer == "https://as.example.com"
def test_prm_401_follows_www_authenticate(self) -> None:
async def _get(url, *args, **kwargs):
if url == "https://mcp.example.com/.well-known/oauth-protected-resource":
return _mk_response(
401,
headers={
"www-authenticate": (
'Bearer error="invalid_token", '
"resource_metadata="
'"https://meta.example.com/prm"'
)
},
json_body=None,
)
if url == "https://meta.example.com/prm":
return _mk_response(
200,
{
"authorization_servers": ["https://as.example.com"],
},
)
if url.endswith("/oauth-authorization-server"):
return _mk_response(200, _good_as_metadata_doc())
raise AssertionError(f"unexpected URL: {url}")
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.token_endpoint == "https://as.example.com/token"
def test_prm_401_without_resource_metadata_raises(self) -> None:
async def _get(url, *args, **kwargs):
return _mk_response(401, headers={"www-authenticate": "Basic realm=x"})
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="resource_metadata"):
asyncio.run(_run())
# ---------------------------------------------------------------------------
# AS metadata validation
# ---------------------------------------------------------------------------
class TestASMetadataValidation:
def test_no_s256_raises(self) -> None:
doc = _good_as_metadata_doc()
doc["code_challenge_methods_supported"] = ["plain"]
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="S256"):
asyncio.run(_run())
def test_missing_endpoints_raises(self) -> None:
doc = _good_as_metadata_doc()
del doc["token_endpoint"]
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="missing required"):
asyncio.run(_run())
def test_third_party_endpoint_rejected(self) -> None:
doc = _good_as_metadata_doc()
doc["token_endpoint"] = "https://attacker.example.com/token"
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="token_endpoint"):
asyncio.run(_run())
def test_ssrf_on_override_rejected(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock()
storage = _mk_storage_mock()
async def _run():
# Resolve to private 10.x — SSRF guard fires before any HTTP call.
with patch(
"socket.getaddrinfo",
return_value=[(2, 1, 6, "", ("10.0.0.1", 0))],
):
await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://internal.corp.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError):
asyncio.run(_run())
client.get.assert_not_called()
# ---------------------------------------------------------------------------
# Caching
# ---------------------------------------------------------------------------
class TestMetadataCache:
def test_cache_miss_then_hit(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
cache: dict[str, tuple[ASMetadata, float]] = {}
async def _run():
with _public_addr_patch():
first = await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
metadata_cache=cache,
)
second = await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer="https://as.example.com",
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
metadata_cache=cache,
)
return first, second
first, second = asyncio.run(_run())
assert first.token_endpoint == second.token_endpoint
# First call hit AS metadata; second call hit the cache.
assert client.get.call_count == 1
def test_cache_expiry_refetches(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
# Pre-populate cache with a very stale entry.
stale_meta = ASMetadata(
issuer="https://as.example.com",
authorization_endpoint="https://as.example.com/authorize",
token_endpoint="https://as.example.com/token",
registration_endpoint=None,
revocation_endpoint=None,
jwks_uri=None,
code_challenge_methods_supported=("S256",),
token_endpoint_auth_methods_supported=(),
)
cache = {"https://as.example.com": (stale_meta, time.monotonic() - 10**6)}
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
metadata_cache=cache,
)
meta = asyncio.run(_run())
# Stale entry was bypassed -> we hit the network.
assert client.get.call_count == 1
assert meta.token_endpoint == "https://as.example.com/token"
def test_persistent_cache_write_on_first_resolution(self) -> None:
async def _get(url, *args, **kwargs):
if url.endswith("/oauth-protected-resource"):
return _mk_response(200, {"authorization_servers": ["https://as.example.com"]})
return _mk_response(200, _good_as_metadata_doc())
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
asyncio.run(_run())
# update_mcp_server was called once with the cached issuer.
storage.update_mcp_server.assert_called_once_with(
"srv-id", oauth_as_issuer_cached="https://as.example.com"
)
def test_persistent_cache_skip_when_already_cached(self) -> None:
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, _good_as_metadata_doc()))
storage = _mk_storage_mock()
async def _run():
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer="https://as.example.com",
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
asyncio.run(_run())
storage.update_mcp_server.assert_not_called()
# ---------------------------------------------------------------------------
# sec-3 — cached_issuer re-validated on read
# ---------------------------------------------------------------------------
class TestCachedIssuerSSRFRevalidation:
"""A cached issuer URL must still pass SSRF validation on every read.
Defense-in-depth: an admin who points ``oauth_as_issuer_cached`` at a
private address (or a hostname that has rebound to one) should not
bypass the guard just because the value was already in the row.
"""
def test_cached_issuer_rejected_clears_row_and_falls_through_to_prm(self) -> None:
async def _get(url: str, *args: Any, **kwargs: Any) -> MagicMock:
if url.endswith("/oauth-protected-resource"):
return _mk_response(200, {"authorization_servers": ["https://as.example.com"]})
if url.endswith("/oauth-authorization-server"):
return _mk_response(200, _good_as_metadata_doc())
raise AssertionError(f"unexpected URL: {url}")
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(side_effect=_get)
storage = _mk_storage_mock()
# cached_issuer points at a private host. SSRF guard fires on
# the cached value first, the row is cleared, and PRM
# discovery runs as a fallback.
async def _run() -> Any:
with patch(
"socket.getaddrinfo",
# Private resolution for "internal.corp", public for everything else.
side_effect=lambda host, *a, **kw: [
(2, 1, 6, "", ("10.0.0.1" if "internal" in host else "93.184.216.34", 0))
],
):
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url=None,
cached_issuer="https://internal.corp.example.com",
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.token_endpoint == "https://as.example.com/token"
# The bad cached_issuer was cleared from the row.
clear_calls = [
c
for c in storage.update_mcp_server.call_args_list
if c.kwargs.get("oauth_as_issuer_cached") is None
]
assert clear_calls, "cached_issuer should have been cleared"
# ---------------------------------------------------------------------------
# revocation_endpoint parsing (RFC 8414)
# ---------------------------------------------------------------------------
class TestASMetadataRevocationEndpoint:
def test_as_metadata_parses_revocation_endpoint(self) -> None:
doc = _good_as_metadata_doc()
doc["revocation_endpoint"] = "https://as.example.com/revoke"
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run() -> ASMetadata:
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.revocation_endpoint == "https://as.example.com/revoke"
def test_as_metadata_revocation_endpoint_absent(self) -> None:
doc = _good_as_metadata_doc()
doc.pop("revocation_endpoint", None)
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run() -> ASMetadata:
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
meta = asyncio.run(_run())
assert meta.revocation_endpoint is None
def test_as_metadata_revocation_endpoint_rejected_when_cross_origin(self) -> None:
doc = _good_as_metadata_doc()
doc["revocation_endpoint"] = "https://attacker.example.com/revoke"
client = MagicMock(spec=httpx.AsyncClient)
client.get = AsyncMock(return_value=_mk_response(200, doc))
storage = _mk_storage_mock()
async def _run() -> ASMetadata:
with _public_addr_patch():
return await discover_authorization_server(
server_name="srv-x",
server_url="https://mcp.example.com/sse",
override_url="https://as.example.com",
cached_issuer=None,
http_client=client,
storage=storage,
server_id="srv-id",
trusted_hosts=frozenset(),
)
with pytest.raises(MCPOAuthDiscoveryError, match="revocation_endpoint"):
asyncio.run(_run())
File diff suppressed because it is too large Load Diff
+133
View File
@@ -0,0 +1,133 @@
"""Smoke tests for the new OAuth-MCP storage tables.
Phase 2 only adds the schema token CRUD lands in Phase 3 and pending-
state CRUD in Phase 4. These tests verify the tables exist after
``init_storage`` and accept the documented row shape via raw SQL.
"""
from __future__ import annotations
import sqlalchemy as sa
from turnstone.core.storage._schema import mcp_oauth_pending, mcp_user_tokens
class TestMcpUserTokensTable:
def test_table_exists_and_accepts_row(self, backend) -> None:
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_user_tokens),
{
"user_id": "u1",
"server_name": "srv-a",
"access_token_ct": b"\x00ciphertext-a",
"refresh_token_ct": b"\x00ciphertext-r",
"expires_at": "2026-05-04T12:00:00",
"scopes": "openid profile",
"as_issuer": "https://auth.example.com",
"audience": "https://mcp.example.com",
"created": "2026-05-04T11:00:00",
"last_refreshed": None,
},
)
conn.commit()
row = conn.execute(
sa.select(mcp_user_tokens).where(
(mcp_user_tokens.c.user_id == "u1") & (mcp_user_tokens.c.server_name == "srv-a")
)
).one()
assert row.access_token_ct == b"\x00ciphertext-a"
assert row.refresh_token_ct == b"\x00ciphertext-r"
assert row.scopes == "openid profile"
assert row.audience == "https://mcp.example.com"
def test_composite_pk_distinguishes_user_server(self, backend) -> None:
"""Same user, different server => two rows; same (user, server) => conflict."""
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_user_tokens),
[
{
"user_id": "u1",
"server_name": "srv-a",
"access_token_ct": b"a",
"refresh_token_ct": None,
"expires_at": None,
"scopes": None,
"as_issuer": "https://auth.example.com",
"audience": "https://a.example.com",
"created": "2026-05-04T11:00:00",
"last_refreshed": None,
},
{
"user_id": "u1",
"server_name": "srv-b",
"access_token_ct": b"b",
"refresh_token_ct": None,
"expires_at": None,
"scopes": None,
"as_issuer": "https://auth.example.com",
"audience": "https://b.example.com",
"created": "2026-05-04T11:00:00",
"last_refreshed": None,
},
],
)
conn.commit()
count = conn.execute(sa.select(sa.func.count()).select_from(mcp_user_tokens)).scalar()
assert count == 2
class TestMcpOauthPendingTable:
def test_table_exists_and_accepts_row(self, backend) -> None:
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_oauth_pending),
{
"state": "rand-state-xyz",
"user_id": "u1",
"server_name": "srv-a",
"code_verifier": "verifier-blob",
"return_url": "/admin/mcp-servers",
"created_at": "2026-05-04T11:00:00",
},
)
conn.commit()
row = conn.execute(
sa.select(mcp_oauth_pending).where(mcp_oauth_pending.c.state == "rand-state-xyz")
).one()
assert row.user_id == "u1"
assert row.server_name == "srv-a"
assert row.return_url == "/admin/mcp-servers"
def test_state_pk_unique(self, backend) -> None:
"""A second insert with the same state value raises IntegrityError."""
with backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_oauth_pending),
{
"state": "dup-state",
"user_id": "u1",
"server_name": "srv-a",
"code_verifier": "v",
"return_url": "/x",
"created_at": "2026-05-04T11:00:00",
},
)
conn.commit()
import pytest
from sqlalchemy.exc import IntegrityError
with pytest.raises(IntegrityError), backend._engine.connect() as conn:
conn.execute(
sa.insert(mcp_oauth_pending),
{
"state": "dup-state",
"user_id": "u2",
"server_name": "srv-b",
"code_verifier": "v",
"return_url": "/y",
"created_at": "2026-05-04T11:01:00",
},
)
conn.commit()
+53
View File
@@ -0,0 +1,53 @@
"""PKCE pair-generation tests for the MCP OAuth flow.
Verifies the contract documented in RFC 7636 §4.1 and §4.2:
- ``code_verifier`` is a high-entropy 43..128 character urlsafe-base64 string.
- ``code_challenge`` is the BASE64URL-NO-PADDING encoding of
``SHA256(verifier)``.
"""
from __future__ import annotations
import base64
import hashlib
import string
from turnstone.core.mcp_oauth import generate_pkce_pair
_URLSAFE_CHARS = set(string.ascii_letters + string.digits + "-_")
class TestGeneratePkcePair:
def test_returns_tuple_of_strings(self) -> None:
verifier, challenge = generate_pkce_pair()
assert isinstance(verifier, str)
assert isinstance(challenge, str)
def test_verifier_length_in_rfc_range(self) -> None:
for _ in range(20):
verifier, _ = generate_pkce_pair()
assert 43 <= len(verifier) <= 128
def test_verifier_is_urlsafe(self) -> None:
for _ in range(20):
verifier, _ = generate_pkce_pair()
assert all(ch in _URLSAFE_CHARS for ch in verifier)
def test_challenge_matches_sha256_of_verifier(self) -> None:
for _ in range(20):
verifier, challenge = generate_pkce_pair()
digest = hashlib.sha256(verifier.encode("ascii")).digest()
expected = base64.urlsafe_b64encode(digest).rstrip(b"=").decode("ascii")
assert challenge == expected
def test_challenge_has_no_padding(self) -> None:
for _ in range(20):
_, challenge = generate_pkce_pair()
assert "=" not in challenge
def test_pairs_are_unique(self) -> None:
pairs = {generate_pkce_pair() for _ in range(50)}
# 50 random draws shouldn't collide; if they do we have a much
# bigger problem than this assertion.
assert len(pairs) == 50
File diff suppressed because it is too large Load Diff
+399
View File
@@ -0,0 +1,399 @@
"""Tests for :func:`turnstone.core.mcp_oauth.revoke_token_at_as`.
The helper is best-effort RFC 7009 token revocation. It must:
- skip cleanly when the AS metadata doesn't advertise a revocation endpoint
- POST the form body when one is present (with optional client_secret)
- never raise on non-2xx, network errors, or timeouts caller doesn't
want try/except in cleanup paths
- never use ``exc_info=True`` chained ``__context__`` may carry an
``httpx.Request`` whose ``Authorization`` header holds a bearer; the
bearer-leak invariant requires structured fields with type names only
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
from turnstone.core.mcp_oauth import (
ASMetadata,
MCPOAuthDiscoveryError,
_attempt_upstream_revoke,
revoke_token_at_as,
)
def _make_as_metadata(
*,
revocation_endpoint: str | None = "https://as.example.com/revoke",
) -> ASMetadata:
return ASMetadata(
issuer="https://as.example.com",
authorization_endpoint="https://as.example.com/authorize",
token_endpoint="https://as.example.com/token",
registration_endpoint=None,
revocation_endpoint=revocation_endpoint,
jwks_uri=None,
code_challenge_methods_supported=("S256",),
token_endpoint_auth_methods_supported=("client_secret_basic",),
)
def _mk_response(status_code: int) -> MagicMock:
resp = MagicMock(spec=httpx.Response)
resp.status_code = status_code
return resp
class TestRevocationUnsupported:
def test_revoke_token_skipped_when_revocation_endpoint_none(self) -> None:
as_meta = _make_as_metadata(revocation_endpoint=None)
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock()
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
client.post.assert_not_called()
info_events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_unsupported" in info_events
class TestRevocationSuccess:
def test_revoke_token_succeeds_on_200(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(200))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret="s-secret",
)
)
# POST shape — URL + form body keys.
client.post.assert_awaited_once()
call_args = client.post.call_args
assert call_args.args[0] == "https://as.example.com/revoke"
body = call_args.kwargs["data"]
assert body == {
"token": "r-secret",
"token_type_hint": "refresh_token",
"client_id": "client-1",
"client_secret": "s-secret",
}
info_events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_succeeded" in info_events
def test_revoke_token_omits_client_secret_when_none(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(200))
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
body = client.post.call_args.kwargs["data"]
assert "client_secret" not in body
assert body["token"] == "r-secret"
assert body["token_type_hint"] == "refresh_token"
assert body["client_id"] == "client-1"
def test_revoke_token_succeeds_on_204(self) -> None:
# RFC 7009 says the AS MAY return any 2xx; treat the whole range
# as success.
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(204))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
info_events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_succeeded" in info_events
class TestRevocationFailureLogged:
def _run_and_capture(self, status: int) -> list[Any]:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(return_value=_mk_response(status))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
return mock_log.info.call_args_list
def test_revoke_token_logs_on_400_does_not_raise(self) -> None:
calls = self._run_and_capture(400)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
# Must include status field.
failed_call = next(c for c in calls if c.args[0] == "mcp_server.oauth.revocation_failed")
assert failed_call.kwargs.get("status") == 400
def test_revoke_token_logs_on_401_does_not_raise(self) -> None:
calls = self._run_and_capture(401)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(c for c in calls if c.args[0] == "mcp_server.oauth.revocation_failed")
assert failed_call.kwargs.get("status") == 401
def test_revoke_token_logs_on_403_does_not_raise(self) -> None:
calls = self._run_and_capture(403)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
def test_revoke_token_logs_on_5xx_does_not_raise(self) -> None:
calls = self._run_and_capture(500)
events = [c.args[0] for c in calls]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(c for c in calls if c.args[0] == "mcp_server.oauth.revocation_failed")
assert failed_call.kwargs.get("status") == 500
class TestRevocationExceptionPaths:
def test_revoke_token_handles_network_error(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(side_effect=httpx.ConnectError("boom"))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(
c
for c in mock_log.info.call_args_list
if c.args[0] == "mcp_server.oauth.revocation_failed"
)
assert failed_call.kwargs.get("error") == "ConnectError"
def test_revoke_token_handles_httpx_timeout(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(side_effect=httpx.TimeoutException("slow"))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(
c
for c in mock_log.info.call_args_list
if c.args[0] == "mcp_server.oauth.revocation_failed"
)
assert failed_call.kwargs.get("error") == "TimeoutException"
def test_revoke_token_handles_asyncio_timeout(self) -> None:
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
async def _slow(*_args: Any, **_kwargs: Any) -> Any:
await asyncio.sleep(10.0)
raise AssertionError("should have timed out")
client.post = AsyncMock(side_effect=_slow)
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
timeout_seconds=0.05,
)
)
events = [c.args[0] for c in mock_log.info.call_args_list]
assert "mcp_server.oauth.revocation_failed" in events
failed_call = next(
c
for c in mock_log.info.call_args_list
if c.args[0] == "mcp_server.oauth.revocation_failed"
)
# ``asyncio.timeout`` raises ``TimeoutError`` (Python's builtin)
# on cancellation.
assert failed_call.kwargs.get("error") == "TimeoutError"
def test_revoke_token_no_exc_info_in_logs(self) -> None:
"""Bearer-leak invariant: the revoke path must NEVER set
``exc_info=True``. Chained ``__context__`` may include an
``httpx.Request`` whose ``Authorization`` header holds a
bearer; the traceback formatter would render it.
"""
as_meta = _make_as_metadata()
client = MagicMock(spec=httpx.AsyncClient)
client.post = AsyncMock(side_effect=httpx.ConnectError("boom"))
with patch("turnstone.core.mcp_oauth.log") as mock_log:
asyncio.run(
revoke_token_at_as(
as_metadata=as_meta,
http_client=client,
refresh_token="r-secret",
client_id="client-1",
client_secret=None,
)
)
# No info call may carry exc_info.
for call in mock_log.info.call_args_list:
assert "exc_info" not in call.kwargs, (
f"mcp_server.oauth log info({call.args[0]!r}) used exc_info — "
"this violates the bearer-leak invariant"
)
# Defensively: also check warning + exception levels for the
# same call site.
for call in mock_log.warning.call_args_list:
assert "exc_info" not in call.kwargs
mock_log.exception.assert_not_called()
class TestAttemptUpstreamRevokeNeverRaises:
"""Round-2 q-3 regression: ``_attempt_upstream_revoke``'s docstring
claims ``Never raises``. Background-task semantics make this load-
bearing a propagated exception logs ``Task exception was never
retrieved`` because the ``set.discard`` done-callback doesn't read
``task.exception()``.
The wrapper's narrow inner ``except`` clauses (``MCPOAuthDiscoveryError``,
``MCPTokenDecryptError``) leave room for any other exception type
raised by ``discover_authorization_server`` /
``storage.get_mcp_oauth_client_secret_ct`` / ``token_store.cipher.decrypt``
to escape. The outer ``try/except Exception`` is what keeps the
contract honest. These tests pin that gate.
"""
def _build_args(self) -> dict[str, Any]:
token_store = MagicMock()
token_store.cipher = MagicMock()
token_store.cipher.decrypt.return_value = b"shh"
storage = MagicMock()
storage.get_mcp_oauth_client_secret_ct.return_value = None
return {
"http_client": MagicMock(spec=httpx.AsyncClient),
"metadata_cache": None,
"storage": storage,
"token_store": token_store,
"server_name": "srv-oauth",
"server_row": {
"url": "https://mcp.example.com",
"oauth_client_id": "client-1",
"oauth_authorization_server_url": None,
"oauth_as_issuer_cached": None,
},
"server_id_for_audit": "srv-id-1",
"refresh_token": "r-secret",
}
def test_attempt_upstream_revoke_swallows_unexpected_exception(self) -> None:
"""A generic exception from a path the inner handlers don't
cover MUST be caught at the outer boundary and logged with type
name only (no exc_info=True per the bearer-leak invariant).
"""
args = self._build_args()
async def _boom(*_a: Any, **_kw: Any) -> Any:
raise RuntimeError("network blew up")
with (
patch("turnstone.core.mcp_oauth.discover_authorization_server", side_effect=_boom),
patch("turnstone.core.mcp_oauth.log") as mock_log,
):
# MUST NOT raise.
asyncio.run(_attempt_upstream_revoke(**args))
events = [call.args[0] for call in mock_log.info.call_args_list]
assert "mcp_server.oauth.upstream_revoke_failed" in events, (
"outer try/except must log mcp_server.oauth.upstream_revoke_failed "
"with the exception type name when an unexpected exception escapes "
"the narrow inner handlers"
)
for call in mock_log.info.call_args_list:
assert "exc_info" not in call.kwargs, (
"outer-block log must not use exc_info=True — chained "
"__context__ may carry an httpx.Request bearer"
)
def test_attempt_upstream_revoke_logs_discovery_failure(self) -> None:
"""Round-2 bug-1: ``MCPOAuthDiscoveryError`` MUST emit
``upstream_revoke_discovery_failed`` so operators have visibility
into a silent-discovery-failure path that previously logged
nothing while the audit row recorded ``upstream_revoke_outcome=scheduled``.
"""
args = self._build_args()
async def _disc_fail(*_a: Any, **_kw: Any) -> Any:
raise MCPOAuthDiscoveryError("PRM fetch 503")
with (
patch(
"turnstone.core.mcp_oauth.discover_authorization_server",
side_effect=_disc_fail,
),
patch("turnstone.core.mcp_oauth.log") as mock_log,
):
asyncio.run(_attempt_upstream_revoke(**args))
events = [call.args[0] for call in mock_log.info.call_args_list]
assert "mcp_server.oauth.upstream_revoke_discovery_failed" in events
assert "mcp_server.oauth.upstream_revoke_failed" not in events
+212
View File
@@ -0,0 +1,212 @@
"""Storage CRUD tests for the per-(user, server) MCP OAuth pending-state table.
Validates the storage-protocol additions for the per-(user, server)
OAuth flow:
- ``create_mcp_oauth_pending_state``
- ``pop_mcp_oauth_pending_state`` (atomic, with TTL)
- ``cleanup_expired_mcp_oauth_pending_states``
- ``get_mcp_oauth_client_secret_ct``
"""
from __future__ import annotations
import sqlalchemy as sa
class TestCreateAndPop:
def test_round_trip(self, backend) -> None:
backend.create_mcp_oauth_pending_state(
"state-1",
"user-a",
"srv-x",
"verifier-blob",
"/admin/mcp-servers",
)
row = backend.pop_mcp_oauth_pending_state("state-1", max_age_seconds=600)
assert row is not None
assert row["state"] == "state-1"
assert row["user_id"] == "user-a"
assert row["server_name"] == "srv-x"
assert row["code_verifier"] == "verifier-blob"
assert row["return_url"] == "/admin/mcp-servers"
def test_pop_consumes_row(self, backend) -> None:
backend.create_mcp_oauth_pending_state("s2", "u", "s", "v", "/r")
first = backend.pop_mcp_oauth_pending_state("s2")
assert first is not None
# Second pop must miss — row was consumed.
second = backend.pop_mcp_oauth_pending_state("s2")
assert second is None
def test_pop_missing_returns_none(self, backend) -> None:
assert backend.pop_mcp_oauth_pending_state("never-existed") is None
class TestTTL:
def test_pop_rejects_expired_row(self, backend) -> None:
backend.create_mcp_oauth_pending_state("old-state", "u", "s", "v", "/r")
# Backdate it so it's older than the TTL window.
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_oauth_pending SET created_at = '2020-01-01T00:00:00' "
"WHERE state = 'old-state'"
)
)
conn.commit()
# Default TTL is 600s — the row is decades old.
row = backend.pop_mcp_oauth_pending_state("old-state")
assert row is None
# Even though pop returned None, the row must have been wiped — a
# second pop with a giant TTL must still see nothing.
again = backend.pop_mcp_oauth_pending_state("old-state", max_age_seconds=10**9)
assert again is None
def test_pop_accepts_fresh_row(self, backend) -> None:
backend.create_mcp_oauth_pending_state("fresh", "u", "s", "v", "/r")
row = backend.pop_mcp_oauth_pending_state("fresh", max_age_seconds=600)
assert row is not None
assert row["state"] == "fresh"
class TestCleanup:
def test_cleanup_deletes_only_expired(self, backend) -> None:
backend.create_mcp_oauth_pending_state("old", "u", "s", "v", "/r")
backend.create_mcp_oauth_pending_state("new", "u", "s", "v", "/r")
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_oauth_pending SET created_at = '2020-01-01T00:00:00' "
"WHERE state = 'old'"
)
)
conn.commit()
deleted = backend.cleanup_expired_mcp_oauth_pending_states(max_age_seconds=600)
assert deleted == 1
# Old gone, new still around.
assert backend.pop_mcp_oauth_pending_state("old") is None
survivor = backend.pop_mcp_oauth_pending_state("new")
assert survivor is not None
def test_cleanup_no_rows(self, backend) -> None:
assert backend.cleanup_expired_mcp_oauth_pending_states() == 0
class TestGetOAuthClientSecretCt:
def test_returns_none_when_unset(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-id",
name="srv-x",
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
)
assert backend.get_mcp_oauth_client_secret_ct("srv-id") is None
def test_returns_ciphertext_after_set(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-id",
name="srv-x",
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
)
ct = b"\x00\xff\x42encrypted-blob"
ok = backend.set_mcp_oauth_client_secret_ct("srv-id", ct)
assert ok is True
out = backend.get_mcp_oauth_client_secret_ct("srv-id")
assert out == ct
def test_returns_none_for_missing_server(self, backend) -> None:
assert backend.get_mcp_oauth_client_secret_ct("does-not-exist") is None
def _create_user_token_row(
backend,
*,
user_id: str,
server_name: str,
created: str,
) -> None:
"""Insert a token row + backdate ``created`` so ordering is deterministic.
The storage helper stamps ``created`` from ``datetime.now(UTC)``; for
multi-row ordering tests we backdate via raw SQL so the inserts stay
independent of clock resolution.
"""
backend.create_mcp_user_token(
user_id,
server_name,
access_token_ct=b"ct-access",
refresh_token_ct=b"ct-refresh",
expires_at="2026-05-04T12:00:00",
scopes="openid",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_user_tokens SET created = :created "
"WHERE user_id = :uid AND server_name = :sn"
),
{"created": created, "uid": user_id, "sn": server_name},
)
conn.commit()
class TestListMCPUserTokenMetadataByUser:
def test_list_mcp_user_token_metadata_by_user_empty(self, backend) -> None:
assert backend.list_mcp_user_token_metadata_by_user("nobody") == []
def test_list_mcp_user_token_metadata_by_user_single_server(self, backend) -> None:
_create_user_token_row(
backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
rows = backend.list_mcp_user_token_metadata_by_user("u1")
assert len(rows) == 1
assert rows[0]["user_id"] == "u1"
assert rows[0]["server_name"] == "srv-a"
assert rows[0]["as_issuer"] == "https://auth.example.com"
assert rows[0]["audience"] == "https://mcp.example.com"
assert rows[0]["scopes"] == "openid"
# Projection MUST omit ciphertext columns — the SQL no longer
# selects them, so the TypedDict has no key.
assert "access_token_ct" not in rows[0]
assert "refresh_token_ct" not in rows[0]
def test_list_mcp_user_token_metadata_by_user_multiple_servers(self, backend) -> None:
_create_user_token_row(
backend, user_id="u1", server_name="srv-c", created="2026-05-03T00:00:00"
)
_create_user_token_row(
backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
_create_user_token_row(
backend, user_id="u1", server_name="srv-b", created="2026-05-02T00:00:00"
)
rows = backend.list_mcp_user_token_metadata_by_user("u1")
assert [r["server_name"] for r in rows] == ["srv-a", "srv-b", "srv-c"]
def test_list_mcp_user_token_metadata_by_user_isolates_by_user(self, backend) -> None:
_create_user_token_row(
backend, user_id="user-a", server_name="srv-a", created="2026-05-01T00:00:00"
)
_create_user_token_row(
backend, user_id="user-a", server_name="srv-b", created="2026-05-02T00:00:00"
)
_create_user_token_row(
backend, user_id="user-b", server_name="srv-a", created="2026-05-03T00:00:00"
)
rows_a = backend.list_mcp_user_token_metadata_by_user("user-a")
assert {r["server_name"] for r in rows_a} == {"srv-a", "srv-b"}
assert all(r["user_id"] == "user-a" for r in rows_a)
rows_b = backend.list_mcp_user_token_metadata_by_user("user-b")
assert len(rows_b) == 1
assert rows_b[0]["user_id"] == "user-b"
assert rows_b[0]["server_name"] == "srv-a"
+304
View File
@@ -0,0 +1,304 @@
"""Boundary tests for the Phase 9 pending-consent write path.
Drives ``MCPClientManager._dispatch_pool_sync`` (and the helper it
calls, ``_record_pending_consent_best_effort``) and asserts that
deferred-consent records reach storage only on non-interactive callers.
Per ``feedback_tests_through_boundaries.md``, at least one test must
drive the real sync dispatcher real ``_is_structured_error``
real ``_record_pending_consent_best_effort`` plumb-through; the
``_helpers`` unit tests below cover the classifier in isolation, but
the end-to-end test is the structural gate that catches
plumb-through regressions.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import threading
from typing import Any
from unittest.mock import patch
import pytest
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import (
_PENDING_CONSENT_PERSIST_CODES,
MCPClientManager,
_parse_pending_consent_envelope,
)
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
# ---------------------------------------------------------------------------
# Helper-level unit tests (cheap, no event loop)
# ---------------------------------------------------------------------------
class TestParseEnvelope:
def test_consent_required_no_scopes(self) -> None:
env = json.dumps({"error": {"code": "mcp_consent_required", "server": "x", "detail": "d"}})
assert _parse_pending_consent_envelope(env) == ("mcp_consent_required", None)
def test_insufficient_scope_with_scopes(self) -> None:
env = json.dumps(
{
"error": {
"code": "mcp_insufficient_scope",
"server": "x",
"detail": "d",
"scopes_required": ["read", "write"],
}
}
)
assert _parse_pending_consent_envelope(env) == (
"mcp_insufficient_scope",
["read", "write"],
)
def test_operator_codes_filtered(self) -> None:
# Key-unknown / url-insecure / *_forbidden are operator-actionable,
# NOT user-consent-shaped. They must not produce pending-consent
# rows, regardless of whether the caller is interactive.
for code in (
"mcp_token_undecryptable_key_unknown",
"mcp_oauth_url_insecure",
"mcp_tool_call_forbidden",
"mcp_resource_read_forbidden",
"mcp_prompt_get_forbidden",
):
env = json.dumps({"error": {"code": code, "server": "x", "detail": "d"}})
assert _parse_pending_consent_envelope(env) is None, code
def test_malformed_json_returns_none(self) -> None:
assert _parse_pending_consent_envelope("not json") is None
assert _parse_pending_consent_envelope("") is None
def test_persist_codes_set_is_expected(self) -> None:
# Pin the contract — adding a new persistable code here is a
# deliberate design decision and should require a test update.
assert {
"mcp_consent_required",
"mcp_insufficient_scope",
} == _PENDING_CONSENT_PERSIST_CODES
# ---------------------------------------------------------------------------
# End-to-end plumb-through (drives _dispatch_pool_sync)
# ---------------------------------------------------------------------------
def _seed_oauth_server(backend: Any, *, name: str = "pool-srv") -> None:
backend.create_mcp_server(
server_id="srv-" + name,
name=name,
transport="streamable-http",
command="",
args="[]",
url="https://example.com/mcp",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-" + name, auth_type="oauth_user")
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="phase9-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _wire_mgr(mgr: MCPClientManager, backend: Any) -> None:
cipher = make_mcp_token_cipher()
from types import SimpleNamespace
from unittest.mock import MagicMock
app_state = SimpleNamespace(
auth_storage=backend,
mcp_token_store=MCPTokenStore(backend, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
mgr.set_storage(backend)
mgr.set_app_state(app_state)
def test_dispatch_persists_pending_for_non_interactive_caller(
running_loop_mgr: Any, backend: Any
) -> None:
"""Non-interactive caller hits ``mcp_consent_required`` → a
``mcp_pending_consent`` row appears for ``(user_id, server_name)``."""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
# Structured error envelope surfaces as RuntimeError to the caller.
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_consent_required"
# Persistent row written for the dashboard badge.
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
assert r["user_id"] == "user-a"
assert r["server_name"] == "pool-srv"
assert r["error_code"] == "mcp_consent_required"
assert r["occurrence_count"] == 1
def test_dispatch_does_not_persist_for_interactive_caller(
running_loop_mgr: Any, backend: Any
) -> None:
"""Interactive caller hits the same error path → NO row written.
Interactive (WEB / CLI) sessions surface the consent prompt in-flight
via the Phase 8 SSE renderer; persisting would just produce
immediately-stale dashboard badges.
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError),
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=True,
)
assert backend.list_mcp_pending_consent_by_user("user-a") == []
def test_dispatch_returns_envelope_unchanged_on_storage_failure(
running_loop_mgr: Any, backend: Any
) -> None:
"""When ``upsert_mcp_pending_consent`` raises, the agent-observable
contract is unchanged: the structured-error ``RuntimeError`` still
surfaces with the original ``mcp_consent_required`` code. The doc-
string promises best-effort persistence; this test pins that
promise so a regression that propagates the storage exception would
fail visibly.
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
original_upsert = backend.upsert_mcp_pending_consent
def _raise(*_a: Any, **_kw: Any) -> None:
raise RuntimeError("storage offline")
backend.upsert_mcp_pending_consent = _raise # type: ignore[method-assign]
try:
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
finally:
backend.upsert_mcp_pending_consent = original_upsert # type: ignore[method-assign]
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_consent_required"
def test_dispatch_does_not_persist_for_operator_actionable_code(
running_loop_mgr: Any, backend: Any
) -> None:
"""Decrypt-failure → operator-actionable; even non-interactive callers
must NOT produce a user-facing pending-consent record (the user can't
resolve this by re-consenting).
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _decrypt_failure(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_decrypt_failure,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_token_undecryptable_key_unknown"
# The operator-actionable code does NOT produce a pending-consent row.
assert backend.list_mcp_pending_consent_by_user("user-a") == []
+259
View File
@@ -0,0 +1,259 @@
"""HTTP tests for the Phase 9 pending-consent endpoints.
Covers:
- ``GET /v1/api/mcp/oauth/pending`` (install gate + read path)
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` (single clear)
- ``DELETE /v1/api/mcp/oauth/pending`` (bulk clear)
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.core.auth import AuthResult
from turnstone.core.mcp_oauth import (
handle_mcp_oauth_clear_all_pending,
handle_mcp_oauth_clear_pending,
handle_mcp_oauth_list_pending,
)
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
class _InjectAuthMiddleware(BaseHTTPMiddleware):
"""Stamp a fixed authenticated user on every request."""
def __init__(self, app: Any, user_id: str = "user-1") -> None:
super().__init__(app)
self._user_id = user_id
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id=self._user_id,
scopes=frozenset({"write"}),
token_source="config",
permissions=frozenset({"read", "write"}),
)
return await call_next(request)
def _build_app(storage: SQLiteBackend, *, user_id: str = "user-1") -> Starlette:
class _Mw(_InjectAuthMiddleware):
def __init__(self, app: Any) -> None:
super().__init__(app, user_id=user_id)
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/mcp/oauth/pending", handle_mcp_oauth_list_pending),
Route(
"/api/mcp/oauth/pending",
handle_mcp_oauth_clear_all_pending,
methods=["DELETE"],
),
Route(
"/api/mcp/oauth/pending/{server_name}",
handle_mcp_oauth_clear_pending,
methods=["DELETE"],
),
],
),
],
middleware=[Middleware(_Mw)],
)
app.state.auth_storage = storage
return app
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
backend = SQLiteBackend(str(tmp_path / "test.db"))
backend.create_user("user-1", "user1", "User One", "hash")
backend.create_user("user-2", "user2", "User Two", "hash")
return backend
def _seed_oauth_server(backend: SQLiteBackend, *, name: str = "srv-x") -> None:
backend.create_mcp_server(
server_id="srv-id-" + name,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="oauth_user",
)
def _seed_pending(
backend: SQLiteBackend,
*,
user_id: str = "user-1",
server_name: str = "srv-x",
error_code: str = "mcp_consent_required",
now_iso: str = "2026-05-11T12:00:00",
) -> None:
backend.upsert_mcp_pending_consent(
user_id=user_id,
server_name=server_name,
error_code=error_code,
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=now_iso,
)
class TestListPending:
def test_install_gate_short_circuits_on_no_oauth_servers(self, storage: SQLiteBackend) -> None:
# Seed a pending row but NO oauth_user MCP server — the gate
# must short-circuit to {pending: 0} regardless.
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"pending": 0, "servers": []}
def test_lists_pending_records_for_authenticated_user(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
body = resp.json()
assert body["pending"] == 1
assert len(body["servers"]) == 1
assert body["servers"][0]["server_name"] == "srv-x"
assert body["servers"][0]["error_code"] == "mcp_consent_required"
def test_does_not_leak_cross_user_records(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage, user_id="user-2")
client = TestClient(_build_app(storage, user_id="user-1"))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"pending": 0, "servers": []}
class TestClearPending:
def test_delete_single(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
assert storage.list_mcp_pending_consent_by_user("user-1") == []
def test_delete_missing_still_returns_204(self, storage: SQLiteBackend) -> None:
# Idempotent — must not leak cross-user existence info via 404.
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
assert resp.status_code == 204
def test_delete_does_not_touch_cross_user_rows(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage, user_id="user-1")
_seed_pending(storage, user_id="user-2")
client = TestClient(_build_app(storage, user_id="user-1"))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
# User-2's row survives.
assert len(storage.list_mcp_pending_consent_by_user("user-2")) == 1
class TestAuditTrail:
def test_single_dismiss_audits(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "single"
assert detail.get("cleared") == 1
def test_single_dismiss_audits_even_when_no_row_existed(self, storage: SQLiteBackend) -> None:
# Cross-tenant non-observability requires a 204 in the never-existed
# case — the audit row distinguishes a real dismiss from a stuffed
# attempt by recording ``cleared=0``.
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
assert resp.status_code == 204
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "single"
assert detail.get("cleared") == 0
def test_bulk_dismiss_audits(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_oauth_server(storage, name="srv-y")
_seed_pending(storage, server_name="srv-x")
_seed_pending(storage, server_name="srv-y")
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 2}
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "bulk"
assert detail.get("cleared") == 2
class TestClearAllPending:
def test_bulk_clear(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_oauth_server(storage, name="srv-y")
_seed_pending(storage, server_name="srv-x")
_seed_pending(storage, server_name="srv-y")
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 2}
assert storage.list_mcp_pending_consent_by_user("user-1") == []
def test_bulk_clear_zero_when_empty(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 0}
+263
View File
@@ -0,0 +1,263 @@
"""Storage CRUD tests for the Phase 9 ``mcp_pending_consent`` table.
Validates protocol additions backing the dashboard pending-consent badge:
- ``upsert_mcp_pending_consent`` insert + on-conflict refresh
- ``list_mcp_pending_consent_by_user`` read path
- ``delete_mcp_pending_consent`` single-row clear
- ``delete_all_mcp_pending_consent_by_user`` bulk clear
- ``count_mcp_consented_users_by_server`` admin status pill
- ``any_oauth_user_mcp_servers`` install-level gate
"""
from __future__ import annotations
def _iso(ts: str = "2026-05-11T12:00:00") -> str:
return ts
class TestUpsertAndList:
def test_insert_round_trip(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required="read write",
last_ws_id="ws-1",
last_tool_call_id="tool-1",
now_iso=_iso(),
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
assert r["user_id"] == "user-a"
assert r["server_name"] == "srv-x"
assert r["error_code"] == "mcp_consent_required"
assert r["scopes_required"] == "read write"
assert r["last_ws_id"] == "ws-1"
assert r["last_tool_call_id"] == "tool-1"
assert r["occurrence_count"] == 1
assert r["first_seen_at"] == r["last_seen_at"]
def test_upsert_bumps_count_and_refreshes_recency(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T12:00:00",
)
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_insufficient_scope",
scopes_required="read",
last_ws_id="ws-2",
last_tool_call_id="tool-2",
now_iso="2026-05-11T13:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
# Recency fields refreshed to the second call's values; count bumped.
assert r["occurrence_count"] == 2
assert r["error_code"] == "mcp_insufficient_scope"
assert r["scopes_required"] == "read"
assert r["last_ws_id"] == "ws-2"
assert r["last_tool_call_id"] == "tool-2"
assert r["last_seen_at"] == "2026-05-11T13:00:00"
# first_seen_at preserved — that's the load-bearing audit value.
assert r["first_seen_at"] == "2026-05-11T12:00:00"
def test_list_orders_by_last_seen_desc(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-old",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T10:00:00",
)
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-new",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T11:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert [r["server_name"] for r in rows] == ["srv-new", "srv-old"]
def test_per_user_isolation(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.list_mcp_pending_consent_by_user("user-b") == []
class TestDelete:
def test_delete_single(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is True
assert backend.list_mcp_pending_consent_by_user("user-a") == []
# Second delete returns False (no row).
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is False
def test_delete_missing_returns_false(self, backend) -> None:
assert backend.delete_mcp_pending_consent("never", "missing") is False
def test_delete_all_by_user(self, backend) -> None:
for name in ("srv-a", "srv-b", "srv-c"):
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name=name,
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
# Cross-user row that must NOT be touched.
backend.upsert_mcp_pending_consent(
user_id="user-b",
server_name="srv-z",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_all_mcp_pending_consent_by_user("user-a") == 3
assert backend.list_mcp_pending_consent_by_user("user-a") == []
assert len(backend.list_mcp_pending_consent_by_user("user-b")) == 1
class TestCountConsentedUsersByServer:
def _seed_server(self, backend, name: str = "srv-x") -> None:
backend.create_mcp_server(
server_id="srv-id-" + name,
name=name,
transport="streamable-http",
command="",
args="[]",
url="https://example.com/mcp",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-id-" + name, auth_type="oauth_user")
def test_counts_distinct_non_expired_users(self, backend) -> None:
self._seed_server(backend)
future = "2099-01-01T00:00:00"
backend.create_mcp_user_token(
"alice",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=future,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
backend.create_mcp_user_token(
"bob",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=None, # null treated as non-expired
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
# Different server — must not count.
self._seed_server(backend, name="srv-y")
backend.create_mcp_user_token(
"carol",
"srv-y",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=future,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
assert backend.count_mcp_consented_users_by_server("srv-x") == 2
assert backend.count_mcp_consented_users_by_server("srv-y") == 1
def test_excludes_expired(self, backend) -> None:
self._seed_server(backend)
backend.create_mcp_user_token(
"alice",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at="2020-01-01T00:00:00", # well in the past
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
assert backend.count_mcp_consented_users_by_server("srv-x") == 0
def test_zero_when_no_rows(self, backend) -> None:
assert backend.count_mcp_consented_users_by_server("missing") == 0
class TestInstallGate:
def test_any_oauth_user_returns_false_on_empty(self, backend) -> None:
assert backend.any_oauth_user_mcp_servers() is False
def test_any_oauth_user_ignores_static_rows(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-1",
name="static-only",
transport="streamable-http",
command="",
args="[]",
url="https://example.com",
headers='{"Authorization": "Bearer x"}',
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
assert backend.any_oauth_user_mcp_servers() is False
def test_any_oauth_user_returns_true_when_one_exists(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-2",
name="oauth-srv",
transport="streamable-http",
command="",
args="[]",
url="https://example.com",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-2", auth_type="oauth_user")
assert backend.any_oauth_user_mcp_servers() is True
+912
View File
@@ -0,0 +1,912 @@
"""Phase 6 integration tests — real-transport drives 401/403 through the SDK.
These are the structural exit criterion for Phase 6. They MUST drive
through the real ``streamablehttp_client``, the real httpx response-hook
path, and a REAL upstream MCP server (a ``FastMCP`` in-process subprocess
with a starlette middleware that programmatically returns 401/403 with
crafted ``WWW-Authenticate`` headers).
Direct ``httpx.HTTPStatusError`` injection is FORBIDDEN here Phase 5
bug-1 was masked precisely by that pattern (the production code path
was structurally unreachable, but the unit-test injection bypassed the
SDK's swallow). The integration tests gate that the production path
actually receives the carrier signal end-to-end.
The fixture upstream is built in-thread (uvicorn on its own asyncio
loop in a background thread) same pattern as
``tests/spike_sdk_concurrency.py``. Per the orchestrator's startup-cost
note, measured at ~0.05s per fixture spin-up locally; well under the
2s threshold for default-collection inclusion.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import socket
import threading
import time
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
import pytest
import uvicorn
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
# Quiet noisy logs during tests.
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
# ---------------------------------------------------------------------------
# Fixture upstream — programmable BehaviorMiddleware
# ---------------------------------------------------------------------------
class BehaviorMiddleware(BaseHTTPMiddleware):
"""Inspects per-request behaviour state and returns 401/403 on demand.
The behaviour is steered by a mutable ``behaviour`` dict on the
middleware instance; tests mutate it via the fixture handle.
Records every request's Authorization header for assertion.
Behaviour semantics:
* ``"once_401"``: return 401 once, then 200 thereafter.
* ``"always_401"``: always return 401.
* ``"once_403_insufficient"``: return 403 with insufficient_scope once.
* ``"once_403_generic"``: return 403 without error param once.
* ``"once_multi_www_auth_403"``: return 403 with TWO
``WWW-Authenticate`` headers first ``Bearer`` challenge
carries the SAFE scopes, second carries INJECTED scopes. The
dispatcher must report only the first.
* ``"never"`` (default): pass through to the real handler.
``www_authenticate`` overrides the default header crafted per shape.
"""
def __init__(self, app: Any, behaviour: dict[str, Any]) -> None:
super().__init__(app)
self._behaviour = behaviour
async def dispatch(self, request: Request, call_next: Callable[..., Any]) -> Response:
from starlette.responses import Response as StarletteResponse
# Record the Authorization header for assertion. POST is the
# tools/call request the dispatcher sends.
if request.method == "POST" and "/mcp" in str(request.url):
self._behaviour.setdefault("post_auth_headers", []).append(
request.headers.get("authorization")
)
mode = self._behaviour.get("mode", "never")
if mode == "once_401":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "always_401":
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "once_403_insufficient":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate",
'Bearer error="insufficient_scope", scope="files:write mail:send"',
)
},
)
elif mode == "once_403_generic" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get("www_authenticate", "Bearer realm=mcp")
},
)
elif mode == "once_multi_www_auth_403" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
# Two ``WWW-Authenticate: Bearer ...`` challenges. The
# first carries ``error=insufficient_scope`` but NO
# ``scope=`` parameter; the second carries the INJECTED
# scopes the dispatcher must NOT report. The first
# challenge intentionally lacks ``scope`` because
# ``parse_www_authenticate_bearer`` uses ``setdefault`` —
# if the first challenge HAD a scope, ``setdefault`` would
# already win on first-occurrence. The vector this test
# guards is the case where a defended absence becomes a
# silent presence: a hook regression to ``get(...)`` joins
# repeated headers with ``, `` and the parser then folds
# the second challenge's scope into the first challenge's
# params dict because there is no first-occurrence to
# protect.
response = StarletteResponse("forbidden", status_code=403)
response.headers.append(
"www-authenticate",
'Bearer realm="legit", error="insufficient_scope"',
)
response.headers.append(
"www-authenticate",
'Bearer error="insufficient_scope", scope="org:admin db:write"',
)
return response
return await call_next(request)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _build_server(port: int, behaviour: dict[str, Any]) -> uvicorn.Server:
mcp = FastMCP(name="phase6-target", streamable_http_path="/mcp")
@mcp.tool()
async def echo(payload: str = "default") -> str:
return f"echoed:{payload}"
app = mcp.streamable_http_app()
app.add_middleware(BehaviorMiddleware, behaviour=behaviour)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"upstream at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def upstream():
"""Boot a FastMCP fixture upstream in a background thread.
Yields ``(url, behaviour)`` where ``behaviour`` is a mutable dict
the test mutates to steer the middleware (set ``mode`` to one of
the BehaviorMiddleware shapes).
"""
port = _find_free_port()
behaviour: dict[str, Any] = {}
server = _build_server(port, behaviour)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="phase6-upstream")
t.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp", behaviour
finally:
server.should_exit = True
t.join(timeout=5)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
refresh_token: str | None = "refresh-rrr",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token=refresh_token,
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
# ---------------------------------------------------------------------------
# Test 21: 401 → refresh-and-retry → success
# ---------------------------------------------------------------------------
def test_integration_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Real upstream returns 401 once with ``WWW-Authenticate: Bearer
error="invalid_token"``, then 200. Dispatcher carrier captures the
401, ``force_refresh=True`` mints a new bearer (stubbed), retry
succeeds. Hard invariant 3: breaker counter remains 0.
Drives through the REAL ``streamablehttp_client`` and a REAL
upstream subprocess (no ``httpx.HTTPStatusError`` injection). This
is the structural exit gate for Phase 6 the equivalent unit
tests CANNOT prove the production wiring works because the SDK
swallows the underlying exception.
"""
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
# Override URL to point at the local upstream (loopback http:// is
# exempt from the URL-validator).
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
result = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "hi"}, user_id="user-1", timeout=15
)
assert "echoed:hi" in result
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# Server saw at least 2 POSTs to /mcp (initial + retry).
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, f"expected >=2 POSTs; got {len(post_headers)}"
# Retry carries a different bearer than the initial.
initial = post_headers[0]
retry = post_headers[1]
assert initial != retry, (
"retry attached the same bearer as the initial; the dispatcher "
"did not pick up the refreshed token."
)
# Pool entry has a session after the successful retry.
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
# ---------------------------------------------------------------------------
# Test 22: 401 + refresh failure → mcp_consent_required
# ---------------------------------------------------------------------------
def test_integration_401_with_refresh_failure_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
# Structured-error envelopes flow back via ``RuntimeError(json_str)``
# so the session-layer ``except Exception`` handler routes the
# consent card uniformly across tool / resource / prompt dispatchers.
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
# Phase 8 — consent_url surfaces a /start URL the dashboard can open
# in a popup. URL-encoded server name; no scopes baked in (the AS
# picks up the configured scopes server-side at /start).
assert payload["error"]["consent_url"] == "/v1/api/mcp/oauth/start?server=pool-srv"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Test 23: 403 + insufficient_scope → mcp_insufficient_scope with parsed scopes
# ---------------------------------------------------------------------------
def test_integration_403_insufficient_scope_emits_structured_error(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_insufficient"
behaviour["www_authenticate"] = (
'Bearer error="insufficient_scope", scope="files:write mail:send"'
)
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_insufficient_scope"
assert payload["error"]["scopes_required"] == ["files:write", "mail:send"]
# Phase 8 — consent_url carries the step-up scopes URL-encoded so the
# dashboard can union them with the configured set at /start.
assert payload["error"]["consent_url"] == (
"/v1/api/mcp/oauth/start?server=pool-srv&scopes=files%3Awrite%20mail%3Asend"
)
# No retry — exactly ONE POST attempted before the structured error.
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Test 24: 403 without insufficient_scope → generic forbidden
# ---------------------------------------------------------------------------
def test_integration_403_no_insufficient_scope_emits_generic_forbidden(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_generic"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_tool_call_forbidden"
assert "scopes_required" not in payload["error"]
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
# ---------------------------------------------------------------------------
# sec-1: multi-WWW-Authenticate header injection — only the FIRST
# Bearer challenge feeds the structured-error / audit emission.
# ---------------------------------------------------------------------------
def test_integration_403_multi_www_authenticate_drops_injected_scopes(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Upstream returns a 403 with TWO ``WWW-Authenticate: Bearer ...``
challenges. The first carries ``error=insufficient_scope`` but NO
``scope=`` parameter; the second carries INJECTED scopes
(``["org:admin", "db:write"]``). The dispatcher must report
``scopes_required == []`` derived from the first challenge alone
never the second challenge's injected scopes.
Two layers of defence cooperate (either alone neutralises the
vector; both run together so a regression in one cannot silently
re-open it):
1. ``_make_capturing_http_factory._hook`` reads
``response.headers.get_list("www-authenticate")[0]`` rather than
``response.headers.get(...)`` the latter joins repeated
headers with ``", "`` which the RFC 7235 tokenizer would
otherwise consume as a continuation of the first challenge.
2. ``parse_www_authenticate_bearer`` stops at the first ``Bearer``
challenge boundary even if the input was already joined, so a
hook regression to ``get(...)`` would NOT re-open the vector.
The first challenge intentionally lacks ``scope=`` the parser
uses ``setdefault`` so a first-occurrence ``scope`` would already
win and mask a single-layer regression. The undefended-absence
case is what proves both layers actually do their job.
Negative-test (CRITICAL Phase 5 lesson): verified by reverting
the hook to ``response.headers.get("www-authenticate")`` AND
removing the ``_looks_like_bearer_challenge_start`` guard in
``parse_www_authenticate_bearer``. The test then fails because
``scopes_required`` becomes ``["org:admin", "db:write"]`` the
injected scopes from the second challenge silently fold into the
first challenge's params dict via httpx's comma-joined header
value (the absence of a first-occurrence scope means nothing
blocks the fold).
"""
url, behaviour = upstream
behaviour["mode"] = "once_multi_www_auth_403"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_insufficient_scope", (
f"expected mcp_insufficient_scope; got {payload!r}"
)
# ``scopes_required`` derives from the FIRST challenge alone, which
# carries no ``scope=`` parameter. The injected second challenge
# MUST NOT appear here.
assert payload["error"]["scopes_required"] == [], (
"Multi-header injection slipped through: dispatcher reported "
"scopes from the SECOND Bearer challenge. Got "
f"{payload['error']['scopes_required']!r}; expected []."
)
# ---------------------------------------------------------------------------
# Test 25: 401 retry ceiling — never recurse
# ---------------------------------------------------------------------------
def test_integration_401_retry_ceiling(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Upstream always returns 401; refresh stub keeps minting tokens.
After exactly ONE retry, dispatcher emits ``mcp_consent_required``.
"""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
refresh_count = 0
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
nonlocal refresh_count
if kwargs.get("force_refresh"):
refresh_count += 1
return TokenLookupResult(kind="token", token=f"refreshed-{refresh_count}")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync("mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
# Exactly ONE refresh round-trip.
assert refresh_count == 1, f"expected exactly 1 refresh round-trip; got {refresh_count}"
# Server saw EXACTLY 2 POSTs (initial + 1 retry).
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 2, (
f"expected exactly 2 POSTs (initial + 1 retry); got {len(post_headers)}"
)
# ---------------------------------------------------------------------------
# Test 26: breaker unaffected by repeated auth failures (slow — 50 cycles)
# ---------------------------------------------------------------------------
def test_integration_breaker_unaffected_by_auth_failures(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""50 sequential dispatches all hit 401 with refresh-failed → 50
cycles of ``mcp_consent_required``. ``_consecutive_failures`` MUST
stay at 0 throughout (hard invariant 3 verified end-to-end).
"""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
for _ in range(50):
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "x"}, user_id="user-1", timeout=15
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Test 27: static path unaffected by Phase 6 changes
# ---------------------------------------------------------------------------
def test_integration_static_path_unaffected(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Static-path connect against an unauthed upstream succeeds without
going through the capturing factory. This is the integration-level
mirror of ``test_reconnect_preserves_static_state_identity``.
Drives the static path against the same fixture upstream (with
``behaviour={}`` so middleware passes through) confirms the
static path's session lifecycle is byte-identical even when the
pool path's auth introspection is wired up.
"""
url, _behaviour = upstream
# No mode → middleware passes through to FastMCP.
mgr, loop, _ = running_loop_mgr
# Manually configure mgr with a static-path server pointing at the
# fixture upstream. Use _connect_one (not the pool path).
cfg = {"type": "streamable-http", "url": url}
async def _connect_static() -> None:
await mgr._connect_one("static-srv", cfg)
fut = asyncio.run_coroutine_threadsafe(_connect_static(), loop)
fut.result(timeout=15)
state_before = mgr._static_servers.get("static-srv")
assert state_before is not None
assert state_before.session is not None
# Snapshot identity.
state_id_before = id(state_before)
session_before = state_before.session
# Reconnect — the canonical regression check is that the
# StaticServerState object identity is preserved.
fut = asyncio.run_coroutine_threadsafe(_connect_static(), loop)
fut.result(timeout=15)
state_after = mgr._static_servers.get("static-srv")
assert state_after is not None
assert id(state_after) == state_id_before, (
"Static path StaticServerState identity changed across reconnect; "
"hard invariant 1 violated."
)
assert state_after.session is not None
assert state_after.session is not session_before, (
"Reconnect did not actually replace the session"
)
# ---------------------------------------------------------------------------
# Test 27b: static dispatch unaffected by Phase 8 consent_url kwarg
# ---------------------------------------------------------------------------
def test_static_dispatch_unaffected_by_consent_url_kwarg(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Static-auth tool dispatch must be byte-identical post-Phase 8.
The Phase 8 changes only ADD a ``consent_url`` kwarg to
``_structured_error`` invocations on the pool path. Static dispatch
must not pick up the field there's no consent flow for
``auth_type='none'`` / ``'static'`` servers, and exposing one would
confuse the dashboard renderer. Asserts a successful tool result is
a plain string with no JSON envelope and no ``consent_url`` substring.
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — succeeds
mgr, loop, _ = running_loop_mgr
cfg = {"type": "streamable-http", "url": url}
async def _connect_static() -> None:
await mgr._connect_one("static-srv", cfg)
fut = asyncio.run_coroutine_threadsafe(_connect_static(), loop)
fut.result(timeout=15)
# Drive call_tool_sync without a user_id — the static path is taken.
result = mgr.call_tool_sync("mcp__static-srv__echo", {"payload": "static-x"}, timeout=15)
# Static path returns the FastMCP fixture's echo string.
assert "echoed:static-x" in result
# No JSON envelope leaked through; specifically no consent_url field.
assert "consent_url" not in result, (
f"Static-auth tool dispatch surfaced a consent_url; result: {result!r}"
)
# Defensive: result is not a JSON-encoded structured error.
try:
parsed = json.loads(result)
except (json.JSONDecodeError, ValueError):
parsed = None
if isinstance(parsed, dict):
assert "error" not in parsed, (
f"Static-auth dispatch returned a structured-error envelope; got {parsed!r}"
)
# ---------------------------------------------------------------------------
# Test 28: pool reuse — 401 on a SECOND dispatch (carrier owned by entry)
# ---------------------------------------------------------------------------
def test_integration_pool_reuse_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Reused pool sessions still capture 401 correctly.
Dispatch 1 hits a passthrough upstream (200) and populates
``entry.session``. Dispatch 2 reuses that session no fresh
connect, so a per-dispatch ``_AuthCapture`` would never reach
the httpx response hook (the hook closes over the carrier passed
at first connect, which lives on the entry). A correctly-wired
entry-owned carrier is the only shape that lets dispatch 2's 401
surface to the dispatcher.
Two independent production bugs gate this test passing; both must
hold for reused-session 401 recovery to work end-to-end:
1. The carrier must live on the pool entry (not per-dispatch) so
the response hook bound at first connect writes to the same
object the dispatcher reads across reuse. Verified by reverting
``PoolEntryState.auth_capture`` to a per-dispatch
``_AuthCapture()`` allocation: the carrier-fired event never
reaches the dispatcher and the test times out.
2. The dispatcher must race ``call_tool`` against the carrier's
fired event. The SDK's ``_receive_loop`` runs in BaseSession's
TaskGroup nested inside ``streamablehttp_client``'s TaskGroup;
when an upstream 4xx fires, the outer TaskGroup cancels
``_receive_loop`` mid-finally before it can deliver
``CONNECTION_CLOSED`` to the response stream's waiting
receiver. anyio's ``send_nowait`` skips waiters with pending
cancellation but our dispatch task (created via
``run_coroutine_threadsafe`` for the reused-session case) has
NO pending cancellation, so the send delivers but the receiver
never wakes (the waiter's Event is set on stale state). Result:
a forever-hung ``response_stream_reader.receive()``. Verified
by reverting the ``asyncio.wait({call_task, fired_task})``
race in ``_dispatch_pool_with_entry`` to a bare ``await
session.call_tool(...)``: the test times out.
This test is the structural gate against the per-dispatch carrier
pattern: it looks right in code review and passes single-dispatch
integration tests, but breaks silently on session reuse and the
SDK-level hang the carrier fix exposes silently strands the
dispatcher even when the carrier is correct.
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — dispatch 1 succeeds
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
# Dispatch 1: passthrough success. Establishes the pooled session.
result1 = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "first"}, user_id="user-1", timeout=15
)
assert "echoed:first" in result1
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
session_after_first = entry.session
assert session_after_first is not None, (
"test setup: dispatch 1 did not populate entry.session; "
"subsequent dispatch will not exercise the reuse path"
)
# Reconfigure upstream to 401 once on the next call. Reset the
# auth-headers log so we can count dispatch-2's POSTs cleanly.
behaviour["post_auth_headers"] = []
behaviour["mode"] = "once_401"
behaviour["_fired"] = False
# Dispatch 2: same (user, server). The hook from dispatch 1's
# connect is still bound to entry.auth_capture. The 401 fires;
# the dispatcher's auth_401 path triggers refresh-and-retry.
result2 = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "second"}, user_id="user-1", timeout=15
)
assert "echoed:second" in result2, (
f"reused-session 401 retry did not succeed. result: {result2!r}. "
"If this is JSON with mcp_consent_required, the dispatcher "
"fell through to consent_required emission; if a generic "
"tool error, the carrier was empty (auth branch unreachable)."
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0, (
"auth failures must not trip the per-server breaker"
)
# Dispatch 2 produces multiple POSTs: the original 401 with the
# rejected bearer, then the retry's full connect handshake
# (initialize + notifications/initialized + tools/list) followed by
# the actual tools/call — all under the refreshed bearer. The retry
# reconnects because the auth_401 handler evicted the broken session.
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, (
f"expected >=2 POSTs after dispatch 2 (401 + retry); "
f"got {len(post_headers)}: {post_headers}"
)
# First POST is the original bearer that got 401'd.
assert post_headers[0] == "Bearer access-aaa", (
f"first POST was {post_headers[0]!r}; expected the original bearer"
)
# Every subsequent POST carries the refreshed bearer (the retry
# ran with force_refresh=True and reconnected with the new token).
refreshed = post_headers[1:]
assert all(h == "Bearer refreshed-bearer" for h in refreshed), (
f"retry POSTs carried unexpected bearer(s); observed: {post_headers}"
)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,720 @@
"""Phase 7b integration tests — real-transport prompt get 401/403/etc.
Mirror of :mod:`tests.test_mcp_pool_auth_resource_integration` for the
prompt path (RFC §3.3). Drives through the real ``streamablehttp_client``,
real httpx response-hook plumbing, and a real upstream subprocess
(``FastMCP`` with a programmable ``BehaviorMiddleware``). Direct
``httpx.HTTPStatusError`` injection is forbidden (invariant 14).
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import socket
import threading
import time
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
import pytest
import uvicorn
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
class BehaviorMiddleware(BaseHTTPMiddleware):
"""Programmable upstream behaviour — see
:mod:`tests.test_mcp_pool_auth_integration` for the semantics. This
copy serves the prompt integration tests.
"""
def __init__(self, app: Any, behaviour: dict[str, Any]) -> None:
super().__init__(app)
self._behaviour = behaviour
async def dispatch(self, request: Request, call_next: Callable[..., Any]) -> Response:
from starlette.responses import Response as StarletteResponse
if request.method == "POST" and "/mcp" in str(request.url):
self._behaviour.setdefault("post_auth_headers", []).append(
request.headers.get("authorization")
)
mode = self._behaviour.get("mode", "never")
if mode == "once_401":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "always_401":
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "once_403_insufficient":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate",
'Bearer error="insufficient_scope", scope="prompts:read"',
)
},
)
elif mode == "once_403_generic" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get("www_authenticate", "Bearer realm=mcp")
},
)
return await call_next(request)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _build_server(port: int, behaviour: dict[str, Any]) -> uvicorn.Server:
mcp = FastMCP(name="phase7b-prompt-target", streamable_http_path="/mcp")
@mcp.prompt()
def greet(who: str = "world") -> str:
return f"Hello, {who}!"
@mcp.prompt()
def summarize(topic: str = "today") -> str:
return f"Please summarize {topic}."
app = mcp.streamable_http_app()
app.add_middleware(BehaviorMiddleware, behaviour=behaviour)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"upstream at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def upstream():
port = _find_free_port()
behaviour: dict[str, Any] = {}
server = _build_server(port, behaviour)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="phase7b-prompt-upstream")
t.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp", behaviour
finally:
server.should_exit = True
t.join(timeout=5)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
refresh_token: str | None = "refresh-rrr",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token=refresh_token,
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _seed_pool_prompt_map(
mgr: MCPClientManager,
user_id: str,
server_name: str,
prefixed_name: str,
original_name: str,
) -> None:
"""Pre-seed ``_user_prompt_map`` so ``_resolve_pool_target_prompt``
finds the prefixed name. Production wires this through
``_connect_one_pool``; the integration tests seed it directly so the
test focuses on the dispatch behaviour after resolution succeeds.
"""
async def _seed() -> None:
entry = await mgr._ensure_pool_entry((user_id, server_name))
entry.prompts = [
{
"name": prefixed_name,
"original_name": original_name,
"server": server_name,
"description": "",
"arguments": [],
}
]
mgr._rebuild_user_prompt_map(user_id)
assert mgr._loop is not None
asyncio.run_coroutine_threadsafe(_seed(), mgr._loop).result(timeout=5)
# ---------------------------------------------------------------------------
# I-PR-1: 401 → refresh → retry → success (prompt path)
# ---------------------------------------------------------------------------
def test_prompt_get_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Real upstream returns 401 once, then 200. Carrier captures 401,
force_refresh=True mints a new bearer, retry returns the prompt
messages. Hard invariant 3: breaker counter remains 0.
"""
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
messages = mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "everyone"},
user_id="user-1",
timeout=15,
)
assert isinstance(messages, list)
assert len(messages) == 1
assert messages[0]["role"] == "user"
assert "everyone" in messages[0]["content"]
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, f"expected >=2 POSTs; got {len(post_headers)}"
assert post_headers[0] != post_headers[1], (
"retry attached the same bearer as the initial; the dispatcher "
"did not pick up the refreshed token."
)
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
# ---------------------------------------------------------------------------
# I-PR-2: persistent 401 → mcp_consent_required (prompt path) → RuntimeError
# ---------------------------------------------------------------------------
def test_prompt_get_persistent_401_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-PR-3: 403 + insufficient_scope → mcp_insufficient_scope (prompt path)
# ---------------------------------------------------------------------------
def test_prompt_get_403_insufficient_scope_emits_structured_error(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_insufficient"
behaviour["www_authenticate"] = 'Bearer error="insufficient_scope", scope="prompts:read"'
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_insufficient_scope"
assert payload["error"]["scopes_required"] == ["prompts:read"]
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-PR-3b: 403 generic → mcp_prompt_get_forbidden
# ---------------------------------------------------------------------------
def test_prompt_get_403_generic_forbidden(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_generic"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
# Per the kind="prompt" wiring of `_handle_auth_403`, the
# operation-specific code surfaces here rather than the tool path's
# generic mcp_tool_call_forbidden.
assert payload["error"]["code"] == "mcp_prompt_get_forbidden"
assert "scopes_required" not in payload["error"]
# ---------------------------------------------------------------------------
# I-PR-6: breaker isolation — auth failures NEVER trip the breaker
# ---------------------------------------------------------------------------
def test_prompt_get_breaker_unaffected_by_auth_failures(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Repeated 401 + refresh-failed cycles leave breaker at 0
(hard invariant 3 verified end-to-end for the prompt path)."""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
# Re-seed each iteration: symmetric eviction (Phase 7b) clears
# ``_user_prompt_map`` on auth failure so the next dispatch's
# resolver would miss without a fresh seed. Production reconnect
# repopulates this; the test simulates that out-of-band.
for _ in range(10):
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Negative tests — token lookup edge cases (prompt path)
# ---------------------------------------------------------------------------
def test_prompt_get_missing_token_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=10,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_consent_required"
def test_prompt_get_decrypt_failure_emits_token_undecryptable(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=10,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_token_undecryptable_key_unknown"
def test_prompt_get_http_url_emits_url_insecure(
running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""An ``http://`` (non-loopback) oauth_user URL must surface
``mcp_oauth_url_insecure`` BEFORE the bearer is attached.
"""
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://example.com/mcp")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_prompt_map(mgr, "user-1", "pool-srv", "mcp__pool-srv__greet", "greet")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as excinfo,
):
mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(excinfo.value))
assert payload["error"]["code"] == "mcp_oauth_url_insecure"
def test_prompt_get_unknown_name_raises_value_error(
running_loop_mgr: Any,
) -> None:
"""When the prefixed name doesn't resolve to either pool or static,
the static-path code raises ``ValueError``. Per-user-first
resolution (scope decision 0.1) means user_id-bearing callers still
hit this path when their pool catalog doesn't carry the name."""
mgr, _loop, _ = running_loop_mgr
with pytest.raises(ValueError, match="Unknown MCP prompt"):
mgr.get_prompt_sync(
"mcp__nonexistent__missing",
None,
user_id="user-1",
timeout=5,
)
# ---------------------------------------------------------------------------
# I-PR-E2E: real discovery + dispatch in same connect (no _seed_pool_prompt_map)
# ---------------------------------------------------------------------------
def test_prompt_get_e2e_discovery_then_dispatch_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Drive REAL discovery + dispatch end-to-end through the pool path.
Mirror of the tool path's
``test_integration_pool_reuse_401_refresh_and_retry_succeeds``: skips
the ``_seed_pool_prompt_map`` shortcut and lets ``_connect_one_pool``
populate ``_user_prompt_map`` from the real ``prompts/list``
upstream response. Verifies that the entry's discovered prompts
match what the FastMCP fixture advertises AND that
``_user_prompt_map[user_id]`` is populated with the prefixed name
after dispatch proving the discovery path actually fired.
This is the structural gate against a regression where prompt
dispatch silently bypasses discovery (e.g., a mis-wired resolver
that finds the (server, original) via prefix-parsing alone never
populates the per-user catalog).
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — discovery + dispatch both succeed
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
# NB: no `_seed_pool_prompt_map` — the resolver finds (server, original)
# via the `mcp__{server}__{prompt}` prefix and hands off to
# ``_dispatch_pool_prompt_sync``, which lazy-connects via
# ``_connect_one_pool``. The connect runs the real ``prompts/list``
# against the FastMCP fixture and populates the per-user catalog.
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
messages = mgr.get_prompt_sync(
"mcp__pool-srv__greet",
{"who": "world"},
user_id="user-1",
timeout=15,
)
assert isinstance(messages, list)
assert len(messages) == 1
assert messages[0]["role"] == "user"
assert "world" in messages[0]["content"]
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# Discovery populated the entry's prompts with both fixtures
# (``greet`` and ``summarize``) — proves real ``prompts/list``
# ran during the connect, not just the targeted ``prompts/get``.
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
assert entry.prompts is not None
discovered_names = {p["name"] for p in entry.prompts}
assert "mcp__pool-srv__greet" in discovered_names
assert "mcp__pool-srv__summarize" in discovered_names
# ``_rebuild_user_prompt_map`` ran during the connect, populating the
# per-user catalog. This is the signal that discovery wired into the
# routing tables — without it, a follow-up ``get_prompt_sync`` would
# need to re-resolve via prefix parsing every time.
user_prompt_map = mgr._user_prompt_map.get("user-1") or {}
assert "mcp__pool-srv__greet" in user_prompt_map
assert "mcp__pool-srv__summarize" in user_prompt_map
@@ -0,0 +1,690 @@
"""Phase 7b integration tests — real-transport resource read 401/403/etc.
Mirror of :mod:`tests.test_mcp_pool_auth_integration` for the resource
path (RFC §3.2). Drives through the real ``streamablehttp_client``,
real httpx response-hook plumbing, and a real upstream subprocess
(``FastMCP`` with a programmable ``BehaviorMiddleware``). Direct
``httpx.HTTPStatusError`` injection is forbidden (invariant 14).
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import socket
import threading
import time
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
import pytest
import uvicorn
from mcp.server.fastmcp import FastMCP
from starlette.middleware.base import BaseHTTPMiddleware
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from collections.abc import Callable
from starlette.requests import Request
from starlette.responses import Response
logging.getLogger("uvicorn.error").setLevel(logging.WARNING)
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("mcp").setLevel(logging.WARNING)
class BehaviorMiddleware(BaseHTTPMiddleware):
"""Programmable upstream behaviour — see
:mod:`tests.test_mcp_pool_auth_integration` for the semantics. This
copy serves the resource integration tests.
"""
def __init__(self, app: Any, behaviour: dict[str, Any]) -> None:
super().__init__(app)
self._behaviour = behaviour
async def dispatch(self, request: Request, call_next: Callable[..., Any]) -> Response:
from starlette.responses import Response as StarletteResponse
if request.method == "POST" and "/mcp" in str(request.url):
self._behaviour.setdefault("post_auth_headers", []).append(
request.headers.get("authorization")
)
mode = self._behaviour.get("mode", "never")
if mode == "once_401":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "always_401":
return StarletteResponse(
"unauthorized",
status_code=401,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate", 'Bearer error="invalid_token"'
)
},
)
elif mode == "once_403_insufficient":
if not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get(
"www_authenticate",
'Bearer error="insufficient_scope", scope="files:read"',
)
},
)
elif mode == "once_403_generic" and not self._behaviour.get("_fired"):
self._behaviour["_fired"] = True
return StarletteResponse(
"forbidden",
status_code=403,
headers={
"www-authenticate": self._behaviour.get("www_authenticate", "Bearer realm=mcp")
},
)
return await call_next(request)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return port
def _build_server(port: int, behaviour: dict[str, Any]) -> uvicorn.Server:
mcp = FastMCP(name="phase7b-resource-target", streamable_http_path="/mcp")
@mcp.resource("res://hello")
def hello() -> str:
return "world"
@mcp.resource("res://json/data")
def jdata() -> str:
return '{"k": 1}'
# Echo tool exists so the e2e test can trigger ``_connect_one_pool``
# (and the full tool + resource + prompt discovery) via prefix-parsed
# ``call_tool_sync`` BEFORE the resource read. The other tests in this
# module use ``_seed_pool_resource_map`` and never invoke tools, so
# adding the tool is invisible to them.
@mcp.tool()
async def echo(payload: str = "default") -> str:
return f"echoed:{payload}"
app = mcp.streamable_http_app()
app.add_middleware(BehaviorMiddleware, behaviour=behaviour)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"upstream at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def upstream():
port = _find_free_port()
behaviour: dict[str, Any] = {}
server = _build_server(port, behaviour)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
t = threading.Thread(target=_run, daemon=True, name="phase7b-resource-upstream")
t.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp", behaviour
finally:
server.should_exit = True
t.join(timeout=5)
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
refresh_token: str | None = "refresh-rrr",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token=refresh_token,
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _seed_pool_resource_map(
mgr: MCPClientManager, user_id: str, server_name: str, uri: str
) -> None:
"""Pre-seed ``_user_resource_map`` so ``_resolve_pool_target_resource``
finds the URI. Production wires this through ``_connect_one_pool``;
the integration tests seed it directly so the test focuses on the
dispatch behaviour after resolution succeeds.
"""
async def _seed() -> None:
entry = await mgr._ensure_pool_entry((user_id, server_name))
entry.resources = [
{
"uri": uri,
"name": "",
"description": "",
"mimeType": "",
"server": server_name,
}
]
mgr._rebuild_user_resource_map(user_id)
assert mgr._loop is not None
asyncio.run_coroutine_threadsafe(_seed(), mgr._loop).result(timeout=5)
# ---------------------------------------------------------------------------
# I-RP-1: 401 → refresh → retry → success (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_401_refresh_and_retry_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Real upstream returns 401 once, then 200. Carrier captures 401,
force_refresh=True mints a new bearer, retry returns the resource.
Hard invariant 3: breaker counter remains 0.
"""
url, behaviour = upstream
behaviour["mode"] = "once_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
result = mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
assert result == "world"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) >= 2, f"expected >=2 POSTs; got {len(post_headers)}"
assert post_headers[0] != post_headers[1], (
"retry attached the same bearer as the initial; the dispatcher "
"did not pick up the refreshed token."
)
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
# ---------------------------------------------------------------------------
# I-RP-2: persistent 401 → mcp_consent_required (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_persistent_401_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="token", token="refreshed-bearer")
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-RP-3: 403 + insufficient_scope → mcp_insufficient_scope (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_403_insufficient_scope_emits_structured_error(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_insufficient"
behaviour["www_authenticate"] = 'Bearer error="insufficient_scope", scope="files:read"'
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_insufficient_scope"
assert payload["error"]["scopes_required"] == ["files:read"]
post_headers = behaviour.get("post_auth_headers", [])
assert len(post_headers) == 1, (
f"403 must NOT trigger a retry; observed {len(post_headers)} POSTs"
)
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# I-RP-3b: 403 generic → mcp_resource_read_forbidden
# ---------------------------------------------------------------------------
def test_resource_read_403_generic_forbidden(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, behaviour = upstream
behaviour["mode"] = "once_403_generic"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
# Per the kind="resource" wiring of `_handle_auth_403`, the
# operation-specific code surfaces here rather than the tool path's
# generic mcp_tool_call_forbidden.
assert payload["error"]["code"] == "mcp_resource_read_forbidden"
assert "scopes_required" not in payload["error"]
# ---------------------------------------------------------------------------
# I-RP-6: breaker isolation — auth failures NEVER trip the breaker
# ---------------------------------------------------------------------------
def test_resource_read_breaker_unaffected_by_auth_failures(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Repeated 401 + refresh-failed cycles leave breaker at 0
(hard invariant 3 verified end-to-end for the resource path)."""
url, behaviour = upstream
behaviour["mode"] = "always_401"
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**kwargs: Any) -> TokenLookupResult:
if kwargs.get("force_refresh"):
return TokenLookupResult(kind="refresh_failed")
return TokenLookupResult(kind="token", token="access-aaa")
# Re-seed each iteration: symmetric eviction (Phase 7b) clears
# ``_user_resource_map`` on auth failure so the next dispatch's
# resolver would miss without a fresh seed. Production reconnect
# repopulates this; the test simulates that out-of-band.
for _ in range(10):
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# ---------------------------------------------------------------------------
# Negative tests — token lookup edge cases (resource path)
# ---------------------------------------------------------------------------
def test_resource_read_missing_token_emits_consent_required(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=10)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
def test_resource_read_decrypt_failure_emits_token_undecryptable(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
url, _behaviour = upstream
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=10)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_token_undecryptable_key_unknown"
def test_resource_read_http_url_emits_url_insecure(
running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""An ``http://`` (non-loopback) oauth_user URL must surface
``mcp_oauth_url_insecure`` BEFORE the bearer is attached.
"""
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://example.com/mcp")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
_seed_pool_resource_map(mgr, "user-1", "pool-srv", "res://hello")
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.read_resource_sync("res://hello", user_id="user-1", timeout=5)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_oauth_url_insecure"
def test_resource_read_unknown_uri_raises_value_error(
running_loop_mgr: Any,
) -> None:
"""When the URI doesn't resolve to either pool or static, the
static-path code raises ``ValueError``. Per-user-first resolution
(scope decision 0.1) means user_id-bearing callers still hit this
path when their pool catalog doesn't carry the URI."""
mgr, _loop, _ = running_loop_mgr
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("res://nonexistent", user_id="user-1", timeout=5)
# ---------------------------------------------------------------------------
# I-RP-E2E: real discovery + dispatch in same connect (no _seed_pool_resource_map)
# ---------------------------------------------------------------------------
def test_resource_read_e2e_discovery_then_dispatch_succeeds(
upstream: Any, running_loop_mgr: Any, storage: SQLiteBackend
) -> None:
"""Drive REAL discovery + dispatch end-to-end through the pool path.
Mirror of the tool path's
``test_integration_pool_reuse_401_refresh_and_retry_succeeds``: skips
the ``_seed_pool_resource_map`` shortcut and lets ``_connect_one_pool``
populate ``_user_resource_map`` from the real ``resources/list``
upstream response. Verifies that the entry's discovered resources
match what the FastMCP fixture advertises AND that
``_user_resource_map[user_id]`` is populated with the URI(s) after
discovery proving the discovery path actually fired.
Resource URIs do NOT carry a server-name prefix (unlike tools and
prompts), so the resource resolver cannot derive (server, uri) by
parsing alone. The test triggers the connect via a prefix-parsed
``call_tool_sync`` first (which runs the full
tools+resources+prompts discovery against the FastMCP fixture),
then drives ``read_resource_sync`` against a URI that the
upstream advertised proving that real discovery wired the URI
into the per-user catalog.
Structural gate against a regression where resource discovery is
silently skipped (e.g., a capability-gating bug that drops the
``resources/list`` call but keeps the connect succeeding).
"""
url, behaviour = upstream
behaviour["mode"] = "never" # passthrough — discovery + dispatch both succeed
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url=url)
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
mgr.set_app_state(_make_app_state(storage, cipher=cipher))
# NB: no `_seed_pool_resource_map` — the connect runs the real
# ``resources/list`` against the FastMCP fixture and populates the
# per-user catalog. The tool call below triggers that connect because
# ``_resolve_pool_target`` derives (server, original) from the
# ``mcp__pool-srv__echo`` prefix and lazy-connects via
# ``_connect_one_pool``.
async def _fake_classified(**_kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="token", token="access-aaa")
with patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_fake_classified,
):
# Step 1: trigger the connect via prefix-parsed tool dispatch.
# Discovery (tools + resources + prompts) populates the per-user
# catalogs.
tool_result = mgr.call_tool_sync(
"mcp__pool-srv__echo", {"payload": "ignite"}, user_id="user-1", timeout=15
)
assert "echoed:ignite" in tool_result
# Step 2: now that discovery has populated ``_user_resource_map``,
# the resource resolver finds ``res://hello`` and dispatches the
# read on the SAME pool entry / session.
result = mgr.read_resource_sync("res://hello", user_id="user-1", timeout=15)
assert result == "world"
assert mgr._consecutive_failures.get("pool-srv", 0) == 0
# Discovery populated the entry's resources with both fixtures
# (``res://hello`` and ``res://json/data``) — proves real
# ``resources/list`` ran during the connect, not just the targeted
# ``resources/read``.
entry = mgr._user_pool_entries[("user-1", "pool-srv")]
assert entry.session is not None
assert entry.resources is not None
discovered_uris = {r["uri"] for r in entry.resources if not r.get("template")}
assert "res://hello" in discovered_uris
assert "res://json/data" in discovered_uris
# ``_rebuild_user_resource_map`` ran during the connect, populating
# the per-user catalog. This is the signal that discovery wired into
# the routing tables — without it, ``read_resource_sync`` would have
# raised ValueError because the resolver had no entry for the URI.
user_resource_map = mgr._user_resource_map.get("user-1") or {}
assert "res://hello" in user_resource_map
assert "res://json/data" in user_resource_map
+286
View File
@@ -0,0 +1,286 @@
"""Tests for ``MCPTokenStore`` ciphertext-aware CRUD.
Phase 3 of the OAuth-MCP RFC: validates the encrypt/decrypt boundary
between :class:`MCPTokenStore` and the storage protocol's ciphertext-only
columns. Exercises the row-not-deleted-on-decrypt-failure invariant.
"""
from __future__ import annotations
import base64
import pytest
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenDecryptError,
MCPTokenStore,
)
def _make_cipher() -> MCPTokenCipher:
raw = base64.urlsafe_b64decode(Fernet.generate_key())
return MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,)))
def _make_store(backend, *, audit: bool = False) -> tuple[MCPTokenStore, MCPTokenCipher]:
cipher = _make_cipher()
store = MCPTokenStore(
backend,
cipher,
node_id="test-node",
audit_storage=backend if audit else None,
)
return store, cipher
def _seed_server(backend, *, server_id: str = "srv-id-1", name: str = "srv-a") -> str:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://mcp.example.com/sse",
auth_type="oauth_user",
)
return server_id
# ---------------------------------------------------------------------------
# User-token CRUD
# ---------------------------------------------------------------------------
class TestUserTokenCRUD:
def test_create_and_get_round_trip(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="access-aaa",
refresh_token="refresh-bbb",
expires_at="2026-05-04T12:00:00",
scopes="openid profile",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
plain = store.get_user_token("u1", "srv-a")
assert plain is not None
assert plain["user_id"] == "u1"
assert plain["server_name"] == "srv-a"
assert plain["access_token"] == "access-aaa"
assert plain["refresh_token"] == "refresh-bbb"
assert plain["scopes"] == "openid profile"
assert plain["audience"] == "https://mcp.example.com"
def test_create_with_no_refresh_token(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="access-only",
refresh_token=None,
expires_at=None,
scopes=None,
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
plain = store.get_user_token("u1", "srv-a")
assert plain is not None
assert plain["access_token"] == "access-only"
assert plain["refresh_token"] is None
def test_get_missing_returns_none(self, backend) -> None:
store, _ = _make_store(backend)
assert store.get_user_token("nobody", "srv-a") is None
def test_update_after_refresh(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="old-access",
refresh_token="old-refresh",
expires_at="2026-05-04T12:00:00",
scopes="openid",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
ok = store.update_user_token_after_refresh(
"u1",
"srv-a",
access_token="new-access",
refresh_token="new-refresh",
expires_at="2026-05-04T13:00:00",
)
assert ok is True
plain = store.get_user_token("u1", "srv-a")
assert plain is not None
assert plain["access_token"] == "new-access"
assert plain["refresh_token"] == "new-refresh"
assert plain["expires_at"] == "2026-05-04T13:00:00"
# Preserved columns:
assert plain["scopes"] == "openid"
assert plain["as_issuer"] == "https://auth.example.com"
# last_refreshed got stamped:
assert plain["last_refreshed"] is not None
def test_update_after_refresh_missing_row_returns_false(self, backend) -> None:
store, _ = _make_store(backend)
ok = store.update_user_token_after_refresh(
"u1",
"srv-a",
access_token="x",
refresh_token=None,
expires_at=None,
)
assert ok is False
def test_delete(self, backend) -> None:
store, _ = _make_store(backend)
store.create_user_token(
"u1",
"srv-a",
access_token="a",
refresh_token=None,
expires_at=None,
scopes=None,
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
assert store.delete_user_token("u1", "srv-a") is True
assert store.get_user_token("u1", "srv-a") is None
# Idempotent: deleting again returns False.
assert store.delete_user_token("u1", "srv-a") is False
# ---------------------------------------------------------------------------
# Client-secret writer
# ---------------------------------------------------------------------------
class TestClientSecretWriter:
def test_set_oauth_client_secret_round_trip(self, backend) -> None:
store, cipher = _make_store(backend)
server_id = _seed_server(backend)
ok = store.set_oauth_client_secret(server_id, "plaintext-secret")
assert ok is True
# Read raw via get_mcp_server: ciphertext != plaintext, decrypts back.
raw = backend.get_mcp_server(server_id)
assert raw is not None
ct = raw["oauth_client_secret_ct"]
assert isinstance(ct, (bytes, bytearray, memoryview))
ct_bytes = bytes(ct)
assert ct_bytes != b"plaintext-secret"
assert cipher.decrypt(ct_bytes) == b"plaintext-secret"
def test_set_oauth_client_secret_clear_with_none(self, backend) -> None:
store, _ = _make_store(backend)
server_id = _seed_server(backend)
store.set_oauth_client_secret(server_id, "x")
assert store.set_oauth_client_secret(server_id, None) is True
raw = backend.get_mcp_server(server_id)
assert raw is not None
assert raw["oauth_client_secret_ct"] is None
def test_set_oauth_client_secret_missing_server_returns_false(self, backend) -> None:
store, _ = _make_store(backend)
ok = store.set_oauth_client_secret("does-not-exist", "x")
assert ok is False
# ---------------------------------------------------------------------------
# Decrypt failure: row preservation invariant
# ---------------------------------------------------------------------------
class TestDecryptFailureInvariant:
def test_get_user_token_with_wrong_key_raises_decrypt_error(self, backend) -> None:
"""CRITICAL: when no installed key can decrypt a stored row,
``get_user_token`` MUST NOT auto-delete the row. The row is
still valid; this node just doesn't have the right key.
"""
# Write under cipher A.
store_a, _cipher_a = _make_store(backend)
store_a.create_user_token(
"u1",
"srv-a",
access_token="secret-access",
refresh_token="secret-refresh",
expires_at="2026-05-04T12:00:00",
scopes="openid",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
raw_before = backend.get_mcp_user_token("u1", "srv-a")
assert raw_before is not None
ct_before = bytes(raw_before["access_token_ct"])
# Read under cipher B (different key).
store_b, cipher_b = _make_store(backend)
with pytest.raises(MCPTokenDecryptError) as exc_info:
store_b.get_user_token("u1", "srv-a")
# The exception carries the keys we tried — useful for audit.
assert exc_info.value.key_fingerprints_attempted == cipher_b.key_fingerprints
# Row MUST still exist with ciphertext intact.
raw_after = backend.get_mcp_user_token("u1", "srv-a")
assert raw_after is not None
assert bytes(raw_after["access_token_ct"]) == ct_before
def test_decrypt_failure_emits_audit_when_configured(self, backend) -> None:
"""When ``audit_storage`` is set, decrypt failures emit a
``mcp_server.oauth.token_decrypt_failure`` audit event."""
store_a, _ = _make_store(backend)
store_a.create_user_token(
"u1",
"srv-a",
access_token="x",
refresh_token=None,
expires_at=None,
scopes=None,
as_issuer="https://a",
audience="https://m",
)
store_b, cipher_b = _make_store(backend, audit=True)
with pytest.raises(MCPTokenDecryptError):
store_b.get_user_token("u1", "srv-a")
events = backend.list_audit_events(limit=10)
actions = {ev.get("action") for ev in events}
assert "mcp_server.oauth.token_decrypt_failure" in actions
# ---------------------------------------------------------------------------
# Client-secret reader — q-9
# ---------------------------------------------------------------------------
class TestClientSecretReader:
def test_get_oauth_client_secret_returns_none_when_row_absent(self, backend) -> None:
store, _ = _make_store(backend)
assert store.get_oauth_client_secret("does-not-exist") is None
def test_get_oauth_client_secret_returns_none_when_column_null(self, backend) -> None:
store, _ = _make_store(backend)
server_id = _seed_server(backend)
# No set_oauth_client_secret call — column stays NULL.
assert store.get_oauth_client_secret(server_id) is None
def test_get_oauth_client_secret_round_trip(self, backend) -> None:
store, _ = _make_store(backend)
server_id = _seed_server(backend)
store.set_oauth_client_secret(server_id, "shhh-its-secret")
assert store.get_oauth_client_secret(server_id) == "shhh-its-secret"
def test_get_oauth_client_secret_raises_on_key_mismatch(self, backend) -> None:
store_a, _ = _make_store(backend)
server_id = _seed_server(backend)
store_a.set_oauth_client_secret(server_id, "secret-under-key-a")
# Cipher B has a different key — decrypt fails loudly.
store_b, _ = _make_store(backend)
with pytest.raises(MCPTokenDecryptError):
store_b.get_oauth_client_secret(server_id)
+107
View File
@@ -0,0 +1,107 @@
"""Tests for ``MCPTokenStore.list_user_token_metadata``.
Validates the non-secret projection used by the settings UI: ciphertext
columns are stripped, ordering is preserved, and the empty case returns
``[]``. Decrypt is intentionally skipped the list view must never need
the access/refresh secrets.
"""
from __future__ import annotations
import base64
import sqlalchemy as sa
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenStore,
)
def _make_cipher() -> MCPTokenCipher:
raw = base64.urlsafe_b64decode(Fernet.generate_key())
return MCPTokenCipher(MCPTokenCipherConfig(keys=(raw,)))
def _make_store(backend) -> MCPTokenStore:
return MCPTokenStore(backend, _make_cipher(), node_id="test-node")
def _seed_token(
store: MCPTokenStore,
backend,
*,
user_id: str,
server_name: str,
created: str,
) -> None:
"""Create a token via the store and backdate ``created`` for ordering."""
store.create_user_token(
user_id,
server_name,
access_token="access-secret",
refresh_token="refresh-secret",
expires_at="2026-05-04T12:00:00",
scopes="openid profile",
as_issuer="https://auth.example.com",
audience="https://mcp.example.com",
)
with backend._engine.connect() as conn:
conn.execute(
sa.text(
"UPDATE mcp_user_tokens SET created = :created "
"WHERE user_id = :uid AND server_name = :sn"
),
{"created": created, "uid": user_id, "sn": server_name},
)
conn.commit()
class TestListUserTokenMetadata:
def test_list_user_token_metadata_returns_non_secret_fields_only(self, backend) -> None:
store = _make_store(backend)
_seed_token(
store, backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
rows = store.list_user_token_metadata("u1")
assert len(rows) == 1
meta = rows[0]
# Secrets MUST be absent.
assert "access_token" not in meta
assert "refresh_token" not in meta
assert "access_token_ct" not in meta
assert "refresh_token_ct" not in meta
# Non-secret columns surface verbatim.
assert meta["user_id"] == "u1"
assert meta["server_name"] == "srv-a"
assert meta["scopes"] == "openid profile"
assert meta["as_issuer"] == "https://auth.example.com"
assert meta["audience"] == "https://mcp.example.com"
assert meta["expires_at"] == "2026-05-04T12:00:00"
assert meta["created"] == "2026-05-01T00:00:00"
assert meta["last_refreshed"] is None
def test_list_user_token_metadata_empty(self, backend) -> None:
store = _make_store(backend)
assert store.list_user_token_metadata("nobody") == []
def test_list_user_token_metadata_preserves_creation_order(self, backend) -> None:
store = _make_store(backend)
_seed_token(
store, backend, user_id="u1", server_name="srv-c", created="2026-05-03T00:00:00"
)
_seed_token(
store, backend, user_id="u1", server_name="srv-a", created="2026-05-01T00:00:00"
)
_seed_token(
store, backend, user_id="u1", server_name="srv-b", created="2026-05-02T00:00:00"
)
rows = store.list_user_token_metadata("u1")
assert [r["server_name"] for r in rows] == ["srv-a", "srv-b", "srv-c"]
assert [r["created"] for r in rows] == [
"2026-05-01T00:00:00",
"2026-05-02T00:00:00",
"2026-05-03T00:00:00",
]
File diff suppressed because it is too large Load Diff
+972
View File
@@ -0,0 +1,972 @@
"""Tests for the per-(user, server) MCP session pool.
Covers Phase 5 of the OAuth-MCP rollout: pool data structures,
``_ensure_pool_entry`` lazy allocation, ``_connect_one_pool`` plumbing,
the dispatch state machine in ``_dispatch_pool``, idle / LRU eviction,
failure classification, and ``user_id`` thread-through.
The static path (``auth_type {none, static}``) MUST stay
byte-identical see ``test_mcp_client.py``'s
``test_reconnect_preserves_static_state_identity``.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import threading
import time
from contextlib import AsyncExitStack
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import MCPClientManager, PoolEntryState
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Fixtures and helpers
# ---------------------------------------------------------------------------
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
"""A fresh SQLite backend per test (not the shared singleton)."""
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
storage: SQLiteBackend,
*,
name: str = "pool-srv",
server_id: str = "srv-pool",
url: str = "https://mcp.example.com/sse",
) -> None:
storage.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url=url,
auth_type="oauth_user",
oauth_client_id="client-abc",
oauth_scopes="openid",
oauth_audience=url,
)
def _seed_user_token(
storage: SQLiteBackend,
cipher: Any,
*,
user_id: str = "user-1",
server_name: str = "pool-srv",
expires_in_seconds: int = 3600,
access_token: str = "access-aaa",
) -> None:
expires_at = (datetime.now(UTC) + timedelta(seconds=expires_in_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
store = MCPTokenStore(storage, cipher, node_id="test")
store.create_user_token(
user_id,
server_name,
access_token=access_token,
refresh_token="refresh-rrr",
expires_at=expires_at,
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
def _make_app_state(storage: SQLiteBackend, *, cipher: Any) -> SimpleNamespace:
return SimpleNamespace(
auth_storage=storage,
mcp_token_store=MCPTokenStore(storage, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
@pytest.fixture
def running_loop_mgr():
"""Background-loop fixture matching the static-path test convention.
Tests that need a wired-up app_state assign it via ``mgr.set_app_state``.
"""
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="mcp-pool-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
# Drain the eviction task before stopping the loop so its log/stream
# handlers don't fire after pytest has torn its handlers down. Mirrors
# the production ``shutdown()`` shape.
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _run_on_loop(loop: asyncio.AbstractEventLoop, coro: Any) -> Any:
"""Submit *coro* to *loop*, wait for the result with a 5s timeout."""
fut = asyncio.run_coroutine_threadsafe(coro, loop)
return fut.result(timeout=5)
# ---------------------------------------------------------------------------
# Pool data structures
# ---------------------------------------------------------------------------
class TestPoolDataStructures:
"""``_user_pool_entries``, ``_user_pool_locks``, eviction-task state."""
def test_pool_state_starts_empty(self) -> None:
mgr = MCPClientManager({})
assert mgr._user_pool_entries == {}
assert mgr._user_pool_last_used == {}
assert mgr._user_pool_locks == {}
assert mgr._user_pool_eviction_task is None
def test_set_app_state_persists(self) -> None:
mgr = MCPClientManager({})
sentinel = SimpleNamespace(token_store=object())
mgr.set_app_state(sentinel)
assert mgr._app_state is sentinel
def test_ensure_pool_entry_allocates_lock_on_loop(self, running_loop_mgr) -> None:
"""``asyncio.Lock`` MUST be created on the mcp-loop (RFC §2.0 #2)."""
mgr, loop, _thread = running_loop_mgr
key = ("user-A", "pool-srv")
entry = _run_on_loop(loop, mgr._ensure_pool_entry(key))
assert isinstance(entry, PoolEntryState)
assert entry.key == key
assert isinstance(entry.open_lock, asyncio.Lock)
# Calling again returns the same entry / lock object.
entry2 = _run_on_loop(loop, mgr._ensure_pool_entry(key))
assert entry2 is entry
assert entry2.open_lock is entry.open_lock
# ---------------------------------------------------------------------------
# Lazy connect (`_connect_one_pool`)
# ---------------------------------------------------------------------------
class _AsyncCM:
"""Awaitable async context manager that returns ``value`` from __aenter__."""
def __init__(self, value: Any) -> None:
self._value = value
async def __aenter__(self) -> Any:
return self._value
async def __aexit__(self, *exc: Any) -> bool:
return False
class TestLazyConnect:
def test_connect_pool_injects_authorization_header(self, running_loop_mgr) -> None:
from unittest.mock import patch
mgr, loop, _ = running_loop_mgr
observed_kwargs: dict[str, Any] = {}
async def _probe(*_args: Any, **_kwargs: Any) -> None:
return None
fake_session = MagicMock()
fake_session.initialize = AsyncMock(return_value=None)
# Phase 7b: ``_connect_one_pool`` discovers tools, resources,
# and prompts after ``initialize()`` returns (resources/prompts
# capability-gated). The capability stub returns a tools-only
# advertisement so the test can keep its narrow focus on the
# bearer-injection contract; resources/prompts paths are
# exercised by the real-transport tests in
# ``tests/test_mcp_user_catalog.py``.
fake_caps = MagicMock()
fake_caps.resources = None
fake_caps.prompts = None
fake_session.get_server_capabilities = MagicMock(return_value=fake_caps)
fake_session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
def _stream_factory(*, url: str, headers: dict[str, str]) -> _AsyncCM:
observed_kwargs["url"] = url
observed_kwargs["headers"] = dict(headers)
return _AsyncCM((AsyncMock(), AsyncMock(), lambda: None))
with (
patch("turnstone.core.mcp_client.streamablehttp_client", side_effect=_stream_factory),
patch.object(mgr, "_tcp_probe", side_effect=_probe),
patch("turnstone.core.mcp_client.ClientSession", return_value=_AsyncCM(fake_session)),
):
cfg = {
"type": "streamable-http",
"url": "https://mcp.example.com/sse",
"headers": {},
}
entry = _run_on_loop(
loop,
mgr._connect_one_pool(("user-1", "pool-srv"), cfg, "access-aaa"),
)
assert entry.session is fake_session
assert observed_kwargs["headers"]["Authorization"] == "Bearer access-aaa"
def test_connect_pool_rejects_non_http_transport(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
cfg = {"type": "stdio", "command": "echo"}
with pytest.raises(RuntimeError, match="streamable-http"):
_run_on_loop(
loop,
mgr._connect_one_pool(("user-1", "pool-srv"), cfg, "access-aaa"),
)
def test_pool_path_does_not_touch_static_servers(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
# Pre-seed a static-path entry so accidental writes are observable.
from turnstone.core.mcp_client import StaticServerState
sentinel = StaticServerState(name="static-srv", session=MagicMock())
mgr._static_servers["static-srv"] = sentinel
async def _seed_pool() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = MagicMock()
entry.last_used = time.monotonic()
_run_on_loop(loop, _seed_pool())
# Pool side has its own state; the static dict is untouched.
assert mgr._static_servers["static-srv"] is sentinel
assert mgr._user_pool_entries[("user-1", "pool-srv")].session is not None
# ---------------------------------------------------------------------------
# Eviction
# ---------------------------------------------------------------------------
class TestEviction:
def test_idle_eviction_closes_stale_entries(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0 # everything is stale
async def _seed() -> list[PoolEntryState]:
entries = []
for i in range(3):
entry = await mgr._ensure_pool_entry((f"u{i}", "pool-srv"))
entry.session = MagicMock()
entries.append(entry)
return entries
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
assert mgr._user_pool_entries == {}
def test_eviction_skips_locked_entries(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0
async def _seed_and_lock() -> tuple[asyncio.Lock, asyncio.Event]:
entry = await mgr._ensure_pool_entry(("u-busy", "pool-srv"))
entry.session = MagicMock()
held = asyncio.Event()
async def _hold() -> None:
async with entry.open_lock:
held.set()
await asyncio.sleep(0.5)
asyncio.create_task(_hold())
await held.wait()
return entry.open_lock, held
_run_on_loop(loop, _seed_and_lock())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
# Entry survives because eviction skipped the locked key.
assert ("u-busy", "pool-srv") in mgr._user_pool_entries
def test_lru_cap_evicts_oldest(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 999_999.0 # TTL effectively disabled
mgr._user_pool_lru_max = 2
async def _seed() -> None:
base = time.monotonic()
for i in range(5):
key = (f"u{i}", "pool-srv")
entry = await mgr._ensure_pool_entry(key)
entry.session = MagicMock()
# Recent timestamps so TTL doesn't fire — only LRU should.
entry.last_used = base + i
mgr._user_pool_last_used[key] = base + i
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
assert len(mgr._user_pool_entries) <= 2
# The two newest survive (u3, u4).
assert ("u4", "pool-srv") in mgr._user_pool_entries
assert ("u3", "pool-srv") in mgr._user_pool_entries
def test_eviction_resilient_to_close_errors(self, running_loop_mgr) -> None:
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 0.0
broken_stack = MagicMock(spec=AsyncExitStack)
broken_stack.aclose = AsyncMock(side_effect=RuntimeError("close failed"))
async def _seed() -> None:
for i in range(2):
entry = await mgr._ensure_pool_entry((f"u{i}", "pool-srv"))
entry.session = MagicMock()
entry.stack = broken_stack
_run_on_loop(loop, _seed())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
# Eviction must not raise even if close fails.
_run_on_loop(loop, _evict())
# All entries removed from the dict regardless.
assert mgr._user_pool_entries == {}
# ---------------------------------------------------------------------------
# Dispatch state machine
# ---------------------------------------------------------------------------
class TestDispatchStateMachine:
"""One row per state in the §1.5 / RFC §6 state machine."""
def _wire_pool(
self, mgr: MCPClientManager, storage: SQLiteBackend, cipher: Any
) -> SimpleNamespace:
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
return state
def test_no_token_emits_consent_required(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
self._wire_pool(mgr, storage, cipher)
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
assert payload["error"]["server"] == "pool-srv"
def test_decrypt_failure_does_not_emit_consent(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
from turnstone.core.mcp_crypto import MCPTokenDecryptError
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher)
state = self._wire_pool(mgr, storage, cipher)
def _raise(*args, **kwargs):
raise MCPTokenDecryptError(
"no installed key can decrypt",
key_fingerprints_attempted=("aabbccdd",),
)
state.mcp_token_store.get_user_token = _raise
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_token_undecryptable_key_unknown"
# Operator fingerprints stay server-side (audit log + structured log);
# the agent-facing payload must NOT carry them onward to the LLM
# provider.
assert "key_fingerprints_attempted" not in payload["error"]
def test_refresh_failure_emits_consent(self, running_loop_mgr, storage: SQLiteBackend) -> None:
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
# Seed an expired token with no refresh — the classified getter
# treats this as "refresh_failed" (deletes the row, returns the
# tagged result).
_seed_user_token(storage, cipher, expires_in_seconds=-1000)
state = self._wire_pool(mgr, storage, cipher)
# Drop the refresh token to force the no-refresh-token branch.
state.mcp_token_store.delete_user_token("user-1", "pool-srv")
state.mcp_token_store.create_user_token(
"user-1",
"pool-srv",
access_token="access-aaa",
refresh_token=None,
expires_at=(datetime.now(UTC) - timedelta(seconds=1000)).strftime("%Y-%m-%dT%H:%M:%S"),
scopes="openid",
as_issuer="https://as.example.com",
audience="https://mcp.example.com",
)
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_consent_required"
def test_token_present_dispatches_to_session(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher, expires_in_seconds=3600)
self._wire_pool(mgr, storage, cipher)
# Pre-seed a connected pool entry so dispatch never touches the
# SDK or the network.
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "tool-result"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
async def _seed_entry() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = fake_session
_run_on_loop(loop, _seed_entry())
result = mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{"q": "hi"},
user_id="user-1",
timeout=5,
)
assert result == "tool-result"
# ---------------------------------------------------------------------------
# Failure classification
# ---------------------------------------------------------------------------
class TestClassifyFailure:
def test_transport_failure_classified_as_transport(self) -> None:
mgr = MCPClientManager({})
for exc in (
BrokenPipeError(),
ConnectionResetError(),
EOFError(),
TimeoutError("net"),
):
assert mgr._classify_failure(exc) == "transport"
def test_protocol_error_classified_as_protocol(self) -> None:
from mcp import McpError
from mcp.types import ErrorData
mgr = MCPClientManager({})
err = McpError(ErrorData(code=-32600, message="bad request"))
assert mgr._classify_failure(err) == "protocol"
def test_other_classified_as_other(self) -> None:
mgr = MCPClientManager({})
assert mgr._classify_failure(ValueError("nope")) == "other"
def test_http_401_classified_as_auth_401(self) -> None:
"""Defense-in-depth: ``HTTPStatusError`` classification still works
even though Phase 6 normally consults the carrier instead.
Phase 6 split ``"auth"`` into ``"auth_401"`` / ``"auth_403"``
so the dispatcher can refresh-and-retry only on 401.
"""
import httpx
mgr = MCPClientManager({})
req = httpx.Request("POST", "https://mcp.example.com/sse")
resp = httpx.Response(401, request=req)
exc = httpx.HTTPStatusError("unauthorized", request=req, response=resp)
assert mgr._classify_failure(exc) == "auth_401"
def test_http_403_classified_as_auth_403(self) -> None:
import httpx
mgr = MCPClientManager({})
req = httpx.Request("POST", "https://mcp.example.com/sse")
resp = httpx.Response(403, request=req)
exc = httpx.HTTPStatusError("forbidden", request=req, response=resp)
assert mgr._classify_failure(exc) == "auth_403"
def test_http_500_not_classified_as_auth(self) -> None:
import httpx
mgr = MCPClientManager({})
req = httpx.Request("POST", "https://mcp.example.com/sse")
resp = httpx.Response(500, request=req)
exc = httpx.HTTPStatusError("server", request=req, response=resp)
# 5xx is not auth — falls through to "other".
assert mgr._classify_failure(exc) == "other"
# ---------------------------------------------------------------------------
# Wired-failure paths in _dispatch_pool
# ---------------------------------------------------------------------------
class TestDispatchFailureWiring:
"""``_classify_failure`` is consulted in production, not just tests."""
def _wire_pool(
self, mgr: MCPClientManager, storage: SQLiteBackend, cipher: Any
) -> SimpleNamespace:
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
return state
def _seed_connected_session(
self, mgr: MCPClientManager, loop: asyncio.AbstractEventLoop, exc: BaseException
) -> None:
async def _seed() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
sess = MagicMock()
async def _raise(*_args: Any, **_kwargs: Any) -> Any:
raise exc
sess.call_tool = _raise
entry.session = sess
_run_on_loop(loop, _seed())
def test_dispatch_pool_transport_failure_trips_breaker(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher)
self._wire_pool(mgr, storage, cipher)
self._seed_connected_session(mgr, loop, BrokenPipeError("dead"))
with pytest.raises(BrokenPipeError):
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
# Transport failure ticks the breaker.
assert mgr._consecutive_failures.get("pool-srv", 0) == 1
# ---------------------------------------------------------------------------
# HTTPS enforcement (sec-1)
# ---------------------------------------------------------------------------
class TestHttpsEnforcement:
def test_pool_rejects_http_url_for_oauth_user(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
mgr, _loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://insecure.example.com/sse")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
with pytest.raises(RuntimeError) as exc_info:
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
payload = json.loads(str(exc_info.value))
assert payload["error"]["code"] == "mcp_oauth_url_insecure"
assert payload["error"]["server"] == "pool-srv"
def test_pool_accepts_loopback_http(self, running_loop_mgr, storage: SQLiteBackend) -> None:
"""``http://127.0.0.1`` and ``http://localhost`` should not be blocked."""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv", url="http://127.0.0.1:8000/sse")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
# Pre-seed a connected pool entry so dispatch succeeds without
# touching the network.
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "ok"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
async def _seed_entry() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = fake_session
_run_on_loop(loop, _seed_entry())
result = mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
# Loopback URL not rejected — dispatch reaches the (fake) session.
assert result == "ok"
def test_validate_oauth_user_url_helper(self) -> None:
from turnstone.core.mcp_client import _validate_oauth_user_url
# Acceptable: https + the exact loopback hostnames.
_validate_oauth_user_url("https://mcp.example.com/sse")
_validate_oauth_user_url("http://localhost/sse")
_validate_oauth_user_url("http://127.0.0.1:9000/sse")
_validate_oauth_user_url("http://[::1]/sse")
# Rejected: non-https + non-loopback. The ``*.localhost`` suffix
# bypass is intentionally NOT honored (RFC 6761 localhost-zone
# resolution is configuration-dependent — custom resolvers,
# /etc/hosts, Docker overlays may map ``foo.localhost`` to
# non-loopback IPs).
for bad in (
"http://mcp.example.com/sse",
"http://app.localhost/sse",
"ws://mcp.example.com/sse",
"ftp://mcp.example.com/sse",
"//mcp.example.com/sse",
):
with pytest.raises(ValueError, match="https://"):
_validate_oauth_user_url(bad)
# ---------------------------------------------------------------------------
# _resolve_pool_target parser (q-9)
# ---------------------------------------------------------------------------
class TestResolvePoolTarget:
def _make_mgr_with_oauth_server(
self, storage: SQLiteBackend, *, name: str = "pool-srv"
) -> MCPClientManager:
_seed_oauth_server(storage, name=name)
mgr = MCPClientManager({})
mgr.set_storage(storage)
return mgr
def test_malformed_prefix(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
# Wrong prefix.
assert mgr._resolve_pool_target("xyz__pool-srv__t", None, None) is None
def test_too_few_separators(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
# mcp__server with no original_name segment.
assert mgr._resolve_pool_target("mcp__pool-srv", None, None) is None
def test_empty_server_segment(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
# mcp____tool — server segment is empty.
assert mgr._resolve_pool_target("mcp____tool", None, None) is None
def test_original_with_double_underscore_round_trips(self, storage: SQLiteBackend) -> None:
mgr = self._make_mgr_with_oauth_server(storage)
target = mgr._resolve_pool_target("mcp__pool-srv__do__thing", None, None)
assert target is not None
assert target[0] == "pool-srv"
# Original-name keeps its embedded ``__``.
assert target[1] == "do__thing"
# ---------------------------------------------------------------------------
# LRU + lock interlock (q-7)
# ---------------------------------------------------------------------------
class TestLruInterlock:
def test_lru_cap_skips_locked_oldest(self, running_loop_mgr) -> None:
"""LRU eviction must skip a locked entry the same way TTL does."""
mgr, loop, _ = running_loop_mgr
mgr._user_pool_idle_ttl_s = 999_999.0 # disable TTL
mgr._user_pool_lru_max = 2
async def _seed_and_lock_oldest() -> tuple[asyncio.Lock, asyncio.Event]:
base = time.monotonic()
for i in range(3):
key = (f"u{i}", "pool-srv")
entry = await mgr._ensure_pool_entry(key)
entry.session = MagicMock()
# Older index ⇒ older timestamp.
entry.last_used = base + i
mgr._user_pool_last_used[key] = base + i
# Lock the oldest (u0) so eviction must skip it and pick a younger one.
oldest = mgr._user_pool_entries[("u0", "pool-srv")]
held = asyncio.Event()
async def _hold() -> None:
async with oldest.open_lock:
held.set()
await asyncio.sleep(0.5)
asyncio.create_task(_hold())
await held.wait()
return oldest.open_lock, held
_run_on_loop(loop, _seed_and_lock_oldest())
async def _evict() -> None:
await mgr._evict_idle_pool_entries()
_run_on_loop(loop, _evict())
# Locked u0 must survive.
assert ("u0", "pool-srv") in mgr._user_pool_entries
# The oldest unlocked entry (u1) was evicted to bring count down to cap.
assert ("u1", "pool-srv") not in mgr._user_pool_entries
# ---------------------------------------------------------------------------
# Concurrent dispatch on shared session (M4 / perf-1)
# ---------------------------------------------------------------------------
class TestConcurrentDispatch:
def test_pool_concurrent_dispatch_to_same_user_server_is_serialized(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
"""Phase 6: two tool calls on the SAME (user, server) MUST serialize
on ``open_lock`` so the auth-introspection carrier never crosses
between concurrent dispatches.
Phase 5 perf-1 released ``open_lock`` before ``call_tool`` so two
concurrent same-key calls multiplexed on a shared
``ClientSession``. Phase 6 reverts that for the auth-aware path
because the per-dispatch ``_AuthCapture`` is keyed off the
``httpx.AsyncClient`` event hook releasing the lock would let
a concurrent dispatch overwrite the carrier mid-flight,
attributing one caller's 401 to another (a security bug).
Verified by reverting ``_dispatch_pool_with_entry`` to the
Phase 5 shape (release ``open_lock`` before ``call_tool``
i.e. move the ``in_flight += 1`` / ``call_tool`` / decrement
block out of the ``async with`` body) and confirming this test
observes ``max_concurrency == 2``.
"""
mgr, loop, _ = running_loop_mgr
cipher = make_mcp_token_cipher()
_seed_oauth_server(storage, name="pool-srv")
_seed_user_token(storage, cipher)
mgr.set_storage(storage)
state = _make_app_state(storage, cipher=cipher)
mgr.set_app_state(state)
observed_max_concurrency = 0
in_flight = 0
in_flight_lock = threading.Lock()
async def _call_tool(name, args):
nonlocal observed_max_concurrency, in_flight
with in_flight_lock:
in_flight += 1
observed_max_concurrency = max(observed_max_concurrency, in_flight)
try:
# Hold a moment so concurrent calls would overlap if
# they weren't serialized on ``open_lock``.
await asyncio.sleep(0.1)
content = MagicMock()
content.text = "ok"
res = MagicMock()
res.content = [content]
res.isError = False
return res
finally:
with in_flight_lock:
in_flight -= 1
fake_session = MagicMock()
fake_session.call_tool = _call_tool
async def _seed_entry() -> None:
entry = await mgr._ensure_pool_entry(("user-1", "pool-srv"))
entry.session = fake_session
_run_on_loop(loop, _seed_entry())
results: list[str] = []
errors: list[Exception] = []
def _dispatch() -> None:
try:
results.append(
mgr.call_tool_sync(
"mcp__pool-srv__do_thing",
{},
user_id="user-1",
timeout=5,
)
)
except Exception as exc: # pragma: no cover — diagnostic only
errors.append(exc)
t1 = threading.Thread(target=_dispatch)
t2 = threading.Thread(target=_dispatch)
t1.start()
t2.start()
t1.join(timeout=5)
t2.join(timeout=5)
assert errors == []
assert results == ["ok", "ok"]
# ``open_lock`` held across ``call_tool`` — the second dispatch
# waits for the first to release before entering call_tool.
assert observed_max_concurrency == 1
# ---------------------------------------------------------------------------
# user_id thread-through (signature)
# ---------------------------------------------------------------------------
class TestUserIdThreadThrough:
def test_default_user_id_takes_static_path(self, running_loop_mgr) -> None:
"""``user_id=None`` must leave the static-path call byte-identical."""
mgr, _loop, _ = running_loop_mgr
# Static-path tool registered the standard way.
mgr._tool_map["mcp__static__t"] = ("static-srv", "t")
from turnstone.core.mcp_client import StaticServerState
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "static-output"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
mgr._static_servers["static-srv"] = StaticServerState(
name="static-srv", session=fake_session
)
# No user_id, no app_state — pool branch is skipped entirely.
result = mgr.call_tool_sync("mcp__static__t", {"q": "hi"}, user_id=None, timeout=5)
assert result == "static-output"
def test_user_id_with_static_path_does_not_use_pool(
self, running_loop_mgr, storage: SQLiteBackend
) -> None:
"""Caller passes user_id but the resolved server is static — pool
branch must not run because ``_lookup_server_row`` reports
``auth_type != 'oauth_user'``."""
mgr, _loop, _ = running_loop_mgr
storage.create_mcp_server(
server_id="srv-static",
name="static-srv",
transport="stdio",
url="",
command="echo",
auth_type="static",
)
mgr.set_storage(storage)
mgr.set_app_state(SimpleNamespace())
mgr._tool_map["mcp__static-srv__t"] = ("static-srv", "t")
from turnstone.core.mcp_client import StaticServerState
fake_session = MagicMock()
async def _call_tool(name, args):
content = MagicMock()
content.text = "static-output"
res = MagicMock()
res.content = [content]
res.isError = False
return res
fake_session.call_tool = _call_tool
mgr._static_servers["static-srv"] = StaticServerState(
name="static-srv", session=fake_session
)
result = mgr.call_tool_sync(
"mcp__static-srv__t",
{"q": "hi"},
user_id="user-1",
timeout=5,
)
assert result == "static-output"
# No pool entries were created.
assert mgr._user_pool_entries == {}
+265
View File
@@ -1,6 +1,9 @@
"""Tests for turnstone.core.memory_relevance — scoring, formatting, context extraction."""
from unittest.mock import patch
from turnstone.core.memory_relevance import (
MemoryConfig,
build_memory_context,
extract_recent_context,
score_memories,
@@ -192,3 +195,265 @@ class TestExtractRecentContext:
def test_empty_messages(self):
assert extract_recent_context([]) == ""
# ---------------------------------------------------------------------------
# Composition candidate-selection (_init_system_messages)
# ---------------------------------------------------------------------------
def _make_mem(name: str, content: str = "", memory_id: str | None = None) -> dict[str, str]:
return {
"name": name,
"memory_id": memory_id or f"mid_{name}",
"type": "project",
"scope": "global",
"scope_id": "",
"description": "",
"content": content or name,
"updated": "2024-01-01T00:00:00",
}
def _make_session(fetch_limit: int = 5, relevance_k: int = 3, **kwargs: object):
"""Composition tests need a real ChatSession (constructor calls
``_init_system_messages`` once, unpatched, before the test gets a chance
to install patches). ``tmp_db`` initializes the storage singleton that
constructor needs; tests then patch the visibility helpers and call
``_init_system_messages`` a second time to exercise the new logic.
"""
from tests._helpers import make_chat_session
return make_chat_session(
memory_config=MemoryConfig(fetch_limit=fetch_limit, relevance_k=relevance_k),
**kwargs,
)
class TestCompositionCandidateSelection:
"""Verify the query-aware candidate set in _init_system_messages."""
def test_recency_ceiling_regression(self, tmp_db):
"""Old relevant memory not in recency top-N still injected via search path."""
session = _make_session(fetch_limit=5, relevance_k=3)
session.messages = [{"role": "user", "content": "postgres database configuration"}]
old_mem = _make_mem(
"ancient_db_config",
content="postgres database configuration connection host port",
memory_id="m_old",
)
# Recency top-5 do not include old_mem
recent = [_make_mem(f"recent_{i}", memory_id=f"mr{i}") for i in range(5)]
with (
patch.object(session, "_search_visible_memories", return_value=[old_mem]),
patch.object(session, "_list_visible_memories", return_value=recent),
):
session._init_system_messages()
joined = "\n".join(m["content"] for m in session.system_messages if m["role"] == "system")
# With the fix, old_mem enters the candidate pool via search and wins BM25
assert "ancient_db_config" in joined
def test_empty_query_falls_back_to_recency(self, tmp_db):
"""No user messages → empty context → recency path, search never called."""
session = _make_session()
session.messages = [] # extract_recent_context returns ""
recency = [_make_mem("note_alpha"), _make_mem("note_beta")]
with (
patch.object(session, "_list_visible_memories", return_value=recency),
patch.object(session, "_search_visible_memories") as search_mock,
):
session._init_system_messages()
search_mock.assert_not_called()
joined = "\n".join(m["content"] for m in session.system_messages if m["role"] == "system")
assert "note_alpha" in joined
def test_sparse_match_union_fills_candidate_pool(self, tmp_db):
"""Search returning < fetch_limit results unions with recency fillers."""
session = _make_session(fetch_limit=5, relevance_k=4)
session.messages = [{"role": "user", "content": "unique_term xyzzy"}]
hit_a = _make_mem("hit_alpha", content="unique_term xyzzy alpha", memory_id="m_ha")
hit_b = _make_mem("hit_beta", content="unique_term xyzzy beta", memory_id="m_hb")
search_hits = [hit_a, hit_b] # 2 < fetch_limit=5 → triggers union
# Recency overlaps on hit_a/hit_b and adds 3 fillers
filler = [_make_mem(f"filler_{i}", memory_id=f"mf{i}") for i in range(3)]
recency = [hit_a, hit_b] + filler
with (
patch.object(session, "_search_visible_memories", return_value=search_hits),
patch.object(session, "_list_visible_memories", return_value=recency),
):
session._init_system_messages()
joined = "\n".join(m["content"] for m in session.system_messages if m["role"] == "system")
# Both hits match "unique_term xyzzy" well → appear after BM25 ranking
assert "hit_alpha" in joined
assert "hit_beta" in joined
def test_recency_preserved_when_search_returns_noise_above_relevance_k(self, tmp_db):
"""Pool guarantee: recency-50 always reaches BM25, even when search
returns enough noise hits to clear ``relevance_k``.
Closes the narrow regression vs. the original bug without the
``fetch_limit`` threshold, a stopword-dominated cap-search that
returned >= relevance_k irrelevant hits would short-circuit and
evict the recency-only memory the bug had been surfacing.
"""
session = _make_session(fetch_limit=10, relevance_k=3)
session.messages = [{"role": "user", "content": "configure host"}]
# Search returns relevance_k=3 noise hits — enough to skip recency
# under the OLD threshold, not enough to fill fetch_limit=10.
noise = [
_make_mem(f"noise_{i}", content="generic content", memory_id=f"mn{i}") for i in range(3)
]
# The memory the user actually wants — distinctive, in recency,
# but its content doesn't share any token with the noise hits.
wanted = _make_mem(
"host_config_v2",
content="host=localhost port=5432 db=production",
memory_id="m_wanted",
)
recency = [wanted] + [_make_mem(f"recent_{i}", memory_id=f"mr{i}") for i in range(5)]
with (
patch.object(session, "_search_visible_memories", return_value=noise),
patch.object(session, "_list_visible_memories", return_value=recency),
):
session._init_system_messages()
joined = "\n".join(m["content"] for m in session.system_messages if m["role"] == "system")
# ``wanted`` reached BM25 via the union and matched "host" → injected.
assert "host_config_v2" in joined
def test_recency_tail_preserved_when_search_adds_distinct_hits(self, tmp_db):
"""SUPERSET invariant: every recency item is in the candidate pool
when search adds hits, even if the resulting union exceeds
fetch_limit. Truncating the union at fetch_limit (the prior
behavior) evicted the recency tail which is exactly where
ancient-but-recently-touched memories live, the recall this PR
sets out to improve.
"""
session = _make_session(fetch_limit=10, relevance_k=3)
session.messages = [{"role": "user", "content": "alpha"}]
# 5 search hits, none of which appear in recency.
search_hits = [
_make_mem(f"search_{i}", content="alpha", memory_id=f"ms{i}") for i in range(5)
]
# 10 recency items; without the union uncap, the 5 oldest of these
# would be displaced by the 5 search hits.
recency = [_make_mem(f"recency_{i}", memory_id=f"mr{i}") for i in range(10)]
with (
patch.object(session, "_search_visible_memories", return_value=search_hits),
patch.object(session, "_list_visible_memories", return_value=recency),
):
candidates, source = session._select_memory_candidates("alpha")
candidate_ids = {c["memory_id"] for c in candidates}
# Pool is search_hits recency — 15 items, no truncation.
assert len(candidates) == 15
assert source == "union"
# Every recency item present (no tail eviction).
for i in range(10):
assert f"mr{i}" in candidate_ids, f"recency item {i} evicted"
# And every search hit is also in the pool.
for i in range(5):
assert f"ms{i}" in candidate_ids, f"search hit {i} missing"
def test_coord_scope_isolated_visibility(self, tmp_db):
"""Coord composition queries the coord scope alone, never the
global/workstream/user union."""
from turnstone.core.workstream import WorkstreamKind
coord = _make_session(
fetch_limit=5,
relevance_k=3,
ws_id="coord-1",
user_id="user-1",
kind=WorkstreamKind.COORDINATOR,
)
scopes = coord._visible_scopes()
assert scopes == [("coordinator", "coord-1")]
# And: search uses those same scopes (no global/user fan-in)
coord.messages = [{"role": "user", "content": "anything"}]
with patch(
"turnstone.core.session.search_visible_structured_memories",
return_value=[],
) as search_mock:
coord._search_visible_memories("anything", limit=5)
search_mock.assert_called_once()
# Second positional arg is the scopes list
assert search_mock.call_args.args[1] == [("coordinator", "coord-1")]
class TestMemorySearchToolExecution:
"""End-to-end test of ``memory(action='search')`` through _exec_memory.
Drives the actual tool dispatch (not just the storage facade) so the
OR-of-terms fix and the coalesced ``memory.search`` log get exercised
together.
"""
def test_search_action_returns_or_of_terms_results(self, tmp_db):
"""Multi-word query returns rows where ANY term matches — not all."""
from turnstone.core.memory import save_structured_memory
save_structured_memory("postgres_notes", "host=localhost port=5432")
save_structured_memory("redis_notes", "host=redis port=6379")
save_structured_memory("unrelated", "completely different")
session = _make_session()
item = session._prepare_memory(
"call-1",
{"action": "search", "query": "postgres no_such_word_a no_such_word_b"},
)
# Sanity: prepare returned a search-ready dispatch (not an error item)
assert item.get("action") == "search"
call_id, msg = session._exec_memory(item)
assert call_id == "call-1"
assert "postgres_notes" in msg
# Other memories don't match any query term
assert "unrelated" not in msg
class TestPerTurnSearchCache:
"""The per-turn cache spares redundant SQL across mid-turn rebuilds."""
def test_repeated_search_in_same_turn_hits_cache(self, tmp_db):
from turnstone.core.memory import save_structured_memory
save_structured_memory("hello_mem", "alpha beta gamma")
session = _make_session()
with patch(
"turnstone.core.session.search_visible_structured_memories",
return_value=[],
) as backend_mock:
session._search_visible_memories("alpha beta", limit=5)
session._search_visible_memories("alpha beta", limit=5)
session._search_visible_memories("alpha beta", limit=5)
# 3 calls but only 1 backend hit — cache absorbed the rest
assert backend_mock.call_count == 1
def test_user_turn_invalidates_cache(self, tmp_db):
from turnstone.core.memory import save_structured_memory
save_structured_memory("hello_mem", "alpha")
session = _make_session()
with patch(
"turnstone.core.session.search_visible_structured_memories",
return_value=[],
) as backend_mock:
session._search_visible_memories("alpha", limit=5)
session._invalidate_memory_cache() # simulates new user turn
session._search_visible_memories("alpha", limit=5)
assert backend_mock.call_count == 2
+277
View File
@@ -4,12 +4,16 @@ from turnstone.core.metacognition import (
NUDGE_COMPLETION,
NUDGE_CORRECTION,
NUDGE_DENIAL,
NUDGE_IDLE_CHILDREN_DISPLAY_CAP,
NUDGE_IDLE_CHILDREN_WAIT_CAP,
NUDGE_REPEAT,
NUDGE_RESUME,
NUDGE_START,
NUDGE_TOOL_ERROR,
RepeatDetector,
detect_completion,
detect_correction,
format_idle_children_nudge,
format_nudge,
should_nudge,
)
@@ -308,3 +312,276 @@ class TestRepeatNudge:
"""Repeat nudge should fire even with zero memories."""
state: dict[str, float] = {}
assert should_nudge("repeat", state, message_count=5, memory_count=0) is True
class TestRepeatDetector:
"""Repeat-detection streak machine — fires only when the same signature
is recorded ``threshold`` times *consecutively* (default 3). Recording
any different signature resets the streak, so an interrupted repeat
isn't flagged as a stuck loop."""
def test_below_threshold_does_not_fire(self):
det = RepeatDetector()
assert det.record("a") is False
assert det.record("a") is False # second call still under threshold
def test_at_threshold_fires(self):
det = RepeatDetector()
det.record("a")
det.record("a")
assert det.record("a") is True
def test_continues_to_fire_past_threshold(self):
# Caller is responsible for clearing after a fire — until they do,
# subsequent identical calls keep returning True.
det = RepeatDetector()
det.record("a")
det.record("a")
assert det.record("a") is True
assert det.record("a") is True
def test_clear_resets_count(self):
det = RepeatDetector()
det.record("a")
det.record("a")
det.clear()
assert det.record("a") is False # back to 1 after clear
def test_intervening_sig_resets_streak(self):
# The streak is consecutive: recording any other sig mid-streak
# discards the in-progress count. An alternating pattern like
# [A, A, B, A, A] is two short streaks of 2, not a streak of 4.
det = RepeatDetector()
det.record("a")
det.record("a")
assert det.record("b") is False # b at count 1; a's streak is gone
assert det.record("a") is False # a starts fresh at 1
assert det.record("a") is False # a at 2
assert det.record("a") is True # a hits 3 — fresh streak completes
def test_errored_signature_counts_toward_repeat(self):
# Regression: when metacog was split out of the system message,
# the error-output skip got reintroduced and stuck-loop detection
# silently broke for tools that kept failing. Detector itself is
# signature-only — error vs. success is the caller's policy.
det = RepeatDetector()
# Caller records an errored call's sig the same as a successful one;
# the streak is what matters.
for _ in range(3):
last = det.record("bash:ls /nonexistent")
assert last is True
def test_custom_threshold(self):
det = RepeatDetector(threshold=2)
assert det.record("a") is False
assert det.record("a") is True
def test_threshold_one_fires_immediately(self):
det = RepeatDetector(threshold=1)
assert det.record("a") is True
class TestFormatIdleChildrenNudge:
"""``format_idle_children_nudge`` renders the wake-driven idle_children
body no ``<system-reminder>`` envelope (the side-channel splice
wraps it at the wire boundary).
"""
def test_empty_list_returns_empty_string(self):
# Caller short-circuits on `if not text: return` — so empty
# input MUST produce empty output, not a header-only stub.
assert format_idle_children_nudge([]) == ""
def test_single_child_renders(self):
children = [{"ws_id": "ws-abc12345", "name": "research-task", "state": "running"}]
text = format_idle_children_nudge(children)
assert "ws-abc12" in text # short-id form (8 chars)
assert "research-task" in text
assert "running" in text
assert "wait_for_workstream" in text
assert "ws-abc12345" in text # full id appears in the suggestion's ws_ids list
def test_under_display_cap_no_overflow_line(self):
children = [
{"ws_id": f"ws-{i:08d}", "name": f"task-{i}", "state": "running"} for i in range(3)
]
text = format_idle_children_nudge(children)
assert "...and" not in text
for i in range(3):
assert f"task-{i}" in text
def test_over_display_cap_renders_overflow_line(self):
n = NUDGE_IDLE_CHILDREN_DISPLAY_CAP + 4
children = [
{"ws_id": f"ws-{i:08d}", "name": f"task-{i}", "state": "thinking"} for i in range(n)
]
text = format_idle_children_nudge(children)
assert f"...and {n - NUDGE_IDLE_CHILDREN_DISPLAY_CAP} more" in text
# First N children are inline; later ones are folded into "...and N more".
for i in range(NUDGE_IDLE_CHILDREN_DISPLAY_CAP):
assert f"task-{i}" in text
for i in range(NUDGE_IDLE_CHILDREN_DISPLAY_CAP, n):
# Names beyond the display cap aren't visible; only counted.
assert f"task-{i}" not in text
def test_over_wait_cap_truncates_suggestion_ws_ids(self):
n = NUDGE_IDLE_CHILDREN_WAIT_CAP + 5
children = [
{"ws_id": f"ws-{i:08d}", "name": f"task-{i}", "state": "running"} for i in range(n)
]
text = format_idle_children_nudge(children)
# The first WAIT_CAP ids appear in the suggestion; later ones don't.
first_in_suggestion = f"ws-{NUDGE_IDLE_CHILDREN_WAIT_CAP - 1:08d}"
first_excluded = f"ws-{NUDGE_IDLE_CHILDREN_WAIT_CAP:08d}"
assert first_in_suggestion in text
assert first_excluded not in text
def test_unnamed_child_falls_back(self):
children = [{"ws_id": "ws-deadbeef", "name": "", "state": "attention"}]
text = format_idle_children_nudge(children)
assert "(unnamed)" in text
assert "attention" in text
def test_newline_in_name_does_not_forge_extra_bullet(self):
"""A workstream name with embedded ``\\n`` / ``\\t`` / ``\\r`` MUST
NOT break the bullet structure :func:`sanitize_name`'s strict
regex strips control chars (incl. TAB/LF/CR) so the name stays
on a single line under its own bullet. Without this, a
malicious child name like ``"foo\\n - ws-fake (running): bar"``
would forge a fake sibling row in the rendered list.
"""
children = [
{"ws_id": "ws-real0001", "name": "real", "state": "running"},
{
"ws_id": "ws-evil0002",
"name": "evil\n - ws-fake (running): forged",
"state": "thinking",
},
{"ws_id": "ws-real0003", "name": "tail", "state": "running"},
]
text = format_idle_children_nudge(children)
bullet_rows = [ln for ln in text.splitlines() if ln.startswith(" - ")]
assert len(bullet_rows) == 3, (
f"expected 3 bullet rows; got {len(bullet_rows)}: {bullet_rows!r}"
)
evil_row = next(row for row in bullet_rows if "ws-evil" in row)
assert "\n" not in evil_row
assert "\t" not in evil_row
assert "\r" not in evil_row
assert "evil" in evil_row
assert "ws-real" in bullet_rows[2]
assert "tail" in bullet_rows[2]
def test_missing_state_renders_question_mark(self):
children = [{"ws_id": "ws-12345678", "name": "x"}]
text = format_idle_children_nudge(children)
# Defensive default — exotic state keys / partial dicts shouldn't crash.
assert "?" in text
def test_no_system_reminder_envelope(self):
# The side-channel ``_apply_reminders_for_provider`` splice
# adds ``<system-reminder>`` at the wire boundary; the formatter
# MUST NOT wrap, or the model would see a doubled envelope.
text = format_idle_children_nudge([{"ws_id": "ws-x", "name": "y", "state": "running"}])
assert "<system-reminder>" not in text
assert "</system-reminder>" not in text
def test_format_nudge_returns_empty_for_idle_children(self):
# The static map's idle_children entry is the empty string by
# design — format_idle_children_nudge produces the real body.
assert format_nudge("idle_children") == ""
def test_should_nudge_recognises_idle_children_type(self, monkeypatch):
# Type registration in ``_NUDGE_MAP`` makes ``should_nudge``
# recognise it for cooldown gating; without the entry it would
# silently return False on every call.
state: dict[str, float] = {}
# message_count > 1 to clear the first-message gate.
assert should_nudge("idle_children", state, message_count=4, memory_count=0) is True
# Cooldown set on success → second immediate call returns False.
assert should_nudge("idle_children", state, message_count=5, memory_count=0) is False
class TestSanitizeName:
"""Strict sanitiser for single-line user-controlled name fields
(used by :func:`format_idle_children_nudge` for the workstream
``name``). Strips ASCII control chars **including** TAB/LF/CR
plus Unicode steering vectors and angle-bracket tag breakers.
"""
def test_empty_input_returns_empty(self):
from turnstone.core.metacognition import sanitize_name
assert sanitize_name("") == ""
def test_strips_tab_lf_cr(self):
"""Strict variant: TAB/LF/CR are stripped so a hostile name with
an embedded newline can't break a bullet's one-line structure.
"""
from turnstone.core.metacognition import sanitize_name
# All three become spaces (then collapsed to one inline space
# by the trailing ``strip()``-on-leading/trailing-only step
# — interior runs stay as multiple spaces, that's fine for a
# one-line name).
assert sanitize_name("a\tb") == "a b"
assert sanitize_name("a\nb") == "a b"
assert sanitize_name("a\rb") == "a b"
def test_strips_other_ascii_control_chars(self):
from turnstone.core.metacognition import sanitize_name
assert sanitize_name("a\x07b\x0bc\x0cd") == "a b c d"
assert sanitize_name("a\x7fb") == "a b"
def test_strips_angle_bracket_tag_breakers(self):
from turnstone.core.metacognition import sanitize_name
assert sanitize_name("a</thinking>b") == "a/thinkingb"
class TestSanitizePayload:
"""Permissive sanitiser used by the ``watch_triggered`` producer.
Strips ASCII control chars (except TAB/LF/CR), Unicode steering
vectors (bidi, zero-width, BOM, tag chars), and angle-bracket
tag breakers keeps everything else intact, so multi-line shell
output retains its line structure.
"""
def test_empty_input_returns_empty(self):
from turnstone.core.metacognition import sanitize_payload
assert sanitize_payload("") == ""
def test_strips_ascii_control_chars(self):
"""``\\x00``-``\\x1f`` minus TAB/LF/CR plus ``\\x7f`` (DEL) become spaces."""
from turnstone.core.metacognition import sanitize_payload
# BEL (0x07), VT (0x0b), FF (0x0c) — all in strip set.
assert sanitize_payload("a\x07b\x0bc\x0cd") == "a b c d"
# DEL (0x7f).
assert sanitize_payload("a\x7fb") == "a b"
def test_preserves_tab_lf_cr(self):
"""TAB / LF / CR are intentionally preserved so multi-line shell
output keeps its line structure when sanitised as a watch payload.
"""
from turnstone.core.metacognition import sanitize_payload
# Newlines kept; only the leading + trailing strip happens.
out = sanitize_payload("line1\nline2\n\tindented\rline3")
assert out == "line1\nline2\n\tindented\rline3"
def test_strips_bidi_and_zero_width(self):
from turnstone.core.metacognition import sanitize_payload
# U+202E RIGHT-TO-LEFT OVERRIDE; U+200B ZERO WIDTH SPACE.
assert sanitize_payload("abc") == "a b c"
def test_strips_angle_bracket_tag_breakers(self):
from turnstone.core.metacognition import sanitize_payload
# "<" / ">" go away entirely (not replaced with space) so a name
# like "</thinking>" doesn't leave a hole the model can read as
# a structural marker.
assert sanitize_payload("a</thinking>b") == "a/thinkingb"
+166
View File
@@ -0,0 +1,166 @@
"""Tests for alembic migration 049 (OAuth-MCP schema).
Drives ``command.upgrade`` from a programmatic Alembic config against
an isolated SQLite database per test, then asserts:
* the two new tables (``mcp_user_tokens``, ``mcp_oauth_pending``) exist,
* the eight new ``mcp_servers`` columns exist,
* the post-upgrade ``UPDATE mcp_servers`` normalization rewrites rows
with empty / missing headers to ``auth_type='none'`` while leaving
rows with non-empty headers at ``auth_type='static'``.
"""
from __future__ import annotations
from pathlib import Path
import sqlalchemy as sa
from alembic import command
from alembic.config import Config
_MIGRATIONS_DIR = str(
Path(__file__).resolve().parent.parent / "turnstone" / "core" / "storage" / "migrations"
)
def _alembic_cfg(db_path: Path) -> Config:
cfg = Config()
cfg.set_main_option("script_location", _MIGRATIONS_DIR)
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{db_path}")
return cfg
class TestMigration049:
def test_creates_new_tables_and_columns(self, tmp_path: Path) -> None:
db_path = tmp_path / "049.db"
cfg = _alembic_cfg(db_path)
# Walk forward through 048 first, then explicitly to 049 so we
# exercise the *upgrade* function (not just the schema's `head`).
command.upgrade(cfg, "048")
command.upgrade(cfg, "049")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
inspector = sa.inspect(engine)
tables = set(inspector.get_table_names())
assert "mcp_user_tokens" in tables
assert "mcp_oauth_pending" in tables
mcp_cols = {c["name"] for c in inspector.get_columns("mcp_servers")}
new_cols = {
"auth_type",
"oauth_client_id",
"oauth_client_secret_ct",
"oauth_scopes",
"oauth_audience",
"oauth_registration_mode",
"oauth_authorization_server_url",
"oauth_as_issuer_cached",
}
assert new_cols.issubset(mcp_cols), new_cols - mcp_cols
# Index check on mcp_oauth_pending.
indexes = {ix["name"] for ix in inspector.get_indexes("mcp_oauth_pending")}
assert "idx_mcp_pending_created" in indexes
finally:
engine.dispose()
def test_normalizes_empty_headers_to_none(self, tmp_path: Path) -> None:
"""Streamable-http rows with NULL / '' / '{}' headers become
auth_type='none'; rows with non-empty headers stay 'static'.
Stdio rows always stay 'static' regardless of headers the
column value is opaque when there is no HTTP transport."""
db_path = tmp_path / "049-norm.db"
cfg = _alembic_cfg(db_path)
# Apply everything up to 048, seed rows, then apply 049.
command.upgrade(cfg, "048")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
with engine.begin() as conn:
conn.execute(
sa.text(
"""
INSERT INTO mcp_servers (
server_id, name, transport, command, args, url,
headers, env, auto_approve, enabled, created_by,
registry_name, registry_version, registry_meta,
created, updated
) VALUES (
:sid, :name, :transport, '', '[]',
'https://x', :headers, '{}', 0, 1, '', NULL, '',
'{}', '2026-05-04T11:00:00', '2026-05-04T11:00:00'
)
"""
),
[
{
"sid": "s-empty-str",
"name": "empty-str",
"transport": "streamable-http",
"headers": "",
},
{
"sid": "s-empty-obj",
"name": "empty-obj",
"transport": "streamable-http",
"headers": "{}",
},
{
"sid": "s-with-headers",
"name": "with-headers",
"transport": "streamable-http",
"headers": '{"Authorization":"Bearer x"}',
},
# Stdio rows must keep the 'static' default, even
# though their headers are empty — auth_type is
# opaque for stdio.
{
"sid": "s-stdio-empty",
"name": "stdio-empty",
"transport": "stdio",
"headers": "{}",
},
{
"sid": "s-stdio-null",
"name": "stdio-null",
"transport": "stdio",
"headers": "",
},
],
)
command.upgrade(cfg, "049")
with engine.connect() as conn:
rows = dict(conn.execute(sa.text("SELECT name, auth_type FROM mcp_servers")).all())
assert rows["empty-str"] == "none"
assert rows["empty-obj"] == "none"
assert rows["with-headers"] == "static"
# Stdio rows must remain at the 'static' column default even
# when headers are empty — the migration only touches HTTP
# rows where auth_type is semantically meaningful.
assert rows["stdio-empty"] == "static"
assert rows["stdio-null"] == "static"
finally:
engine.dispose()
def test_full_chain_to_head(self, tmp_path: Path) -> None:
"""Sanity: running ``upgrade head`` on a fresh DB yields the
same end-state column set as ``_schema.metadata``."""
db_path = tmp_path / "049-head.db"
cfg = _alembic_cfg(db_path)
command.upgrade(cfg, "head")
engine = sa.create_engine(f"sqlite:///{db_path}")
try:
from turnstone.core.storage._schema import mcp_servers
inspector = sa.inspect(engine)
actual = {c["name"] for c in inspector.get_columns("mcp_servers")}
expected = {c.name for c in mcp_servers.columns}
assert expected.issubset(actual), expected - actual
finally:
engine.dispose()
+70
View File
@@ -212,3 +212,73 @@ class TestModelDefinitionStorage:
m = db.get_model_definition(did)
assert m is not None
assert m["temperature"] is None
def test_reasoning_flags_default(self, db: SQLiteBackend) -> None:
"""surface_persisted_reasoning defaults True; replay_reasoning_to_model defaults False."""
did = _make_id()
db.create_model_definition(definition_id=did, alias="reason-default", model="gpt-5")
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is True
assert m["replay_reasoning_to_model"] is False
def test_create_with_explicit_reasoning_flags(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(
definition_id=did,
alias="reason-explicit",
model="claude-opus-4-7",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is False
assert m["replay_reasoning_to_model"] is True
# Same values must round-trip via the alias lookup too.
m_alias = db.get_model_definition_by_alias("reason-explicit")
assert m_alias is not None
assert m_alias["surface_persisted_reasoning"] is False
assert m_alias["replay_reasoning_to_model"] is True
def test_update_surface_persisted_reasoning(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(definition_id=did, alias="upd-persist", model="gpt-5")
ok = db.update_model_definition(did, surface_persisted_reasoning=False)
assert ok is True
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is False
assert m["replay_reasoning_to_model"] is False # untouched
def test_update_replay_reasoning_to_model(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(definition_id=did, alias="upd-replay", model="gpt-5")
ok = db.update_model_definition(did, replay_reasoning_to_model=True)
assert ok is True
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is True # untouched
assert m["replay_reasoning_to_model"] is True
def test_list_returns_reasoning_flags(self, db: SQLiteBackend) -> None:
db.create_model_definition(
definition_id=_make_id(),
alias="list-a",
model="gpt-5",
surface_persisted_reasoning=True,
replay_reasoning_to_model=False,
)
db.create_model_definition(
definition_id=_make_id(),
alias="list-b",
model="claude-opus-4-7",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
models = db.list_model_definitions()
by_alias = {m["alias"]: m for m in models}
assert by_alias["list-a"]["surface_persisted_reasoning"] is True
assert by_alias["list-a"]["replay_reasoning_to_model"] is False
assert by_alias["list-b"]["surface_persisted_reasoning"] is False
assert by_alias["list-b"]["replay_reasoning_to_model"] is True
+277 -16
View File
@@ -76,6 +76,23 @@ class TestModelConfig:
assert cfg.temperature == 0.0
assert cfg.temperature is not None
def test_reasoning_flags_default(self) -> None:
cfg = ModelConfig(alias="x", base_url="x", api_key="x", model="x")
assert cfg.surface_persisted_reasoning is True
assert cfg.replay_reasoning_to_model is False
def test_reasoning_flags_set(self) -> None:
cfg = ModelConfig(
alias="x",
base_url="x",
api_key="x",
model="x",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
assert cfg.surface_persisted_reasoning is False
assert cfg.replay_reasoning_to_model is True
# ---------------------------------------------------------------------------
# ModelRegistry
@@ -314,7 +331,11 @@ class TestLoadModelRegistry:
api_key="dummy",
model="local-model",
)
assert reg.count == 2 # "openai" + "default"
# The CLI ``"default"`` shim is suppressed once ``[models.*]``
# populates configs — only the explicit alias survives.
assert reg.count == 1
assert reg.has_alias("openai")
assert not reg.has_alias("default")
assert reg.default == "openai"
_, model, _ = reg.resolve()
assert model == "gpt-4o"
@@ -545,7 +566,12 @@ class TestLoadModelRegistryWithDB:
assert cfg.source == "config"
def test_db_only_models_coexist(self) -> None:
"""DB models coexist alongside config.toml models."""
"""DB models coexist alongside config.toml models.
The CLI ``"default"`` shim is suppressed when DB / config models
already populate the registry see
``test_cli_default_shim_skipped_when_db_models_present``.
"""
storage = _MockStorage(
[
{
@@ -569,7 +595,7 @@ class TestLoadModelRegistryWithDB:
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
assert reg.has_alias("db-only")
assert reg.has_alias("config-only")
assert reg.has_alias("default")
assert not reg.has_alias("default")
assert reg.get_config("db-only").source == "db"
assert reg.get_config("config-only").source == "config"
@@ -589,10 +615,12 @@ class TestLoadModelRegistryWithDB:
}
]
)
# The CLI default shim is suppressed when the DB row populates
# configs, so only the DB-sourced alias exists here.
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
assert reg.get_config("from-db").source == "db"
assert reg.get_config("default").source == ""
assert not reg.has_alias("default")
def test_disabled_db_models_excluded(self) -> None:
"""Disabled DB models are not loaded."""
@@ -686,6 +714,53 @@ class TestLoadModelRegistryWithDB:
assert cfg.max_tokens is None
assert cfg.reasoning_effort is None
def test_db_reasoning_flags_loaded(self) -> None:
"""Per-model reasoning flags from DB are carried in ModelConfig."""
storage = _MockStorage(
[
{
"alias": "anth-thinking",
"model": "claude-opus-4-7",
"provider": "anthropic",
"base_url": "",
"api_key": "sk-anth",
"context_window": 200000,
"capabilities": "{}",
"enabled": True,
"surface_persisted_reasoning": False,
"replay_reasoning_to_model": True,
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
cfg = reg.get_config("anth-thinking")
assert cfg.surface_persisted_reasoning is False
assert cfg.replay_reasoning_to_model is True
def test_db_reasoning_flags_default_when_absent(self) -> None:
"""Pre-052 rows without the columns degrade to dataclass defaults."""
storage = _MockStorage(
[
{
"alias": "legacy-row",
"model": "gpt-5",
"provider": "openai",
"base_url": "",
"api_key": "",
"context_window": 32768,
"capabilities": "{}",
"enabled": True,
# surface_persisted_reasoning + replay_reasoning_to_model intentionally absent
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
cfg = reg.get_config("legacy-row")
assert cfg.surface_persisted_reasoning is True
assert cfg.replay_reasoning_to_model is False
def test_db_default_alias_not_clobbered(self) -> None:
"""DB model with alias='default' is not overwritten by CLI args."""
storage = _MockStorage(
@@ -770,16 +845,76 @@ class TestRegistryReload:
assert reg.has_alias("b")
assert reg.default == "b"
def test_reload_clears_clients(self) -> None:
models = {"a": ModelConfig("a", "http://x/v1", "key", "m")}
def test_reload_keeps_clients_when_connection_target_unchanged(self) -> None:
"""Selective teardown: a model edit that leaves base_url / api_key /
provider intact (e.g. admin tweaks the underlying ``model`` name or
``temperature``) keeps the cached HTTP client warm no need to
re-establish TLS+pool when the endpoint is the same."""
models = {"a": ModelConfig("a", "http://x/v1", "key", "m1", provider="openai")}
reg = ModelRegistry(models=models, default="a")
# Force client creation
reg.get_client("a")
assert "a" in reg._clients
client_before = reg._clients["a"]
provider_before = reg.get_provider("a")
# Same endpoint (base_url, api_key, provider), only ``model`` changed.
new_models = {"a": ModelConfig("a", "http://x/v1", "key", "m2", provider="openai")}
reg.reload(new_models, "a")
assert "a" in reg._clients
assert reg._clients["a"] is client_before
assert "a" in reg._providers
assert reg._providers["a"] is provider_before
def test_reload_drops_client_when_base_url_changes(self) -> None:
"""A ``base_url`` change drops the cached client (different
endpoint = new connection) but keeps the cached provider
``LLMProvider`` is keyed only on the provider string, which
didn't change."""
models = {"a": ModelConfig("a", "http://x/v1", "key", "m", provider="openai")}
reg = ModelRegistry(models=models, default="a")
reg.get_client("a")
provider_before = reg.get_provider("a")
new_models = {"a": ModelConfig("a", "http://y/v1", "key", "m", provider="openai")}
reg.reload(new_models, "a")
# Reload with same models — clients should be cleared
reg.reload(dict(models), "a")
assert "a" not in reg._clients
assert "a" in reg._providers
assert reg._providers["a"] is provider_before
def test_reload_drops_provider_when_provider_string_changes(self) -> None:
"""A provider-type swap (e.g. openai → anthropic) drops both the
client AND the provider so the next resolve picks up the right
``LLMProvider`` implementation against the new SDK."""
models = {"a": ModelConfig("a", "http://x/v1", "key", "m", provider="openai")}
reg = ModelRegistry(models=models, default="a")
reg.get_client("a")
reg.get_provider("a")
new_models = {"a": ModelConfig("a", "http://x/v1", "key", "m", provider="anthropic")}
reg.reload(new_models, "a")
assert "a" not in reg._clients
assert "a" not in reg._providers
def test_reload_drops_clients_for_removed_aliases(self) -> None:
"""Aliases removed from the registry must release their cached
clients otherwise a deleted endpoint's connection pool would
outlive the alias indefinitely."""
models = {
"a": ModelConfig("a", "http://x/v1", "key", "m"),
"b": ModelConfig("b", "http://y/v1", "key", "m"),
}
reg = ModelRegistry(models=models, default="a")
reg.get_client("a")
reg.get_client("b")
# Drop "b" entirely.
new_models = {"a": ModelConfig("a", "http://x/v1", "key", "m")}
reg.reload(new_models, "a")
assert "a" in reg._clients # unchanged endpoint, kept warm
assert "b" not in reg._clients
def test_reload_validates_default(self) -> None:
models_a = {"a": ModelConfig("a", "x", "x", "m")}
@@ -809,6 +944,8 @@ class _FakeUI:
self.infos: list[str] = []
self.errors: list[str] = []
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -1066,24 +1203,57 @@ class TestSessionAgentModel:
def _captured_effort(captured: dict[str, Any]) -> str | None:
"""Pull reasoning_effort out of provider-specific shapes.
openai-compatible servers receive it via extra_body.chat_template_kwargs;
commercial providers receive it as a top-level kwarg.
Chat Completions delivers it as a top-level ``reasoning_effort`` kwarg
(when the model's caps permit it). Operators who route reasoning_effort
through ``chat_template_kwargs`` (gpt-oss-style local templates) get
it inside ``extra_body.chat_template_kwargs``.
"""
if "reasoning_effort" in captured:
return captured["reasoning_effort"]
eb = captured.get("extra_body") or {}
ctk = eb.get("chat_template_kwargs") or {}
return ctk.get("reasoning_effort") or captured.get("reasoning_effort")
return ctk.get("reasoning_effort")
@staticmethod
def _effort_caps() -> dict[str, Any]:
"""Capabilities that allow Chat-Completions reasoning_effort to flow."""
return {
"reasoning_effort_values": [
"minimal",
"low",
"medium",
"high",
"max",
],
}
def _three_model_registry(self, **kwargs: Any) -> ModelRegistry:
caps = self._effort_caps()
return ModelRegistry(
models={
"main": ModelConfig(
"main", "http://m/v1", "k", "main-model", provider="openai-compatible"
"main",
"http://m/v1",
"k",
"main-model",
provider="openai-compatible",
capabilities=dict(caps),
),
"smart": ModelConfig(
"smart", "http://s/v1", "k", "smart-model", provider="openai-compatible"
"smart",
"http://s/v1",
"k",
"smart-model",
provider="openai-compatible",
capabilities=dict(caps),
),
"fast": ModelConfig(
"fast", "http://f/v1", "k", "fast-model", provider="openai-compatible"
"fast",
"http://f/v1",
"k",
"fast-model",
provider="openai-compatible",
capabilities=dict(caps),
),
},
default="main",
@@ -1189,6 +1359,45 @@ class TestSessionAgentModel:
session._run_agent([{"role": "user", "content": "x"}], label="plan", agent_alias="fast")
assert captured["model"] == "fast-model"
def test_session_fallback_inherits_primary_alias_for_caps(self) -> None:
"""When _run_agent has no registry agent route, it must fall back to
the session's primary alias for capability and server_compat lookup —
otherwise per-model caps (reasoning_effort_values, server_compat) get
silently dropped on the agent path."""
reg = self._three_model_registry() # no agent_model / plan_model set
session = _make_session(registry=reg, model_alias="main")
# Probe what _run_agent passes to _provider_extra_params and
# _resolve_capabilities by recording the model_alias on each call.
captured_extra_alias: list[str | None] = []
captured_resolve_alias: list[str | None] = []
original_extra = session._provider_extra_params
original_resolve = session._resolve_capabilities
def spy_extra(*args: Any, **kwargs: Any) -> Any:
captured_extra_alias.append(kwargs.get("model_alias"))
return original_extra(*args, **kwargs)
def spy_resolve(*args: Any, **kwargs: Any) -> Any:
# _resolve_capabilities(provider, model, alias)
alias = args[2] if len(args) >= 3 else kwargs.get("alias")
captured_resolve_alias.append(alias)
return original_resolve(*args, **kwargs)
session._provider_extra_params = spy_extra # type: ignore[method-assign]
session._resolve_capabilities = spy_resolve # type: ignore[method-assign]
self._capture_on(session.client) # patch client.chat.completions.create
session._run_agent([{"role": "user", "content": "x"}], label="plan")
assert captured_extra_alias and captured_extra_alias[-1] == "main", (
f"agent fallback path did not inherit primary alias for extra_params: "
f"{captured_extra_alias!r}"
)
assert captured_resolve_alias and captured_resolve_alias[-1] == "main", (
f"agent fallback path did not inherit primary alias for caps: "
f"{captured_resolve_alias!r}"
)
def test_invalid_alias_raises_in_run_agent(self) -> None:
"""Defence-in-depth: _prepare_* validates first, but _run_agent
rejects unknown aliases too rather than silently falling back."""
@@ -1477,6 +1686,58 @@ class TestLoadModelRegistryDBOnly:
reg = load_model_registry(model="", storage=storage)
assert not reg.has_alias("default")
def test_cli_default_shim_skipped_when_db_models_present(self) -> None:
"""An auto-detected ``--model`` does NOT synthesise a ``default``
alias when the DB already contributes models.
Regression for the silent bypass of ``model.task_alias`` /
``model.plan_alias``: a synthesised ``default`` aliased to whatever
``--base-url`` was at boot leaks into the LLM-visible alias list,
and the LLM picks it for ``task_agent(model="default")`` which
then routes around the operator-configured per-role default.
"""
storage = _MockStorage(
[
{
"alias": "gh200",
"model": "deepseek-ai/DeepSeek-V4-Flash",
"provider": "openai",
"base_url": "http://gh200:8000/v1",
"api_key": "sk-gh200",
"context_window": 1048576,
"capabilities": "{}",
"enabled": True,
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry(
base_url="http://flatspark:8000/v1",
api_key="sk-flatspark",
model="qwen3.6-35B-A3B", # populated by ``detect_model``
storage=storage,
)
assert reg.has_alias("gh200")
assert not reg.has_alias("default")
def test_cli_default_shim_skipped_when_config_models_present(self) -> None:
"""Same shim suppression when only ``[models.*]`` populates configs."""
fake_cfg: dict[str, Any] = {
"models": {"local": {"model": "qwen3-32b"}},
}
with patch("turnstone.core.model_registry.load_config", return_value=fake_cfg):
reg = load_model_registry("http://x/v1", "x", "fallback-model")
assert reg.has_alias("local")
assert not reg.has_alias("default")
def test_cli_default_shim_still_fires_when_registry_empty(self) -> None:
"""Single-model CLI mode (no DB, no config.toml [models.*]) keeps
the back-compat ``default`` alias."""
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "lone-model")
assert reg.has_alias("default")
assert reg.get_config("default").model == "lone-model"
# ---------------------------------------------------------------------------
# server._effective_routing / _apply_routing_overrides
+275
View File
@@ -0,0 +1,275 @@
"""``models_changed`` SSE fanout coverage.
The console pushes a ``models_changed`` cluster event whenever a model
definition is created / updated / deleted / reloaded, or whenever a
setting in :data:`turnstone.console.server._MODEL_AFFECTING_SETTING_KEYS`
is updated or reset. Connected browsers refetch ``/v1/api/models`` on
receipt so the home composer dropdown + admin Models Roles sub-tab
reflect alias edits without a manual reload.
These tests pin two contracts:
- every model-definition CRUD path emits exactly one ``models_changed``
fanout (so the browser stays in sync with the DB);
- settings PUT / DELETE only emit the fanout when the key is
model-affecting unrelated keys (e.g. ``session.retention_days``)
must not trigger spurious dropdown re-renders across the cluster.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route
from starlette.testclient import TestClient
from tests._coord_test_helpers import _AuthMiddleware
from turnstone.console.server import (
_MODEL_AFFECTING_SETTING_KEYS,
admin_create_model_definition,
admin_delete_model_definition,
admin_delete_setting,
admin_model_reload,
admin_update_model_definition,
admin_update_setting,
)
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "models_changed.db"))
def _seed(storage: SQLiteBackend, *, definition_id: str, alias: str) -> None:
storage.create_model_definition(
definition_id=definition_id,
alias=alias,
model="model-x",
provider="openai-compatible",
base_url="http://localhost:8000/v1",
api_key="sk-test",
context_window=8192,
capabilities="{}",
enabled=True,
created_by="admin",
)
def _make_client(storage: SQLiteBackend) -> tuple[TestClient, MagicMock]:
"""Build a TestClient + return the stub collector for assertion.
Wires the four model-definition CRUD/reload routes plus the two
settings mutation routes. Collector is a MagicMock so each
``emit_models_changed`` call lands as a recorded call without
spinning up the full SSE listener queue.
"""
app = Starlette(
routes=[
Route(
"/v1/api/admin/model-definitions",
admin_create_model_definition,
methods=["POST"],
),
Route(
"/v1/api/admin/model-definitions/reload",
admin_model_reload,
methods=["POST"],
),
Route(
"/v1/api/admin/model-definitions/{definition_id}",
admin_update_model_definition,
methods=["PUT"],
),
Route(
"/v1/api/admin/model-definitions/{definition_id}",
admin_delete_model_definition,
methods=["DELETE"],
),
Route(
"/v1/api/admin/settings/{key:path}",
admin_update_setting,
methods=["PUT"],
),
Route(
"/v1/api/admin/settings/{key:path}",
admin_delete_setting,
methods=["DELETE"],
),
],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.auth_storage = storage
app.state.coord_registry = None # CRUD endpoints handle this gracefully
collector = MagicMock()
collector.get_all_nodes.return_value = []
app.state.collector = collector
app.state.proxy_client = MagicMock()
app.state.config_store = MagicMock()
client = TestClient(app)
client.headers.update(
{
"X-Test-User": "admin",
"X-Test-Perms": "admin.models,admin.settings",
}
)
return client, collector
# ---------------------------------------------------------------------------
# Model-definition CRUD endpoints fan out ``models_changed``
# ---------------------------------------------------------------------------
def test_create_emits_models_changed(storage: SQLiteBackend) -> None:
client, collector = _make_client(storage)
resp = client.post(
"/v1/api/admin/model-definitions",
json={
"alias": "fast",
"model": "fast-model",
"provider": "openai-compatible",
"base_url": "http://localhost:9000/v1",
"api_key": "sk-x",
"context_window": 4096,
},
)
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 1
def test_update_emits_models_changed(storage: SQLiteBackend) -> None:
_seed(storage, definition_id="m1", alias="local")
client, collector = _make_client(storage)
resp = client.put(
"/v1/api/admin/model-definitions/m1",
json={"model": "swapped-model"},
)
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 1
def test_update_with_empty_body_does_not_emit(storage: SQLiteBackend) -> None:
"""Empty-body PUT writes no rows + skips the registry refresh — no
SSE fanout either, since nothing actually changed."""
_seed(storage, definition_id="m1", alias="local")
client, collector = _make_client(storage)
resp = client.put("/v1/api/admin/model-definitions/m1", json={})
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 0
def test_delete_emits_models_changed(storage: SQLiteBackend) -> None:
_seed(storage, definition_id="m1", alias="local")
client, collector = _make_client(storage)
resp = client.delete("/v1/api/admin/model-definitions/m1")
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 1
def test_reload_emits_models_changed(storage: SQLiteBackend) -> None:
_seed(storage, definition_id="m1", alias="local")
client, collector = _make_client(storage)
resp = client.post("/v1/api/admin/model-definitions/reload")
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 1
# ---------------------------------------------------------------------------
# Settings PUT / DELETE only emit for model-affecting keys
# ---------------------------------------------------------------------------
# Pinned snapshot of the role-related keys we expect the allowlist to
# cover today. The frozenset itself is asserted further down so a
# stray addition doesn't silently bypass coverage.
_EXPECTED_AFFECTING_KEYS = frozenset(
{
"model.default_alias",
"model.plan_alias",
"model.plan_effort",
"model.task_alias",
"model.task_effort",
"coordinator.model_alias",
"coordinator.reasoning_effort",
"judge.model",
"channels.default_model_alias",
}
)
def _value_for_key(key: str) -> str:
"""Return a registry-valid value for ``key``.
``reasoning_effort`` keys have a fixed choice list; alias-shaped
keys accept arbitrary strings. Avoids per-key custom payloads.
"""
if (
key.endswith("reasoning_effort")
or key.endswith("plan_effort")
or key.endswith("task_effort")
):
return "low"
return "anything"
def test_affecting_keys_set_matches_expected() -> None:
"""Lock in the allowlist so an unintentional removal is caught."""
assert _MODEL_AFFECTING_SETTING_KEYS == _EXPECTED_AFFECTING_KEYS
@pytest.mark.parametrize("key", sorted(_EXPECTED_AFFECTING_KEYS))
def test_settings_put_emits_for_model_affecting_key(storage: SQLiteBackend, key: str) -> None:
client, collector = _make_client(storage)
resp = client.put(
f"/v1/api/admin/settings/{key}",
json={"value": _value_for_key(key)},
)
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 1
@pytest.mark.parametrize("key", sorted(_EXPECTED_AFFECTING_KEYS))
def test_settings_delete_emits_for_model_affecting_key(storage: SQLiteBackend, key: str) -> None:
client, collector = _make_client(storage)
# Seed a row so DELETE has something to remove (otherwise 404).
client.put(
f"/v1/api/admin/settings/{key}",
json={"value": _value_for_key(key)},
)
collector.emit_models_changed.reset_mock()
resp = client.delete(f"/v1/api/admin/settings/{key}")
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 1
def test_settings_put_does_not_emit_for_unrelated_key(
storage: SQLiteBackend,
) -> None:
"""Updating a non-model setting (here: a session retention knob)
must not trigger a cluster-wide dropdown refresh."""
client, collector = _make_client(storage)
resp = client.put(
"/v1/api/admin/settings/session.retention_days",
json={"value": 30},
)
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 0
def test_settings_delete_does_not_emit_for_unrelated_key(
storage: SQLiteBackend,
) -> None:
client, collector = _make_client(storage)
client.put(
"/v1/api/admin/settings/session.retention_days",
json={"value": 30},
)
collector.emit_models_changed.reset_mock()
resp = client.delete("/v1/api/admin/settings/session.retention_days")
assert resp.status_code == 200, resp.text
assert collector.emit_models_changed.call_count == 0
+400
View File
@@ -0,0 +1,400 @@
"""Tests for the console-side ``NotifyDispatcher``.
Exercises the dispatcher against the SQLite synthetic-sweep path so the
suite runs without a Postgres dependency. The PG path is shaped the
same way (same handler invocation semantics) the only difference is
the underlying stream's wake-up source, which is covered separately in
``test_storage_notify.py::TestPostgresNotify``.
"""
from __future__ import annotations
import threading
import time
import pytest
@pytest.fixture
def dispatcher_factory(storage):
"""Yield a factory that constructs + tracks dispatchers for teardown."""
from turnstone.console.notify_dispatcher import NotifyDispatcher
created: list[NotifyDispatcher] = []
def _make(*, channels: list[str]) -> NotifyDispatcher:
d = NotifyDispatcher(storage, channels=channels)
created.append(d)
return d
yield _make
for d in created:
d.stop(timeout=2.0)
def _wait_for(predicate, deadline_sec: float = 3.0) -> bool:
"""Poll ``predicate`` until True or timeout. Returns bool."""
deadline = time.monotonic() + deadline_sec
while time.monotonic() < deadline:
if predicate():
return True
time.sleep(0.02)
return False
def _start_ready(d, *, timeout: float = 5.0) -> None:
"""``d.start()`` + assert the listener is actually listening.
Closes the start-vs-notify race for backends where ``storage.listen``
blocks on the network (Postgres ``LISTEN`` over a fresh psycopg
connection): without the sync, a same-thread ``storage.notify`` can
fire before the LISTEN registers and the notification is lost.
"""
d.start()
if not d.wait_until_ready(timeout=timeout):
msg = f"dispatcher listener did not open within {timeout}s"
raise AssertionError(msg)
class TestSubscribe:
def test_subscribe_registers_handler(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen: list = []
d.subscribe("alpha", lambda n: seen.append(n))
_start_ready(d)
# Fire a notify via the storage layer — dispatcher delivers to handler.
storage.notify("alpha", "hello")
assert _wait_for(lambda: any(n.payload == "hello" for n in seen))
def test_subscribe_undeclared_channel_raises(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
with pytest.raises(ValueError, match="not declared"):
d.subscribe("beta", lambda n: None)
def test_subscribe_returns_unsubscribe_callable(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen: list = []
unsub = d.subscribe("alpha", lambda n: seen.append(n))
_start_ready(d)
storage.notify("alpha", "first")
assert _wait_for(lambda: any(n.payload == "first" for n in seen))
unsub()
# After unsubscribe, the handler no longer fires. Drain old hits
# so the next notify-vs-handler-count check is unambiguous.
seen.clear()
storage.notify("alpha", "second")
# Give the dispatcher a beat to deliver if it were going to.
time.sleep(0.2)
assert not any(n.payload == "second" for n in seen)
def test_construction_requires_at_least_one_channel(self, storage):
from turnstone.console.notify_dispatcher import NotifyDispatcher
with pytest.raises(ValueError, match="at least one"):
NotifyDispatcher(storage, channels=[])
def test_duplicate_channels_deduplicated(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha", "alpha", "beta"])
assert d.channels == ["alpha", "beta"]
class TestDispatch:
def test_multiple_handlers_each_invoked(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen_a: list = []
seen_b: list = []
d.subscribe("alpha", lambda n: seen_a.append(n))
d.subscribe("alpha", lambda n: seen_b.append(n))
_start_ready(d)
storage.notify("alpha", "shared")
assert _wait_for(lambda: seen_a and seen_b)
assert seen_a[0].payload == "shared"
assert seen_b[0].payload == "shared"
def test_handler_exception_does_not_break_dispatch(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
survived: list = []
def _broken(_n):
msg = "boom"
raise RuntimeError(msg)
d.subscribe("alpha", _broken)
d.subscribe("alpha", lambda n: survived.append(n))
_start_ready(d)
storage.notify("alpha", "after_broken")
# The second handler runs even though the first raised.
assert _wait_for(lambda: any(n.payload == "after_broken" for n in survived))
def test_dispatch_filters_by_channel(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha", "beta"])
seen_a: list = []
seen_b: list = []
d.subscribe("alpha", lambda n: seen_a.append(n))
d.subscribe("beta", lambda n: seen_b.append(n))
_start_ready(d)
storage.notify("alpha", "for_a")
storage.notify("beta", "for_b")
assert _wait_for(lambda: seen_a and seen_b)
assert all(n.payload == "for_a" for n in seen_a)
assert all(n.payload == "for_b" for n in seen_b)
class TestReconnect:
"""Reconnect + synthetic ``reconcile`` notify on stream-open success.
Uses a stub storage that owns its own listen stream so the test can
drive a controlled stream-error sequence the SQLite path can't
raise :class:`NotifyConnectionError`, and the PG path requires a
real database outage to exercise this code, neither of which fits a
unit test. The dispatcher's threading and reconcile-pending logic
are storage-agnostic the dispatcher sees the same
:class:`NotifyStream` Protocol regardless of backend.
"""
def test_reconcile_fires_after_reopen_not_before(self):
from turnstone.console.notify_dispatcher import NotifyDispatcher
from turnstone.core.storage._notify import Notify, NotifyConnectionError
# State machine: open -> first poll raises NotifyConnectionError
# -> dispatcher waits backoff then reopens -> second open's first
# poll blocks forever (test stops the dispatcher before then).
# The fix: synthetic reconcile fires AFTER the second open
# succeeds, not after the first open fails.
sequence: list[str] = []
reopen_event = threading.Event()
class _StubStream:
def __init__(self, fail_first_poll: bool):
self._fail = fail_first_poll
self._closed = False
def poll(self, _timeout):
if self._closed:
return []
if self._fail:
self._fail = False
sequence.append("poll_raises")
msg = "fake-disconnect"
raise NotifyConnectionError(msg)
sequence.append("poll_returns")
# Block until close to simulate a quiet steady-state.
time.sleep(0.5)
return []
def close(self):
self._closed = True
class _StubStorage:
def __init__(self):
self._open_count = 0
def listen(self, _channels):
import contextlib as _contextlib
@_contextlib.contextmanager
def _cm():
self._open_count += 1
sequence.append(f"open_{self._open_count}")
if self._open_count == 2:
reopen_event.set()
stream = _StubStream(fail_first_poll=(self._open_count == 1))
try:
yield stream
finally:
stream.close()
return _cm()
# Speed up backoff so the reopen happens promptly in the test.
import turnstone.console.notify_dispatcher as nd_mod
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
try:
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
got: list[Notify] = []
d.subscribe("alpha", lambda n: got.append(n))
d.start()
try:
# Wait for the second open (post-reconnect).
assert reopen_event.wait(3.0), "dispatcher did not reopen after disconnect"
# Reconcile should be delivered shortly after the reopen.
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if any(n.payload == "reconcile" for n in got):
break
time.sleep(0.02)
assert any(n.payload == "reconcile" for n in got), (
f"no reconcile delivered; sequence={sequence}, got={got}"
)
# The reconcile must NOT fire before the second open —
# if it did, the index of 'open_2' in sequence would
# come after any reconcile-emitting work. Check ordering:
# 'open_1' < 'poll_raises' < 'open_2' (synthesize happens
# inside the with-block of the SECOND open).
ix_open_1 = sequence.index("open_1")
ix_raises = sequence.index("poll_raises")
ix_open_2 = sequence.index("open_2")
assert ix_open_1 < ix_raises < ix_open_2
finally:
d.stop(timeout=2.0)
finally:
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
def test_generic_exception_path_also_synthesizes_reconcile(self):
"""Exceptions thrown during ``listen()`` (not via stream.poll) still trigger reconcile.
Models the ``psycopg.connect()`` / initial ``LISTEN`` failure
shape, which doesn't go through the stream's exception
translator and would hit the generic ``except Exception``
branch. Pre-fix, that branch emitted no reconcile.
"""
from turnstone.console.notify_dispatcher import NotifyDispatcher
reopen_event = threading.Event()
class _StubStream:
def __init__(self):
self._closed = False
def poll(self, _timeout):
if self._closed:
return []
time.sleep(0.5)
return []
def close(self):
self._closed = True
class _StubStorage:
def __init__(self):
self._open_count = 0
def listen(self, _channels):
import contextlib as _contextlib
self._open_count += 1
if self._open_count == 1:
# First open raises a generic exception (e.g.
# ``psycopg.OperationalError`` from a failed connect)
# — landing in the dispatcher's generic except branch.
msg = "fake-connect-failure"
raise RuntimeError(msg)
@_contextlib.contextmanager
def _cm():
reopen_event.set()
stream = _StubStream()
try:
yield stream
finally:
stream.close()
return _cm()
import turnstone.console.notify_dispatcher as nd_mod
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
try:
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
got: list = []
d.subscribe("alpha", lambda n: got.append(n))
d.start()
try:
assert reopen_event.wait(3.0), "dispatcher did not reopen after generic exception"
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if any(n.payload == "reconcile" for n in got):
break
time.sleep(0.02)
assert any(n.payload == "reconcile" for n in got), (
"no reconcile delivered after generic-exception recovery"
)
finally:
d.stop(timeout=2.0)
finally:
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
class TestCoalescing:
"""Same-channel burst collapses to one handler invocation per batch."""
def test_burst_coalesces_to_one_handler_call_per_channel(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
invocations: list = []
# Slow handler to ensure all bursts queue up before the first
# call returns — gives the dispatch loop time to coalesce.
coalesce_gate = threading.Event()
def _slow_handler(n):
invocations.append(n)
coalesce_gate.wait(0.05)
d.subscribe("alpha", _slow_handler)
_start_ready(d)
# Burst of 10 notifies on the same channel — should coalesce
# down to many fewer handler invocations.
for i in range(10):
storage.notify("alpha", str(i))
# Wait until the dispatch settles (handler is called at least once
# and the queue empties).
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if invocations and d._dispatch_queue.empty():
time.sleep(0.1) # allow any final coalesced call to land
break
time.sleep(0.02)
coalesce_gate.set()
# At least one handler call; well fewer than 10 (coalescing
# collapsed the burst). Exact count depends on timing — typical
# is 1-2 invocations per burst on a fast machine.
assert invocations, "handler never fired"
assert len(invocations) < 10, (
f"expected coalescing to collapse burst of 10; got {len(invocations)} invocations"
)
class TestLifecycle:
def test_start_is_idempotent(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
d.start() # No-op, no thread doubling
# Single listener + single dispatch thread are spawned regardless.
# Inspect by name so we don't depend on the exact thread count of
# the test runner.
listener_threads = [
t for t in threading.enumerate() if t.name == "notify-dispatcher-listener"
]
dispatch_threads = [
t for t in threading.enumerate() if t.name == "notify-dispatcher-dispatch"
]
assert len(listener_threads) == 1
assert len(dispatch_threads) == 1
def test_stop_is_idempotent(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
d.stop(timeout=2.0)
d.stop(timeout=2.0) # No-op, no error
def test_stop_without_start_is_noop(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.stop(timeout=1.0) # No-op, no thread to join
def test_stop_joins_threads(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
# Capture thread references then stop and assert they exited.
threads_before = [
t
for t in threading.enumerate()
if t.name in {"notify-dispatcher-listener", "notify-dispatcher-dispatch"}
]
assert threads_before
d.stop(timeout=3.0)
time.sleep(0.05)
for t in threads_before:
assert not t.is_alive(), f"{t.name} still alive after stop"
+533
View File
@@ -0,0 +1,533 @@
"""Unit tests for :class:`NudgeQueue`."""
from __future__ import annotations
import threading
import pytest
from turnstone.core.nudge_queue import TOOL_DRAIN, USER_DRAIN, NudgeQueue
class TestEnqueueDrain:
def test_enqueue_drain_fifo_order(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "any")
# Drain everything regardless of channel — preserves insertion order.
out = q.drain({"user", "tool", "any"})
# Drain returns ``(nudge_type, text, metadata)``; producers
# without ``metadata`` see ``None`` in the third slot.
assert out == [("a", "1", None), ("b", "2", None), ("c", "3", None)]
assert len(q) == 0
def test_drain_filter_keeps_non_matching(self):
q = NudgeQueue()
q.enqueue("a", "x", "user")
q.enqueue("b", "y", "tool")
# Drain only user → tool entry stays.
out = q.drain(USER_DRAIN)
assert out == [("a", "x", None)]
assert len(q) == 1
# Now drain tool — gets the remaining entry.
out = q.drain(TOOL_DRAIN)
assert out == [("b", "y", None)]
assert len(q) == 0
def test_any_channel_drains_on_either_seam(self):
q = NudgeQueue()
q.enqueue("c", "z", "any")
# User-seam drain pulls "any".
assert q.drain(USER_DRAIN) == [("c", "z", None)]
assert len(q) == 0
# Re-enqueue and prove tool-seam also drains "any".
q.enqueue("d", "w", "any")
assert q.drain(TOOL_DRAIN) == [("d", "w", None)]
assert len(q) == 0
def test_drain_empty_filter_no_op(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
# Empty filter drains nothing.
assert q.drain(set()) == []
assert len(q) == 1
def test_drain_empty_queue_returns_empty_list(self):
q = NudgeQueue()
# Fast-path: no items → no kept-deque allocation, just `[]`.
assert q.drain(USER_DRAIN) == []
assert q.drain({"user", "tool", "any"}) == []
assert len(q) == 0
def test_drain_preserves_order_across_partial_drain(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "user")
q.enqueue("d", "4", "tool")
# Drain user — should get "a" then "c" in order; "b","d" stay.
assert q.drain({"user"}) == [("a", "1", None), ("c", "3", None)]
# Tool drain follows insertion order on remaining.
assert q.drain({"tool"}) == [("b", "2", None), ("d", "4", None)]
class TestLenAndClear:
def test_len_does_not_mutate(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
assert len(q) == 1
assert len(q) == 1 # second call still 1; not consumed
assert q.pending() == [("a", "1")]
def test_len_empty_is_zero(self):
q = NudgeQueue()
assert len(q) == 0
def test_clear_returns_count(self):
q = NudgeQueue()
assert q.clear() == 0
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "any")
assert q.clear() == 3
assert len(q) == 0
def test_clear_empty_returns_zero(self):
q = NudgeQueue()
assert q.clear() == 0
class TestDropOldestByType:
def test_drop_oldest_by_type_removes_earliest_match(self):
"""Drop the FIRST entry of the matching type; later matches stay."""
q = NudgeQueue()
q.enqueue("other", "first", "any")
q.enqueue("target", "older", "any")
q.enqueue("target", "newer", "any")
# "older" is the earliest target — drop it.
assert q.drop_oldest_by_type("target") is True
assert q.pending() == [("other", "first"), ("target", "newer")]
def test_drop_oldest_by_type_no_match_returns_false(self):
"""Empty queue and unmatched-type cases both return False."""
q = NudgeQueue()
# Empty.
assert q.drop_oldest_by_type("target") is False
# Non-matching items only.
q.enqueue("other", "1", "any")
q.enqueue("other", "2", "tool")
assert q.drop_oldest_by_type("target") is False
# Queue is unaffected.
assert q.pending() == [("other", "1"), ("other", "2")]
def test_drop_oldest_by_type_only_drops_one(self):
"""Multiple matching entries → only the first is removed."""
q = NudgeQueue()
q.enqueue("target", "1", "any")
q.enqueue("target", "2", "any")
q.enqueue("target", "3", "any")
assert q.drop_oldest_by_type("target") is True
assert q.pending() == [("target", "2"), ("target", "3")]
def test_drop_oldest_by_type_channel_filter(self):
"""With ``channel`` set, drop walks only that channel. Pairs with
:meth:`count_by_type(..., channel=...)` so producer-side soft caps
operate on a consistent entry set.
"""
q = NudgeQueue()
q.enqueue("target", "user-1", "user")
q.enqueue("target", "any-1", "any")
q.enqueue("target", "any-2", "any")
# Drop the oldest "any"-channel target — leaves the user one
# untouched even though it's earlier in insertion order.
assert q.drop_oldest_by_type("target", channel="any") is True
assert q.pending() == [
("target", "user-1"),
("target", "any-2"),
]
# And a channel with no matches returns False without touching
# the queue.
assert q.drop_oldest_by_type("target", channel="tool") is False
assert q.pending() == [
("target", "user-1"),
("target", "any-2"),
]
class TestCapAtOrDropOldest:
def test_below_cap_no_drop(self):
q = NudgeQueue()
for i in range(3):
q.enqueue("target", f"t-{i}", "any")
# 3 entries, cap=5 → no drop.
assert q.cap_at_or_drop_oldest("target", 5, channel="any") is False
assert q.count_by_type("target") == 3
def test_at_cap_drops_oldest(self):
q = NudgeQueue()
for i in range(5):
q.enqueue("target", f"t-{i}", "any")
# 5 entries, cap=5 → drop the oldest ("t-0"), leaving 4.
assert q.cap_at_or_drop_oldest("target", 5, channel="any") is True
remaining = q.pending()
assert ("target", "t-0") not in remaining
assert len(remaining) == 4
assert remaining[0] == ("target", "t-1") # FIFO drop-oldest preserved
def test_above_cap_drops_only_one(self):
q = NudgeQueue()
for i in range(7):
q.enqueue("target", f"t-{i}", "any")
# 7 entries, cap=5 → drop only ONE per call (soft-cap regulates over time).
assert q.cap_at_or_drop_oldest("target", 5, channel="any") is True
assert q.count_by_type("target") == 6
def test_channel_filter_respected(self):
q = NudgeQueue()
for i in range(3):
q.enqueue("target", f"any-{i}", "any")
for i in range(3):
q.enqueue("target", f"user-{i}", "user")
# 3 "any"-channel entries; cap=3 on channel="any" → drop oldest "any" only.
assert q.cap_at_or_drop_oldest("target", 3, channel="any") is True
# User-channel entries untouched.
assert q.count_by_type("target", channel="user") == 3
assert q.count_by_type("target", channel="any") == 2
def test_other_types_ignored(self):
q = NudgeQueue()
for i in range(5):
q.enqueue("other", f"o-{i}", "any")
q.enqueue("target", "t-0", "any")
# Only one "target" entry; cap=1 on "target" → drop it. "other"
# entries are untouched even though queue holds 6 total.
assert q.cap_at_or_drop_oldest("target", 1, channel="any") is True
assert q.count_by_type("target") == 0
assert q.count_by_type("other") == 5
def test_zero_or_negative_cap_no_op(self):
q = NudgeQueue()
q.enqueue("target", "t-0", "any")
assert q.cap_at_or_drop_oldest("target", 0, channel="any") is False
assert q.cap_at_or_drop_oldest("target", -1, channel="any") is False
assert q.count_by_type("target") == 1
def test_no_match_returns_false(self):
q = NudgeQueue()
q.enqueue("other", "o-0", "any")
assert q.cap_at_or_drop_oldest("target", 1, channel="any") is False
assert q.count_by_type("other") == 1
class TestCountByType:
def test_count_by_type_no_channel(self):
"""Count across all channels with ``channel=None``."""
q = NudgeQueue()
q.enqueue("target", "1", "user")
q.enqueue("other", "x", "any")
q.enqueue("target", "2", "any")
q.enqueue("target", "3", "tool")
assert q.count_by_type("target") == 3
assert q.count_by_type("other") == 1
assert q.count_by_type("missing") == 0
def test_count_by_type_with_channel_filter(self):
"""Filter narrows the count to one channel — used by producer-side
soft caps that pair with ``drop_oldest_by_type(..., channel=...)``.
"""
q = NudgeQueue()
q.enqueue("target", "u-1", "user")
q.enqueue("target", "a-1", "any")
q.enqueue("target", "a-2", "any")
q.enqueue("target", "t-1", "tool")
assert q.count_by_type("target", channel="any") == 2
assert q.count_by_type("target", channel="user") == 1
assert q.count_by_type("target", channel="tool") == 1
def test_count_by_type_empty_queue(self):
q = NudgeQueue()
assert q.count_by_type("anything") == 0
assert q.count_by_type("anything", channel="any") == 0
class TestPending:
def test_pending_no_filter_returns_all_in_order(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "any")
# All three, in insertion order, as (nudge_type, text) tuples.
assert q.pending() == [("a", "1"), ("b", "2"), ("c", "3")]
def test_pending_channel_filter(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
q.enqueue("c", "3", "user")
q.enqueue("d", "4", "any")
assert q.pending("user") == [("a", "1"), ("c", "3")]
assert q.pending("tool") == [("b", "2")]
assert q.pending("any") == [("d", "4")]
def test_pending_does_not_mutate(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
# Two pending calls return same content; nothing consumed.
first = q.pending()
second = q.pending()
assert first == second
assert len(q) == 2
class TestMetadata:
"""Producer-supplied ``metadata`` rides alongside ``(type, text)`` on
drain. Today only ``watch_triggered`` populates it; the wire shape
accommodates future producers (e.g. structured tool_error context)
without another schema bump.
"""
def test_drain_returns_metadata_when_set(self):
q = NudgeQueue()
meta = {"watch_name": "w1", "command": "ls", "poll_count": 2}
q.enqueue("watch_triggered", "$ ls\nfile.txt", "any", metadata=meta)
out = q.drain({"any"})
assert out == [("watch_triggered", "$ ls\nfile.txt", meta)]
def test_drain_returns_none_when_metadata_unset(self):
q = NudgeQueue()
q.enqueue("idle_children", "kids", "any") # no metadata kwarg
out = q.drain({"any"})
assert out == [("idle_children", "kids", None)]
def test_pending_with_metadata_projects_third_field(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("watch_triggered", "out", "any", metadata={"watch_name": "w"})
snapshot = q.pending_with_metadata()
assert snapshot == [
("a", "1", None),
("watch_triggered", "out", {"watch_name": "w"}),
]
# ``pending`` (without metadata) keeps the legacy 2-tuple shape.
assert q.pending() == [("a", "1"), ("watch_triggered", "out")]
def test_metadata_survives_partial_drain(self):
"""A ``user``-channel drain leaves an unaffected ``tool``-channel
entry its metadata must still be present on the next drain."""
q = NudgeQueue()
q.enqueue("user_thing", "u", "user")
q.enqueue("watch_triggered", "w-out", "tool", metadata={"watch_name": "w1"})
# User drain doesn't touch the tool entry.
assert q.drain({"user"}) == [("user_thing", "u", None)]
# Tool drain still has the metadata.
assert q.drain({"tool"}) == [
("watch_triggered", "w-out", {"watch_name": "w1"}),
]
def test_metadata_with_valid_until_predicate(self):
"""Metadata + ``valid_until`` co-exist on the same entry; the
predicate gate runs as before, and on a True result the metadata
rides the drained tuple.
"""
q = NudgeQueue()
q.enqueue(
"watch_triggered",
"w-out",
"any",
valid_until=lambda: True,
metadata={"watch_name": "w1", "is_final": True},
)
out = q.drain({"any"})
assert out == [
("watch_triggered", "w-out", {"watch_name": "w1", "is_final": True}),
]
class TestHasPending:
def test_has_pending_returns_false_on_empty_queue(self):
q = NudgeQueue()
assert q.has_pending({"user", "any"}) is False
assert q.has_pending({"tool"}) is False
def test_has_pending_short_circuits_on_first_match(self):
q = NudgeQueue()
q.enqueue("a", "1", "tool")
q.enqueue("b", "2", "user")
# First entry doesn't match, second does — true after walking 2.
assert q.has_pending({"user"}) is True
def test_has_pending_returns_false_when_no_match(self):
q = NudgeQueue()
q.enqueue("a", "1", "tool")
q.enqueue("b", "2", "tool")
assert q.has_pending({"user", "any"}) is False
def test_has_pending_matches_any_channel(self):
q = NudgeQueue()
q.enqueue("a", "1", "any")
# USER_DRAIN-shaped filter pulls "any" entries.
assert q.has_pending(USER_DRAIN) is True
# TOOL_DRAIN-shaped filter also pulls "any" entries.
assert q.has_pending(TOOL_DRAIN) is True
def test_has_pending_does_not_mutate(self):
q = NudgeQueue()
q.enqueue("a", "1", "user")
q.enqueue("b", "2", "tool")
before = q.pending()
q.has_pending({"user"})
q.has_pending({"tool"})
q.has_pending(set())
assert q.pending() == before
class TestValidation:
def test_invalid_channel_raises(self):
q = NudgeQueue()
with pytest.raises(ValueError, match="channel"):
q.enqueue("a", "1", "wake") # type: ignore[arg-type]
with pytest.raises(ValueError):
q.enqueue("b", "2", "") # type: ignore[arg-type]
# Queue is unaffected by the failed enqueues.
assert len(q) == 0
def test_channel_is_required(self):
q = NudgeQueue()
# No default — caller MUST pick a seam consciously.
with pytest.raises(TypeError):
q.enqueue("a", "1") # type: ignore[call-arg]
class TestValidUntil:
"""``valid_until`` predicate: drain re-checks freshness; falsy /
raising predicates drop the entry without delivery.
"""
def test_valid_until_true_delivers(self):
q = NudgeQueue()
q.enqueue("a", "1", "any", valid_until=lambda: True)
out = q.drain({"any"})
assert out == [("a", "1", None)]
def test_valid_until_false_drops_silently(self):
q = NudgeQueue()
q.enqueue("a", "1", "any", valid_until=lambda: False)
out = q.drain({"any"})
assert out == []
# Already removed from queue (drain partition removes BEFORE
# predicate check — falsy doesn't return to queue).
assert len(q) == 0
def test_valid_until_exception_drops_silently(self):
q = NudgeQueue()
def boom() -> bool:
raise RuntimeError("predicate crash")
q.enqueue("a", "1", "any", valid_until=boom)
out = q.drain({"any"})
assert out == []
# Crash-on-predicate is treated as "no longer valid" — drop, not propagate.
assert len(q) == 0
def test_valid_until_evaluated_outside_lock(self):
"""The predicate may do non-trivial work (e.g. storage I/O)
without blocking other producers. Verify the predicate runs
outside the queue's internal lock by enqueueing from inside
the predicate would deadlock if the lock was still held.
"""
q = NudgeQueue()
def reentrant() -> bool:
# If the lock is held during predicate eval, this enqueue
# would block forever (RLock would let it through, but the
# queue uses a plain Lock).
q.enqueue("inner", "from-predicate", "any", valid_until=lambda: True)
return True
q.enqueue("outer", "1", "any", valid_until=reentrant)
out = q.drain({"any"})
# Outer's predicate ran outside the lock, enqueued "inner";
# outer's True return delivered "outer". "inner" was enqueued
# AFTER the partition snapshot, so it stays in the queue.
assert out == [("outer", "1", None)]
assert q.pending() == [("inner", "from-predicate")]
def test_valid_until_only_evaluated_for_matching_channel(self):
"""A non-matching entry's predicate must NOT fire — that would
be wasted work (or worse, a side-effecting predicate would run
when the entry is supposed to stay queued).
"""
q = NudgeQueue()
calls = []
def track() -> bool:
calls.append(1)
return True
# Tool-channel entry; we drain user-channel. Predicate must not run.
q.enqueue("a", "1", "tool", valid_until=track)
q.drain({"user", "any"})
assert calls == []
# Entry stays queued.
assert q.pending("tool") == [("a", "1")]
def test_valid_until_default_none_always_delivers(self):
# No predicate → entry behaves identically to pre-PR-3 entries.
q = NudgeQueue()
q.enqueue("a", "1", "any") # no valid_until kwarg
assert q.drain({"any"}) == [("a", "1", None)]
class TestConcurrency:
def test_concurrent_enqueue_drain_no_loss(self):
"""16 producer threads × 64 nudges = 1024 total; one consumer
drains in a loop until producers finish + queue empty. Verify
every produced item is observed exactly once.
"""
q = NudgeQueue()
producers = 16
per_producer = 64
total = producers * per_producer
produced: set[tuple[str, str]] = set()
produced_lock = threading.Lock()
observed: list[tuple[str, str]] = []
observed_lock = threading.Lock()
done_event = threading.Event()
def produce(pid: int) -> None:
for i in range(per_producer):
key = (f"p{pid}", f"i{i}")
with produced_lock:
produced.add(key)
q.enqueue(key[0], key[1], "user")
def consume() -> None:
while not done_event.is_set() or len(q) > 0:
drained = q.drain({"user"})
if drained:
with observed_lock:
# Drop the trailing ``metadata`` slot — every
# entry here was enqueued without metadata, so
# the comparison set / count matches the produced
# ``(type, text)`` shape.
observed.extend((nt, txt) for nt, txt, _meta in drained)
consumer = threading.Thread(target=consume, daemon=True)
consumer.start()
threads = [threading.Thread(target=produce, args=(i,)) for i in range(producers)]
for t in threads:
t.start()
for t in threads:
t.join()
done_event.set()
consumer.join(timeout=5.0)
assert not consumer.is_alive(), "consumer didn't finish in time"
# Every produced key observed; no duplicates.
assert set(observed) == produced
assert len(observed) == total
assert len(q) == 0
+172
View File
@@ -0,0 +1,172 @@
"""Direct tests for the shared SSRF helpers in :mod:`turnstone.core.oauth_ssrf`.
The OIDC test suite already exercises these via the OIDC adapter
(``OIDCError`` re-raises). This file pins the canonical
:class:`OAuthSSRFError` exception so callers that don't go through OIDC
(notably ``mcp_oauth``) can rely on a stable contract.
"""
from __future__ import annotations
import urllib.parse
from unittest.mock import patch
import pytest
from turnstone.core.oauth_ssrf import (
OAuthSSRFError,
effective_port,
is_localhost,
validate_discovered_endpoint,
validate_url_no_ssrf,
)
class TestIsLocalhost:
def test_loopback_names(self) -> None:
assert is_localhost("localhost")
assert is_localhost("127.0.0.1")
assert is_localhost("::1")
assert is_localhost("foo.localhost")
def test_non_loopback(self) -> None:
assert not is_localhost("example.com")
assert not is_localhost("internal.corp")
class TestEffectivePort:
def test_explicit_port(self) -> None:
p = urllib.parse.urlparse("https://idp.example.com:9443/foo")
assert effective_port(p) == 9443
def test_default_https(self) -> None:
p = urllib.parse.urlparse("https://idp.example.com/foo")
assert effective_port(p) == 443
def test_default_http(self) -> None:
p = urllib.parse.urlparse("http://idp.example.com/foo")
assert effective_port(p) == 80
def test_unknown_scheme(self) -> None:
p = urllib.parse.urlparse("ftp://idp.example.com/foo")
assert effective_port(p) is None
class TestValidateUrlNoSSRF:
_PUBLIC_ADDR = [(2, 1, 6, "", ("93.184.216.34", 0))]
_PRIVATE_ADDR = [(2, 1, 6, "", ("10.0.0.1", 0))]
_LOOPBACK_ADDR = [(2, 1, 6, "", ("127.0.0.1", 0))]
def test_valid_https(self) -> None:
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
parsed = validate_url_no_ssrf("https://idp.example.com/foo", allow_http=False)
assert parsed.scheme == "https"
assert parsed.hostname == "idp.example.com"
def test_rejects_http_when_not_allowed(self) -> None:
with pytest.raises(OAuthSSRFError, match="must use HTTPS"):
validate_url_no_ssrf("http://idp.example.com", allow_http=False)
def test_allows_http_localhost_with_flag(self) -> None:
with patch("socket.getaddrinfo", return_value=self._LOOPBACK_ADDR):
validate_url_no_ssrf("http://localhost:8080", allow_http=True)
def test_rejects_http_non_localhost_even_with_flag(self) -> None:
with pytest.raises(OAuthSSRFError, match="must use HTTPS"):
validate_url_no_ssrf("http://idp.example.com", allow_http=True)
def test_rejects_userinfo(self) -> None:
with pytest.raises(OAuthSSRFError, match="embedded credentials"):
validate_url_no_ssrf("https://user:pass@idp.example.com", allow_http=False)
def test_rejects_private_address(self) -> None:
with (
patch("socket.getaddrinfo", return_value=self._PRIVATE_ADDR),
pytest.raises(OAuthSSRFError, match="non-public address"),
):
validate_url_no_ssrf("https://corp.example.com", allow_http=False)
def test_rejects_unresolvable(self) -> None:
import socket
with (
patch("socket.getaddrinfo", side_effect=socket.gaierror("fail")),
pytest.raises(OAuthSSRFError, match="cannot be resolved"),
):
validate_url_no_ssrf("https://no.such.host.invalid", allow_http=False)
class TestValidateDiscoveredEndpoint:
_PUBLIC_ADDR = [(2, 1, 6, "", ("93.184.216.34", 0))]
def test_same_origin_passes(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
validate_discovered_endpoint(
"https://idp.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_third_party_host_rejected(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with (
patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR),
pytest.raises(OAuthSSRFError, match="not trusted"),
):
validate_discovered_endpoint(
"https://attacker.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_trusted_endpoint_host_passes(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
validate_discovered_endpoint(
"https://shard.example.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset({"shard.example.com"}),
)
def test_known_google_alias_passes(self) -> None:
"""The hard-coded Google alias map covers oauth2.googleapis.com."""
issuer = urllib.parse.urlparse("https://accounts.google.com")
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
validate_discovered_endpoint(
"https://oauth2.googleapis.com/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
def test_scheme_mismatch_rejected(self) -> None:
# When the issuer is http://localhost (allow_http=True), an
# https:// endpoint must still be rejected as a scheme mismatch.
issuer = urllib.parse.urlparse("http://localhost:8080")
with (
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("127.0.0.1", 0))]),
pytest.raises(OAuthSSRFError, match="scheme"),
):
validate_discovered_endpoint(
"https://localhost:8080/token",
issuer,
allow_http=True,
trusted_endpoint_hosts=frozenset(),
)
def test_port_mismatch_rejected(self) -> None:
issuer = urllib.parse.urlparse("https://idp.example.com")
with (
patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR),
pytest.raises(OAuthSSRFError, match="port"),
):
validate_discovered_endpoint(
"https://idp.example.com:9443/token",
issuer,
allow_http=False,
trusted_endpoint_hosts=frozenset(),
)
+1432 -149
View File
File diff suppressed because it is too large Load Diff
+259 -45
View File
@@ -18,10 +18,7 @@ from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
from tests.conftest import make_oidc_test_config as _make_oidc_config
from turnstone.console.server import (
admin_delete_oidc_identity,
admin_list_oidc_identities,
@@ -32,34 +29,12 @@ from turnstone.core.auth import (
handle_oidc_authorize,
handle_oidc_callback,
)
from turnstone.core.oidc import OIDCConfig, OIDCError
from turnstone.core.oidc import OIDCConfig, OIDCError, OIDCKeyNotFoundError
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_oidc_config(**overrides: Any) -> OIDCConfig:
"""Build a test OIDCConfig with sensible defaults."""
defaults: dict[str, Any] = {
"enabled": True,
"issuer": "https://idp.example.com",
"client_id": "my-client",
"client_secret": "my-secret",
"scopes": "openid email profile",
"provider_name": "TestIDP",
"role_claim": "",
"role_map": {},
"password_enabled": True,
"authorization_endpoint": "https://idp.example.com/authorize",
"token_endpoint": "https://idp.example.com/token",
"userinfo_endpoint": "https://idp.example.com/userinfo",
"jwks_uri": "https://idp.example.com/.well-known/jwks.json",
}
defaults.update(overrides)
return OIDCConfig(**defaults)
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
# ---------------------------------------------------------------------------
# Thin handler wrappers — match the pattern used in server.py / console
@@ -290,9 +265,9 @@ class TestOIDCCallback:
) -> None:
storage.create_oidc_pending_state(state, nonce, code_verifier, audience)
@patch("turnstone.core.oidc.provision_oidc_user")
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_happy_path(
self,
mock_exchange: AsyncMock,
@@ -379,7 +354,7 @@ class TestOIDCCallback:
assert resp.status_code == 302
assert "Login+session+expired" in resp.headers["location"]
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_code_exchange_failure(
self,
mock_exchange: AsyncMock,
@@ -396,8 +371,8 @@ class TestOIDCCallback:
assert resp.status_code == 302
assert "Authentication+failed" in resp.headers["location"]
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_token_validation_failure(
self,
mock_exchange: AsyncMock,
@@ -416,10 +391,10 @@ class TestOIDCCallback:
assert resp.status_code == 302
assert "Authentication+failed" in resp.headers["location"]
@patch("turnstone.core.oidc.provision_oidc_user")
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_jwks_key_rotation_retry(
self,
mock_exchange: AsyncMock,
@@ -429,13 +404,13 @@ class TestOIDCCallback:
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""First validate raises 'kid not found in JWKS', fetch_jwks retried, second validate succeeds."""
"""First validate raises kid-not-found, fetch_jwks retried, second validate succeeds."""
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
# First call raises kid-not-found; second call (after JWKS refresh) succeeds
mock_validate.side_effect = [
OIDCError("Signing key 'new-kid' not found in JWKS"),
OIDCKeyNotFoundError("Signing key 'new-kid' not found in JWKS"),
{"sub": "user123", "email": "u@example.com", "nonce": "test-nonce"},
]
mock_fetch_jwks.return_value = {"keys": [{"kid": "new-kid", "kty": "RSA"}]}
@@ -450,9 +425,61 @@ class TestOIDCCallback:
mock_fetch_jwks.assert_called_once()
assert mock_validate.call_count == 2
@patch("turnstone.core.oidc.provision_oidc_user")
@patch("turnstone.core.oidc.validate_id_token")
@patch("turnstone.core.oidc.exchange_code", new_callable=AsyncMock)
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_callback_uses_keynotfound_for_jwks_retry(
self,
mock_exchange: AsyncMock,
mock_fetch_jwks: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""Retry path keys off the OIDCKeyNotFoundError type, not message substring."""
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
# First raises subclass; rephrased message must not affect retry behaviour.
mock_validate.side_effect = [
OIDCKeyNotFoundError("rotated key absent from cached set"),
{"sub": "user123", "email": "u@example.com", "nonce": "test-nonce"},
]
mock_fetch_jwks.return_value = {"keys": [{"kid": "new-kid", "kty": "RSA"}]}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
mock_fetch_jwks.assert_called_once()
assert mock_validate.call_count == 2
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_callback_returns_authentication_failed_on_missing_id_token(
self,
mock_exchange: AsyncMock,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""A token endpoint response without id_token must redirect with auth-failed."""
self._seed_pending_state(storage)
mock_exchange.return_value = {"access_token": "x"}
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_error=Authentication+failed" in resp.headers["location"]
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_no_users_after_oidc_success_redirects_setup(
self,
mock_exchange: AsyncMock,
@@ -507,6 +534,193 @@ class TestOIDCCallback:
assert "oidc_error" in resp.headers["location"]
assert "Too+many" in resp.headers["location"]
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_setup_gate_uses_count_users_not_full_scan(
self,
mock_exchange: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""Callback's setup-complete gate must call count_users, not list_users."""
from unittest.mock import patch as obj_patch
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_validate.return_value = {
"sub": "u1",
"email": "u@example.com",
"nonce": "test-nonce",
}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
with (
obj_patch.object(storage, "count_users", wraps=storage.count_users) as count_spy,
obj_patch.object(storage, "list_users", wraps=storage.list_users) as list_spy,
):
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
count_spy.assert_called_once_with()
list_spy.assert_not_called()
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_state_cleanup_is_gated(
self,
mock_exchange: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""Cleanup runs once per cleanup-interval window, not every callback."""
from unittest.mock import patch as obj_patch
# First call seeds the cleanup timestamp; subsequent calls within
# _OIDC_STATE_CLEANUP_INTERVAL_S must NOT trigger cleanup again.
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_validate.return_value = {
"sub": "u1",
"email": "u@example.com",
"nonce": "test-nonce",
}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
with obj_patch.object(
storage, "cleanup_expired_oidc_states", wraps=storage.cleanup_expired_oidc_states
) as cleanup_spy:
for state in ("s1", "s2", "s3"):
self._seed_pending_state(storage, state=state, nonce="test-nonce")
authorize_client.get(
f"/v1/api/auth/oidc/callback?code=c&state={state}",
follow_redirects=False,
)
assert cleanup_spy.call_count == 1
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_callback_uses_pending_audience_not_handler_audience(
self,
mock_exchange: AsyncMock,
mock_validate: Any,
mock_provision: Any,
storage: SQLiteBackend,
) -> None:
"""JWT ``aud`` claim must come from the audience stored at /authorize,
not the audience the callback handler was invoked with.
Regression for the cross-service audience-confusion concern: a
login flow opened against the server (audience ``"turnstone-server"``)
must not be silently re-targeted to ``"turnstone-console"`` when
the callback runs through the console's handler wrapper.
"""
import jwt as pyjwt
# Seed pending state with the SERVER audience.
storage.create_oidc_pending_state(
"audience-state",
"audience-nonce",
"audience-verifier",
"turnstone-server",
)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_validate.return_value = {
"sub": "user-aud",
"email": "u@example.com",
"nonce": "audience-nonce",
}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
# Wire a callback bound to the CONSOLE audience. After bug-3 the
# stored audience must take precedence.
async def _console_callback(request: Request) -> Response:
return await handle_oidc_callback(request, "turnstone-console")
jwt_secret = "test-jwt-secret-key-padded-32b!!"
app = Starlette(
routes=[Mount("/v1", routes=[Route("/api/auth/oidc/callback", _console_callback)])]
)
app.state.oidc_config = _make_oidc_config()
app.state.auth_storage = storage
app.state.jwt_secret = jwt_secret
app.state.jwks_data = {"keys": []}
app.state.login_limiter = None
client = TestClient(app, raise_server_exceptions=False)
resp = client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=audience-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
# Extract the JWT from the Set-Cookie header and decode it.
set_cookie = resp.headers["set-cookie"]
cookie_kv = set_cookie.split(";", 1)[0]
name, _, token = cookie_kv.partition("=")
assert name == "turnstone_auth"
assert token
# Decoding without audience verification first to inspect the claim.
claims = pyjwt.decode(
token, jwt_secret, algorithms=["HS256"], options={"verify_aud": False}
)
assert claims["aud"] == "turnstone-server"
assert claims["aud"] != "turnstone-console"
@patch("turnstone.core.auth.provision_oidc_user")
@patch("turnstone.core.auth.validate_id_token")
@patch("turnstone.core.auth.fetch_jwks", new_callable=AsyncMock)
@patch("turnstone.core.auth.exchange_code", new_callable=AsyncMock)
def test_jwks_refetch_dedup_when_kid_appears(
self,
mock_exchange: AsyncMock,
mock_fetch_jwks: AsyncMock,
mock_validate: Any,
mock_provision: Any,
authorize_client: TestClient,
storage: SQLiteBackend,
) -> None:
"""If a concurrent caller already refreshed JWKS, second caller skips fetch."""
from unittest.mock import patch as obj_patch
self._seed_pending_state(storage)
mock_exchange.return_value = {"id_token": "fake.jwt.token"}
mock_provision.return_value = {"user_id": "test-admin", "username": "testadmin"}
# First validate raises kid-not-found; second succeeds.
mock_validate.side_effect = [
OIDCKeyNotFoundError("Signing key 'k-rotated' not found"),
{"sub": "u1", "email": "u@example.com", "nonce": "test-nonce"},
]
# Pre-populate the JWKS cache so the rotated kid is already
# present — analog of a concurrent caller having won the lock.
# The retry path must short-circuit and skip the network fetch.
authorize_client.app.state.jwks_data = {"keys": [{"kid": "k-rotated", "kty": "RSA"}]}
with obj_patch("jwt.get_unverified_header", return_value={"kid": "k-rotated"}):
resp = authorize_client.get(
"/v1/api/auth/oidc/callback?code=authcode&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
assert "oidc_success=1" in resp.headers["location"]
mock_fetch_jwks.assert_not_called()
# ---------------------------------------------------------------------------
# Admin OIDC identity endpoint tests
+293
View File
@@ -6,6 +6,85 @@ import time
import pytest
from turnstone.core.storage import StorageConflictError
# ---------------------------------------------------------------------------
# Atomic OIDC user provisioning
# ---------------------------------------------------------------------------
class TestCreateOIDCUser:
def test_create_oidc_user_success(self, db):
"""Both rows present after one atomic call."""
db.create_oidc_user(
user_id="u-new",
username="alice",
display_name="Alice",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-1",
email="alice@example.com",
)
user = db.get_user("u-new")
assert user is not None
assert user["username"] == "alice"
assert user["password_hash"] == "!oidc"
identity = db.get_oidc_identity("https://idp.example.com", "sub-1")
assert identity is not None
assert identity["user_id"] == "u-new"
assert identity["email"] == "alice@example.com"
def test_create_oidc_user_username_conflict_rolls_back(self, db):
"""Pre-existing username -> StorageConflictError; identity NOT inserted."""
db.create_user("u-existing", "alice", "Alice", "$2b$12$hash")
with pytest.raises(StorageConflictError, match="username"):
db.create_oidc_user(
user_id="u-new",
username="alice",
display_name="Alice2",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-1",
email="alice2@example.com",
)
# The new user_id row must not exist.
assert db.get_user("u-new") is None
# The identity row must not exist.
assert db.get_oidc_identity("https://idp.example.com", "sub-1") is None
# The pre-existing user is untouched.
existing = db.get_user("u-existing")
assert existing is not None
assert existing["password_hash"] == "$2b$12$hash"
def test_create_oidc_user_identity_conflict_rolls_back(self, db):
"""Pre-existing (issuer, subject) -> StorageConflictError; user row rolled back."""
db.create_user("u-other", "other", "Other", "!oidc")
db.create_oidc_identity("https://idp.example.com", "sub-1", "u-other", "other@example.com")
with pytest.raises(StorageConflictError, match="OIDC identity"):
db.create_oidc_user(
user_id="u-new",
username="bob",
display_name="Bob",
password_hash="!oidc",
issuer="https://idp.example.com",
subject="sub-1",
email="bob@example.com",
)
# The candidate user row was rolled back.
assert db.get_user("u-new") is None
assert db.get_user_by_username("bob") is None
# The pre-existing identity still points at the original user.
identity = db.get_oidc_identity("https://idp.example.com", "sub-1")
assert identity is not None
assert identity["user_id"] == "u-other"
# ---------------------------------------------------------------------------
# OIDC Identity CRUD
# ---------------------------------------------------------------------------
@@ -304,3 +383,217 @@ class TestOIDCPendingState:
.where(oidc_pending_states.c.state == "state-cleanup")
).scalar()
assert count == 0
# ---------------------------------------------------------------------------
# count_users / find_existing_usernames
# ---------------------------------------------------------------------------
class TestCountUsers:
def test_count_users_empty(self, db):
assert db.count_users() == 0
def test_count_users_after_inserts(self, db):
db.create_user("u1", "alice", "Alice", "h1")
db.create_user("u2", "bob", "Bob", "h2")
db.create_user("u3", "carol", "Carol", "h3")
assert db.count_users() == 3
class TestFindExistingUsernames:
def test_empty_input_returns_empty_set(self, db):
db.create_user("u1", "alice", "Alice", "h1")
assert db.find_existing_usernames([]) == set()
def test_returns_subset_present_in_db(self, db):
db.create_user("u1", "alice", "Alice", "h1")
db.create_user("u2", "bob", "Bob", "h2")
existing = db.find_existing_usernames(["alice", "bob", "carol", "dave"])
assert existing == {"alice", "bob"}
def test_no_matches_returns_empty_set(self, db):
db.create_user("u1", "alice", "Alice", "h1")
assert db.find_existing_usernames(["bob", "carol"]) == set()
# ---------------------------------------------------------------------------
# replace_oidc_roles
# ---------------------------------------------------------------------------
class TestReplaceOIDCRoles:
def _seed_role(self, db, role_id):
db.create_role(role_id, role_id, role_id, "perm.read", False, "")
def test_inserts_added_roles(self, db):
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
added, removed = db.replace_oidc_roles("u1", {"role-a", "role-b"})
assert added == {"role-a", "role-b"}
assert removed == set()
roles = {r["role_id"] for r in db.list_user_roles("u1")}
assert roles == {"role-a", "role-b"}
def test_removes_stale_oidc_roles(self, db):
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
db.assign_role("u1", "role-a", "oidc")
db.assign_role("u1", "role-b", "oidc")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == {"role-b"}
roles = {r["role_id"] for r in db.list_user_roles("u1")}
assert roles == {"role-a"}
def test_preserves_non_oidc_roles(self, db):
"""Manually-assigned and oidc-default rows are NOT touched."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-manual")
self._seed_role(db, "role-default")
self._seed_role(db, "role-oidc-old")
db.assign_role("u1", "role-manual", "admin-ui")
db.assign_role("u1", "role-default", "oidc-default")
db.assign_role("u1", "role-oidc-old", "oidc")
added, removed = db.replace_oidc_roles("u1", set())
# Only the oidc-assigned row was diffed
assert added == set()
assert removed == {"role-oidc-old"}
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {
"role-manual": "admin-ui",
"role-default": "oidc-default",
}
def test_no_op_when_desired_matches_current(self, db):
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "oidc")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
assert {r["role_id"] for r in db.list_user_roles("u1")} == {"role-a"}
def test_empty_user_no_oidc_history(self, db):
db.create_user("u1", "alice", "Alice", "h")
added, removed = db.replace_oidc_roles("u1", set())
assert added == set()
assert removed == set()
def test_desired_role_blocked_by_admin_ui_assignment(self, db):
"""Desired role already held via admin-ui: untouched, no PK conflict."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "admin-ui")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "admin-ui"}
def test_desired_role_blocked_by_oidc_default_assignment(self, db):
"""Desired role already held via oidc-default fallback: untouched."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "oidc-default")
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "oidc-default"}
def test_desired_role_added_alongside_blocked_role(self, db):
"""Mixed case: one desired role is blocked (admin-ui), the other inserts cleanly."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
db.assign_role("u1", "role-a", "admin-ui")
added, removed = db.replace_oidc_roles("u1", {"role-a", "role-b"})
assert added == {"role-b"}
assert removed == set()
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "admin-ui", "role-b": "oidc"}
def test_revoke_only_oidc_assigned_roles(self, db):
"""OIDC-assigned roles get revoked when not in desired; admin-ui rows survive."""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-manual")
self._seed_role(db, "role-oidc-old")
self._seed_role(db, "role-default")
db.assign_role("u1", "role-manual", "admin-ui")
db.assign_role("u1", "role-oidc-old", "oidc")
db.assign_role("u1", "role-default", "oidc-default")
added, removed = db.replace_oidc_roles("u1", set())
assert added == set()
assert removed == {"role-oidc-old"}
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-manual": "admin-ui", "role-default": "oidc-default"}
def test_replace_oidc_roles_no_op_steady_state(self, db):
"""Steady-state re-login: claims unchanged, function must short-circuit.
This pins the contract that drives the SQLite optimistic-read fast
path the common case (token refresh with identical role claims)
must not acquire a write lock.
"""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
self._seed_role(db, "role-b")
db.assign_role("u1", "role-a", "oidc")
db.assign_role("u1", "role-b", "oidc")
added, removed = db.replace_oidc_roles("u1", {"role-a", "role-b"})
assert added == set()
assert removed == set()
# All rows still oidc-assigned with identical membership.
roles = {r["role_id"]: r["assigned_by"] for r in db.list_user_roles("u1")}
assert roles == {"role-a": "oidc", "role-b": "oidc"}
def test_replace_oidc_roles_returns_post_lock_diff(self, db):
"""Returned (added, removed) reflects the post-lock state, not the optimistic read.
The SQLite implementation re-reads under the write lock to defend
against races; the values returned must come from that re-read so
callers (apply_role_mapping audit logs) see the actual transition
that hit the table. Steady-state input must collapse to empty
sets and leave row timestamps unchanged.
"""
db.create_user("u1", "alice", "Alice", "h")
self._seed_role(db, "role-a")
db.assign_role("u1", "role-a", "oidc")
before = db.list_user_roles("u1")
assert len(before) == 1
original_created = before[0]["assignment_created"]
added, removed = db.replace_oidc_roles("u1", {"role-a"})
assert added == set()
assert removed == set()
# No write occurred — the assignment row's timestamp is untouched.
after = db.list_user_roles("u1")
assert len(after) == 1
assert after[0]["assignment_created"] == original_created
+6
View File
@@ -11,6 +11,12 @@ from turnstone.core.session import ChatSession, _render_template
class NullUI:
"""UI adapter that discards all output."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
+125
View File
@@ -0,0 +1,125 @@
"""Tests for ``AnthropicProvider.extract_reasoning_text``.
Phase 1 of the optional-reasoning-persistence feature: provider-side
extractor that walks stored ``provider_blocks`` and returns the
concatenated thinking text, capped at the operator-friendly UI display
size.
These tests drive through the real ``AnthropicProvider`` instance no
mocks of the extractor itself using fixture-shaped blocks that match
what ``_iter_anthropic_stream`` actually accumulates at
``_anthropic.py:713-724`` (``thinking_delta`` + ``signature_delta``
combined into ``{"type": "thinking", "thinking": <text>, "signature":
<sig>}``).
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
from turnstone.core.providers._protocol import (
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
)
@pytest.fixture
def anthropic() -> AnthropicProvider:
return AnthropicProvider()
class TestExtractReasoningText:
def test_none_input_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
assert anthropic.extract_reasoning_text(None) == ""
def test_empty_list_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
assert anthropic.extract_reasoning_text([]) == ""
def test_no_thinking_blocks_returns_empty(self, anthropic: AnthropicProvider) -> None:
blocks: list[dict[str, object]] = [
{"type": "text", "text": "hello"},
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_single_thinking_block_returns_text(self, anthropic: AnthropicProvider) -> None:
blocks = [{"type": "thinking", "thinking": "Let me think about this.", "signature": "abc"}]
assert anthropic.extract_reasoning_text(blocks) == "Let me think about this."
def test_multiple_thinking_blocks_joined_with_newline(
self, anthropic: AnthropicProvider
) -> None:
blocks = [
{"type": "thinking", "thinking": "first thought", "signature": "s1"},
{"type": "thinking", "thinking": "second thought", "signature": "s2"},
]
assert anthropic.extract_reasoning_text(blocks) == "first thought\nsecond thought"
def test_mixed_blocks_extracts_only_thinking(self, anthropic: AnthropicProvider) -> None:
blocks = [
{"type": "thinking", "thinking": "reason A", "signature": "s"},
{"type": "text", "text": "visible answer"},
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
{"type": "thinking", "thinking": "reason B", "signature": "s"},
]
assert anthropic.extract_reasoning_text(blocks) == "reason A\nreason B"
def test_thinking_block_without_thinking_field_skipped(
self, anthropic: AnthropicProvider
) -> None:
blocks = [{"type": "thinking", "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_thinking_block_with_empty_text_skipped(self, anthropic: AnthropicProvider) -> None:
blocks = [{"type": "thinking", "thinking": "", "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_truncation_at_64kib_cap(self, anthropic: AnthropicProvider) -> None:
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
blocks = [{"type": "thinking", "thinking": long_text, "signature": "s"}]
result = anthropic.extract_reasoning_text(blocks)
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
def test_just_under_cap_not_truncated(self, anthropic: AnthropicProvider) -> None:
text = "y" * (_MAX_REASONING_DISPLAY_CHARS - 1)
blocks = [{"type": "thinking", "thinking": text, "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == text
def test_malformed_block_entry_skipped(self, anthropic: AnthropicProvider) -> None:
# A defensive sanity check — we should not crash if some
# entry isn't a dict (e.g. a corrupted JSON payload).
blocks = [
"not a dict", # type: ignore[list-item]
{"type": "thinking", "thinking": "good one", "signature": "s"},
]
assert anthropic.extract_reasoning_text(blocks) == "good one" # type: ignore[arg-type]
def test_non_list_input_returns_empty(self, anthropic: AnthropicProvider) -> None:
# Defensive against a corrupted provider_data payload.
assert anthropic.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
assert anthropic.extract_reasoning_text({"type": "thinking"}) == "" # type: ignore[arg-type]
class TestOtherProvidersDefault:
"""Non-Anthropic providers return "" for the same fixture shapes."""
def test_openai_chat_returns_empty(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
assert provider.extract_reasoning_text(blocks) == ""
def test_openai_responses_returns_empty(self) -> None:
provider = OpenAIResponsesProvider()
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
assert provider.extract_reasoning_text(blocks) == ""
def test_openai_responses_extracts_reasoning_summary(self) -> None:
# Phase 3: extractor now walks reasoning items captured via
# include=["reasoning.encrypted_content"] and returns the
# summary[*].text concatenation. Pre-Phase-3 this returned
# "" — the stub was replaced once the wire path landed.
provider = OpenAIResponsesProvider()
blocks = [{"type": "reasoning", "summary": [{"type": "summary_text", "text": "x"}]}]
assert provider.extract_reasoning_text(blocks) == "x"
+532
View File
@@ -0,0 +1,532 @@
"""Tests for Phase 2 wire-build replay flag + shape filter on AnthropicProvider.
Phase 2 of optional reasoning persistence wraps the verbatim
``_provider_content`` replay path at ``_anthropic.py:_convert_messages``
with two gates:
1. ``ANTHROPIC_VALID_BLOCK_TYPES`` per-block shape filter foreign-
shaped blocks (OpenAI Responses ``type="reasoning"``, Gemini thought
parts, the synthetic ``reasoning_text`` from path-3 capture) are
dropped individually; valid Anthropic blocks in the same message
still ride the verbatim path. When NO valid blocks survive, the
converter falls through to the text+tool_calls rebuild path.
2. ``replay_reasoning_to_model`` operator flag when False (the
``model_definitions`` server_default), thinking blocks are
stripped before the wire payload is built. Tool_use /
server_tool_use / web_search_tool_result blocks (which carry
web-search ``encrypted_content``) intentionally survive the
strip predicate is narrow by design.
Drives through the real ``AnthropicProvider._convert_messages`` with
fixture-shaped messages, no mocks of the converter. Edge cases come
from the briefing's "Edges & validation memo" sections.
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._anthropic import (
ANTHROPIC_REASONING_BLOCK_TYPES,
ANTHROPIC_VALID_BLOCK_TYPES,
AnthropicProvider,
)
@pytest.fixture
def provider() -> AnthropicProvider:
return AnthropicProvider()
def _assistant_with_thinking(content: str = "Final answer.") -> dict[str, object]:
"""Build an assistant message with a thinking + text + tool_use shape
matching what the streaming layer captures at ``_anthropic.py:713-724``."""
return {
"role": "assistant",
"content": content,
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "sig"},
{"type": "text", "text": content},
{
"type": "tool_use",
"id": "call_abc",
"name": "search",
"input": {"q": "x"},
},
],
"tool_calls": [
{
"id": "call_abc",
"type": "function",
"function": {"name": "search", "arguments": '{"q": "x"}'},
}
],
}
class TestReplayFlagStripsThinking:
"""``replay_reasoning_to_model=False`` strips thinking; ``True`` preserves."""
def test_replay_true_preserves_thinking_block(self, provider: AnthropicProvider) -> None:
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present
assert "text" in types_present
assert "tool_use" in types_present
def test_replay_false_strips_thinking_block(self, provider: AnthropicProvider) -> None:
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present
assert "text" in types_present # final answer survives
assert "tool_use" in types_present # tool dispatch survives
def test_replay_false_strips_redacted_thinking_too(self, provider: AnthropicProvider) -> None:
# Anthropic emits redacted_thinking blocks when the safety system
# rewrites a thinking block. Phase 2 strip predicate must include
# both shapes.
msg = {
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{"type": "redacted_thinking", "data": "redacted-blob"},
{"type": "text", "text": "Answer."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "redacted_thinking" not in types_present
assert "text" in types_present
def test_default_kwarg_preserves_existing_behaviour(self, provider: AnthropicProvider) -> None:
"""Pre-Phase-2 callers that don't pass the kwarg get the verbatim
replay (default True), matching the behaviour all production
Anthropic-with-thinking turns shipped with for months."""
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg]) # no kwarg
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present
class TestWebSearchBlocksSurviveStrip:
"""Edge 14: Anthropic web-search ``encrypted_content`` rides on
``server_tool_use`` / ``web_search_tool_result`` blocks (NOT
thinking blocks). Strip predicate is intentionally narrow."""
def test_server_tool_use_survives(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "From search: ...",
"_provider_content": [
{"type": "thinking", "thinking": "I should search", "signature": "s"},
{
"type": "server_tool_use",
"id": "stu_1",
"name": "web_search",
"input": {"query": "turnstone bird"},
},
{
"type": "web_search_tool_result",
"tool_use_id": "stu_1",
"content": [{"type": "web_search_result", "url": "https://e.com"}],
"encrypted_content": "abc123encrypted",
"encrypted_index": "idx456encrypted",
},
{"type": "text", "text": "From search: ..."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present # stripped
assert "server_tool_use" in types_present # survives
assert "web_search_tool_result" in types_present # survives
assert "text" in types_present # survives
# encrypted_content rides through intact — required for round-trip continuity
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
assert wsr["encrypted_content"] == "abc123encrypted"
def test_tool_use_block_survives(self, provider: AnthropicProvider) -> None:
# Plain tool_use (not server-side) — used by client-side function
# tools. Strip predicate must not touch these.
msg = {
"role": "assistant",
"content": "Calling tool",
"_provider_content": [
{"type": "thinking", "thinking": "I should call tool", "signature": "s"},
{"type": "tool_use", "id": "tu_1", "name": "f", "input": {"a": 1}},
],
"tool_calls": [
{
"id": "tu_1",
"type": "function",
"function": {"name": "f", "arguments": '{"a": 1}'},
}
],
}
# Provide the tool result so orphan-tool detection doesn't synthesize
msgs = [
msg,
{"role": "tool", "tool_call_id": "tu_1", "content": "ok"},
]
_, converted = provider._convert_messages(msgs, replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present
assert "tool_use" in types_present
def test_orphan_tool_use_synthesized_after_strip(self, provider: AnthropicProvider) -> None:
"""Pin the post-strip orphan-tool branch at _anthropic.py:397-433.
The implementation comment specifically calls out reading
``provider_content`` (not ``wire_blocks``) for the orphan-tool
ID walk after the strip keeping the read on the source-of-
truth list so a future refactor that swapped them would still
get the same set of tool_use IDs. This test exercises that
branch end-to-end: replay=False strips the thinking block,
AND the message has a tool_use whose result is missing. The
converter must synthesize a 'cancelled' tool_result for the
orphaned tool_use ID (matching the existing pre-Phase-2
behaviour for the verbatim path).
"""
msg = {
"role": "assistant",
"content": "Calling tool",
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "s"},
{"type": "tool_use", "id": "orphan_tu", "name": "f", "input": {"a": 1}},
],
"tool_calls": [
{
"id": "orphan_tu",
"type": "function",
"function": {"name": "f", "arguments": '{"a": 1}'},
}
],
}
# NO tool result follows — orphan branch must synthesize one.
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
# Synthetic tool_result lands as a user-role message immediately
# after the assistant turn (per existing behaviour at
# _anthropic.py:421-430).
assistant = next(m for m in converted if m["role"] == "assistant")
# Stripped: thinking gone, tool_use survives.
a_types = [b["type"] for b in assistant["content"]]
assert "thinking" not in a_types
assert "tool_use" in a_types
# Synthesized: cancelled tool_result for orphan_tu attached to a
# following user-role message.
user_msgs_after = [m for m in converted if m["role"] == "user"]
assert user_msgs_after, (
"Expected a synthetic user message carrying the cancelled "
"tool_result for the orphaned tool_use"
)
flat_results = [
block
for um in user_msgs_after
if isinstance(um["content"], list)
for block in um["content"]
if isinstance(block, dict) and block.get("type") == "tool_result"
]
synth = next(
(b for b in flat_results if b.get("tool_use_id") == "orphan_tu"),
None,
)
assert synth is not None, f"Expected synthetic tool_result for orphan_tu in {flat_results}"
assert synth.get("is_error") is True
assert "cancelled" in synth.get("content", "").lower()
class TestShapeFilterFallthrough:
"""Foreign / empty / mixed-shape ``_provider_content`` falls through
to the text+tool_calls rebuild path rather than reaching the wire
as a malformed block."""
def test_foreign_shape_openai_reasoning_falls_through(
self, provider: AnthropicProvider
) -> None:
# OpenAI Responses style block (Phase 3 will land this shape into
# _provider_content via include=["reasoning.encrypted_content"]).
# Mid-workstream model switch from OpenAI -> Anthropic must NOT
# reach the API with an OpenAI-shaped block (which would 400).
msg = {
"role": "assistant",
"content": "Final answer from openai turn.",
"_provider_content": [
{
"type": "reasoning",
"summary": [{"type": "summary_text", "text": "I reasoned..."}],
"encrypted_content": "openai-encrypted",
}
],
"tool_calls": [],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
# Rebuilt from text — no foreign block reached the wire.
for b in assistant["content"]:
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES, (
f"Foreign block type leaked through: {b}"
)
# And the foreign block specifically is NOT present.
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present
def test_mixed_shape_drops_foreign_keeps_valid(self, provider: AnthropicProvider) -> None:
# Per-block filter: a single foreign block in a mostly-Anthropic
# payload no longer forces fall-through. Valid Anthropic blocks
# ride the verbatim path; the foreign block is dropped.
msg = {
"role": "assistant",
"content": "Mixed.",
"_provider_content": [
{"type": "thinking", "thinking": "anth shape", "signature": "s"},
{"type": "text", "text": "Mixed."},
{"type": "reasoning", "summary": []}, # foreign
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present # foreign dropped
assert "thinking" in types_present # valid + replay=True kept
assert "text" in types_present
for b in assistant["content"]:
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES
def test_mixed_shape_preserves_web_search_encrypted_content(
self, provider: AnthropicProvider
) -> None:
# The motivating case for per-block (vs all-or-nothing) filter:
# cross-model resumption stamps a foreign ``reasoning`` block
# alongside Anthropic web-search blocks carrying encrypted
# citations. An all-or-nothing filter would discard the whole
# message and rebuild from text+tool_calls — silently losing
# the encrypted_content the API needs for round-trip continuity.
msg = {
"role": "assistant",
"content": "From search: ...",
"_provider_content": [
{"type": "reasoning", "summary": []}, # foreign (e.g. OpenAI)
{
"type": "server_tool_use",
"id": "stu_1",
"name": "web_search",
"input": {"query": "x"},
},
{
"type": "web_search_tool_result",
"tool_use_id": "stu_1",
"content": [{"type": "web_search_result", "url": "https://e.com"}],
"encrypted_content": "encrypted-blob-must-survive",
"encrypted_index": "encrypted-idx-must-survive",
},
{"type": "text", "text": "From search: ..."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present # foreign dropped
assert "server_tool_use" in types_present
assert "web_search_tool_result" in types_present
assert "text" in types_present
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
assert wsr["encrypted_content"] == "encrypted-blob-must-survive"
assert wsr["encrypted_index"] == "encrypted-idx-must-survive"
def test_all_foreign_blocks_fall_through_to_rebuild(self, provider: AnthropicProvider) -> None:
# When every block is foreign-shaped (no Anthropic-valid block
# survives the per-block filter), the converter still falls
# through to text+tool_calls rebuild rather than emitting an
# empty assistant turn.
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "reasoning", "summary": []},
{"type": "reasoning_text", "text": "synthetic"}, # path-3 shape
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
# Rebuild path: msg.content lifted into a single text block.
assert assistant["content"] == [{"type": "text", "text": "Final answer."}]
def test_empty_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "Plain text answer.",
"_provider_content": [],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
# Falls through to text rebuild
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
def test_none_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "Plain text answer.",
"_provider_content": None,
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
def test_provider_content_not_a_list_falls_through(self, provider: AnthropicProvider) -> None:
# Defensive against a corrupted provider_data deserialization.
msg = {
"role": "assistant",
"content": "Plain.",
"_provider_content": "not a list",
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Plain."}]
def test_non_dict_and_missing_type_blocks_are_dropped(
self, provider: AnthropicProvider
) -> None:
# Defensive branches in the per-block walk: a stray non-dict
# element (corrupted JSON) or a dict with no/None ``type`` key
# (provider drift) must be silently dropped without raising.
# Valid blocks in the same list still ride the verbatim path.
msg = {
"role": "assistant",
"content": "ok",
"_provider_content": [
{"type": "text", "text": "ok"},
"stray-string", # non-dict
{"type": None, "text": "huh"}, # None type
{"no_type_key": 1}, # missing type
{"type": "thinking", "thinking": "t", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b.get("type") for b in assistant["content"]]
assert types_present == ["text", "thinking"]
class TestLegacyAnthropicRowsNoRegression:
"""Critical property: rows persisted before Phase 2 carry valid
Anthropic-shape _provider_content (only Anthropic captured this lane
historically). They must stay in the verbatim path and keep their
thinking context across the migration boundary when replay=True
(the legacy default).
"""
def test_legacy_thinking_row_preserved_with_default_kwarg(
self, provider: AnthropicProvider
) -> None:
"""No kwarg passed (matches the pre-Phase-2 production call site)."""
msg = {
"role": "assistant",
"content": "Old answer from months ago.",
"_provider_content": [
{"type": "thinking", "thinking": "old reasoning", "signature": "s"},
{"type": "text", "text": "Old answer from months ago."},
],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present # preserved -> no regression
# The thinking block IS the same dict as the source (verbatim path).
assert assistant["content"][0]["thinking"] == "old reasoning"
def test_replay_false_only_strips_when_explicitly_requested(
self, provider: AnthropicProvider
) -> None:
# Operator flips persist+replay flags off. Strip fires.
# Pinning that the strip is gated on the explicit flag value,
# not silently triggered by some other condition.
msg = {
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{"type": "thinking", "thinking": "stripped", "signature": "s"},
{"type": "text", "text": "Answer."},
],
}
_, converted_default = provider._convert_messages([msg])
_, converted_strip = provider._convert_messages([msg], replay_reasoning_to_model=False)
default_types = [b["type"] for b in converted_default[0]["content"]]
strip_types = [b["type"] for b in converted_strip[0]["content"]]
assert "thinking" in default_types
assert "thinking" not in strip_types
class TestStripAllBlocksFallthrough:
"""When the message is 100% thinking (no text, no tool_use) and
replay=False strips everything, the message falls through to the
text+tool_calls rebuild path. If both are also empty, the assistant
turn is silently skipped correct: stripped reasoning has nothing
to replay."""
def test_only_thinking_strip_falls_to_rebuild_with_text(
self, provider: AnthropicProvider
) -> None:
# Provider_content = only thinking; msg.content has the spoken text.
# Strip drops thinking; rebuild path picks up the content as a
# text block. No information lost.
msg = {
"role": "assistant",
"content": "Spoken answer.",
"_provider_content": [
{"type": "thinking", "thinking": "internal", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Spoken answer."}]
def test_only_thinking_strip_with_no_content_skips_message(
self, provider: AnthropicProvider
) -> None:
# Edge: provider_content was 100% thinking AND msg.content is
# empty AND no tool_calls. The rebuild path sees nothing to
# emit — assistant turn silently skipped. Anthropic's API
# would reject an empty assistant content array anyway.
msg = {
"role": "assistant",
"content": "",
"_provider_content": [
{"type": "thinking", "thinking": "only", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
# Assistant turn skipped — no entry for it in `converted`.
assert all(m["role"] != "assistant" for m in converted)
class TestConstants:
"""Pin the constant contents so a future edit doesn't accidentally
widen the strip set or narrow the valid set."""
def test_reasoning_block_types_is_narrow(self) -> None:
# Strip predicate MUST cover only reasoning shapes. Adding
# tool_use here would break web-search round-trip.
assert frozenset({"thinking", "redacted_thinking"}) == ANTHROPIC_REASONING_BLOCK_TYPES
def test_valid_block_types_includes_web_search(self) -> None:
# Without server_tool_use / web_search_tool_result, Anthropic
# web-search results would fall through to the rebuild path
# and lose their encrypted_content.
assert "server_tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
assert "web_search_tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
assert "tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
assert "tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
def test_reasoning_subset_of_valid(self) -> None:
# The strip set must be a subset of the valid set — otherwise
# the strip predicate would never match anything (we only
# strip after shape validity passes).
assert ANTHROPIC_REASONING_BLOCK_TYPES.issubset(ANTHROPIC_VALID_BLOCK_TYPES)

Some files were not shown because too many files have changed in this diff Show More