Compare commits

...

72 Commits

Author SHA1 Message Date
Patrick Buckley 4a38b835f5 chore: bump version to 1.5.18 2026-05-19 08:12:32 -07:00
Patrick Buckley 415be00149 docs(changelog): release 1.5.18 notes 2026-05-19 08:12:24 -07:00
Patrick Buckley 346ad2a6aa docs(changelog): note admin config.toml support + load_config perm warning 2026-05-19 08:12:06 -07:00
Patrick Buckley 8003fcbebe feat(admin): align turnstone-admin DB config with server (config.toml + env) (#531)
* feat(admin): align turnstone-admin DB config with server (config.toml + env)

turnstone-admin previously read TURNSTONE_DB_* env vars only, forcing
operators with credentials in config.toml to re-export them just to
run admin commands. Wire add_config_arg + apply_config(["database"])
into main() so admin honors the same precedence as turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded defaults.

Also exposes pool_size + sslmode/sslrootcert/sslcert/sslkey to admin,
which previously dropped any such config silently.

Hardening: load_config() now warns once when config.toml is group- or
world-readable, since DB password and TLS key paths live in [database].

Tests cover precedence (default / config / env / partial fallback /
empty-string-in-config-beats-env), the real init_storage boundary on
a tmp sqlite path, the sys.argv -> main() pre-parser path, and the
new permission check (mode 0644 warns, 0600 quiet).

* test(admin): unify config import style in test_admin_db_config

Use module alias (config_mod.apply_config) instead of mixing
'import turnstone.core.config as config_mod' with 'from
turnstone.core.config import apply_config'.  Addresses
github-code-quality bot feedback on PR #531.
2026-05-19 08:12:06 -07:00
Patrick Buckley d4f3711d63 chore: bump version to 1.5.17 2026-05-18 20:33:49 -07:00
Patrick Buckley cea229206b docs(changelog): release 1.5.17 notes 2026-05-18 20:33:49 -07:00
Patrick Buckley b7b4dcc0df fix(judge): UPSERT intent_verdicts so llm_fallback upgrades land
Async LLM-tier "llm_fallback" verdicts (judge.py:1073, judge.py:1131
via _deliver_fallbacks) deliberately reuse the heuristic verdict's
``verdict_id`` so the row gets "upgraded in place" from heuristic →
llm_fallback when the LLM judge times out, is cancelled, or returns
no content.  The consumer ``_persist_intent_verdict`` was doing a
plain INSERT via ``create_intent_verdict``, hitting the
``intent_verdicts_pkey`` constraint on every llm_fallback delivery.
Postgres logged the duplicate-key error; the application try/except
swallowed it at log.debug — so the row never actually got upgraded
and the LLM judge's annotation ("(LLM judge did not return a
verdict)") was lost.

The collision rate exploded on stable/1.5 smoke tests because
PR #527 (just merged) added two new heuristic-INSERT paths in the
auto-approve early-return branches of ``approve_tools`` — previously
those branches dropped heuristic verdicts on the floor, leaving no
row for the fallback to collide with.

Fix:
- New ``upsert_intent_verdict`` method on the storage protocol +
  sqlite + postgres impls, using dialect-specific
  ``insert(...).on_conflict_do_update(index_elements=["verdict_id"],
  set_={...})``.  Set_ clause updates ONLY the three fields that
  genuinely change between heuristic and llm_fallback: ``tier``,
  ``reasoning``, ``judge_model``.
- Every other column is excluded from set_: identity columns
  (verdict_id, ws_id, call_id, func_name, func_args), carried-
  verbatim columns (intent_summary, risk_level, confidence,
  recommendation, evidence, latency_ms), and ``user_decision``.
- ``user_decision`` exclusion is load-bearing: ``IntentVerdict
  .to_dict()`` doesn't project it, so a fallback verdict reaching
  ``_persist_intent_verdict`` carries the kwarg's ``"pending"``
  default.  If the operator already resolved the approval between
  heuristic INSERT and fallback delivery, the row's user_decision
  has been stamped to ``"approved"``/``"denied"``/``"timeout"`` (or
  an auto-approve reason at heuristic-INSERT time per PR #527).
  Including ``user_decision`` in set_ would silently clobber that
  back to ``"pending"``.
- ``_persist_intent_verdict`` switched from ``create_*`` to
  ``upsert_*``.  Bulk path ``create_intent_verdicts_bulk`` stays as
  plain INSERT — every heuristic ``verdict_id`` is freshly minted
  in ``judge.evaluate`` so in-turn dups can't happen.  The inverse
  race (daemon-judge verdict lands BEFORE the bulk write) IS
  reachable today but its observable behavior is unchanged by the
  per-row UPSERT switch; documented at the bulk site for a future
  hardening pass.

Test coverage:
- TestIntentVerdictUpsert × 4 — fresh-id insert, conflict-upgrade,
  user_decision preservation across heuristic→approved→fallback,
  identity + carried-field preservation.
- Existing tests in test_session_ui_base.py updated to mock the
  new upsert method instead of create_intent_verdict.
2026-05-18 20:32:02 -07:00
Patrick Buckley 97080e1df9 fix(coord): drop unused snip-threshold constants, name elision margin
Two dead module-level constants flagged by github-code-quality on
PR #529: ``_INSPECT_MSG_SNIP_THRESHOLD`` and
``_INSPECT_TOOL_ARG_SNIP_THRESHOLD`` lost their callers when the
content-snip logic moved into the ``_snip_head_tail`` helper.  The
helper now reads ``head + tail + _INSPECT_ELISION_MARGIN`` so the
"reserve bytes for the elision marker" rationale that the dead
constants documented stays named instead of becoming a bare ``64``.
2026-05-18 19:39:30 -07:00
Patrick Buckley af0cbaaec3 feat(coord): three-tier compression for inspect_workstream output
A coord doing a fan-out wave of inspect_workstream calls against
tool-heavy children could blow the context budget on raw output
alone (one child with a 100 KB bash result × N children).  The
previous safety net was ``_truncate_output``'s head+tail strategy,
which silently drops *middle* messages — exactly the wrong shape
for a coordinator trying to understand a child's trajectory (the
LAST message tells the model what the child concluded; the FIRST
sets the brief; the middle is the connective tissue).

Three-tier degradation modeled on the search tool's pattern at
``session.py:_format_search_results``:

  Tier 1 (full):    every message verbatim — used when size fits.
  Tier 2 (compact): per-message head/tail-snipped content (600/300
                    chars) plus snipped ``tool_calls.arguments``
                    (300/100 chars).  When content snipping alone
                    doesn't fit, fall through a message-list trim
                    ladder ((20,30) → (10,20) → (5,10)) that keeps
                    head + tail messages and elides the middle as
                    ``{"_omitted": N}``.
  Tier 3 (skeleton): no messages — counts + role distribution +
                    verdicts-by-risk + last assistant preview.

Budget 32 KB (matches ``_SEARCH_OUTPUT_BUDGET``).  First emission
whose JSON serialization fits the budget wins.  ``_tier`` lands on
every non-error emission so the coordinator LLM and audit readers
can see which compression rung was selected; ``_tier_note`` carries
actionable advice (re-call with a smaller ``message_limit`` etc.).
Error-shape results bypass tiering — they're already small.

Bug fixes caught during review:
- ``_compact_message`` now preserves the assistant-side ``tool_calls``
  list with snipped ``function.arguments``; the pre-fix shape left
  audit readers with tool-result orphans against invisible calls.
- The intermediate Tier-2 list-trim ladder fixes a size-monotonicity
  bug where Tier-2 with un-snippable content (per-message body
  under the 964-char threshold) plus the added ``_tier_note`` came
  out STRICTLY larger than Tier-1, falling through to skeleton
  when a head+tail trim would have preserved dozens of messages.
- ``_inspect_skeleton`` reads ``result["skill_id"]`` (production
  storage row key) with a ``skill`` fallback; pre-fix it read
  ``skill`` only and emitted ``null`` for every real workstream.
2026-05-18 19:39:30 -07:00
Patrick Buckley b9ce0d388e fix(coord): address PR review threads on spawn_workstream rename
Three Copilot threads from PR #526:

1. ``_exec_spawn_workstream`` success path emitted
   ``{"child_ws_id": null}`` when the upstream response unexpectedly
   omitted ``ws_id`` (200-shape with no error field, no id field).
   Adds the missing guard — mirrors ``_exec_spawn_batch`` which
   already surfaces ``"spawn returned no ws_id"`` as a denied row.
   The LLM now sees a tool error and can retry instead of chasing
   a null id through follow-up tools.

2. ``docs/coordinator-skills.md`` UI render note said "keep the
   ws_id as the click-through key" in a paragraph that had just
   introduced ``child_ws_id`` — readable as "the ws_id value" but
   confusable as a field-name claim.  Clarifies that the value
   class is the same regardless of which key carried it.

3. ``docs/bulk-endpoints.md`` ``spawn_batch`` example shows
   ``child_ws_id`` (coord-tool output shape).  The doc title and
   the "model tool" column label already disambiguate it from HTTP
   API responses, but a reader landing at the example section
   directly could miss the framing.  Adds one explicit sentence.
2026-05-18 19:39:30 -07:00
Patrick Buckley d07d2242aa fix(coord): rename ws_id->child_ws_id in spawn return JSON
Coordinator LLMs on large fan-outs recency-bias on seeing `ws_id`
in a `spawn_workstream` / `spawn_batch` return -- calling
`spawn_workstream(ws_id=...)` again instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this
cascades into self-inflicted re-spawn loops.

Rename to `child_ws_id` (already an existing project term -- see
`tasks` tool, `child_event_bus.py`) defuses the recency bias.
Scope is the LLM-facing JSON only -- the server HTTP API at the
spawn endpoint still returns `ws_id`, and the internal reads of
that HTTP response are unchanged.

Also updates the two tool descriptions, the operator-facing skill
doc, and the bulk-endpoints example so docs don't undo the rename.
2026-05-18 19:39:30 -07:00
Patrick Buckley 70514cc406 fix(coord): omit empty allowed_tools in list_skills + clarify semantics
A skill with `allowed_tools=[]` in the coordinator's `list_skills`
response read as "no tools are usable by this skill" to a model that
didn't know the semantics — but the actual meaning is "no tools are
pre-approved for auto-approval (auto-approve exemption list)".  Real
misdiagnosis incident: a code-review child appeared to have been
spawned with zero tool access when in fact the skill simply hadn't
declared an auto-approve allowlist.

Two-part fix:
- `coordinator_client.list_skills` omits the `allowed_tools` key from
  the per-skill dict when empty.  Absence now carries the unambiguous
  meaning "no tool is pre-approved for this skill"; presence (with a
  non-empty list) keeps the standard Claude Code skill-spec shape.
- `turnstone/tools/list_skills.json` description rewrites the field
  doc so the LLM sees: "tool names exempt from the operator approval
  gate ... the field is OMITTED when empty: a skill without
  `allowed_tools` still has access to every tool in its session's
  toolset; absence of the field means no tool is pre-approved for
  this skill, not that the skill has no tools."

Field name stays `allowed_tools` — matches the upstream Claude Code
skill frontmatter (`allowed-tools` hyphenated, stored as
`allowed_tools` internally per `skill_parser.py:241-242`).  Parser,
storage column, admin UI, and SDK unchanged.
2026-05-18 19:39:30 -07:00
Patrick Buckley 687a3367c0 fix(judge): explicit user_decision vocabulary (no more empty strings)
Auto-approved tool calls left intent_verdict rows with `user_decision=""`,
indistinguishable from rows still pending manual review.  Real misdiagnosis
incident: a coord with `recommendation="review"` and `user_decision=""` was
read as "stuck waiting for approval" when in fact the tools had been
auto-approved and the child was running normally.

New vocabulary at the storage API boundary (column server_default stays
`""` so pre-fix legacy rows are still distinguishable as such):

- `pending`           — at insert, before any resolution
- `approved` / `denied` — manual user resolution
- `timeout`           — approval-event timeout (split from `denied` so the
                        audit column alone tells them apart; the feedback
                        string used to carry this distinction)
- `policy` / `blanket` / `skill` / `always` / `auto_approve_tools` —
                        auto-approve reasons (mirror `AutoApproveReason`)

Heuristic verdicts on the two auto-approve early-return branches are now
persisted with `user_decision=<reason>` (previously dropped on the floor).
Late LLM verdicts for already-auto-approved call_ids look up the reason via
a TTL-pruned `_auto_approve_reasons` map (lazy 60s prune at write time, so
no fixed cap can silently regress the fix on the N+1th auto-approve; LLM-
disabled sessions don't leak entries because prune fires whenever auto-
approves happen).

Bug fixes caught during review:
- `on_intent_verdict` early-returns when the verdict already carries an
  auto_reason — without this, a manual `resolve_approval` on a mixed batch
  would overwrite the auto-stamped row with `approved`/`denied`.
- `_record_auto_approves` runs BEFORE `_persist_auto_approved_heuristic_*`
  so the lookup map is populated before any concurrent LLM verdict can
  fire and miss it.
- `resolve_approval(timeout=True, approved=True)` now raises ValueError
  to make the split-brain shape unrepresentable.
- Approval-timeout feedback string derives from `_APPROVAL_WAIT_TIMEOUT`
  rather than the hardcoded "1 hour".
2026-05-18 19:39:30 -07:00
renovate[bot] bfd99c6a81 chore(deps): lock file maintenance 2026-05-18 19:39:30 -07:00
github-actions[bot] 1a813c8130 chore: download vendored JS files 2026-05-18 19:39:30 -07:00
renovate[bot] d6aa85db6d chore(deps): update dependency katex to v0.16.47 2026-05-18 19:39:30 -07:00
renovate[bot] c0c7fda7f9 chore(deps): lock file maintenance 2026-05-18 19:39:29 -07:00
Patrick Buckley 7fae2698d7 docs(storage): clarify LIKE_ESCAPE contract with .like(escape=...)
The previous comment described "\\" as "non-default", which is
backwards — "\\" is the SQL standard escape character.  The
actually-non-default part is SQLAlchemy's ``.like()`` itself: it
defaults to no escape character, so ``escape_like``'s output is only
interpreted correctly when callers pass ``escape=LIKE_ESCAPE``
explicitly.  Reword to put the caller-side requirement first.
2026-05-18 19:39:29 -07:00
Patrick Buckley 4bbe64755e fix(watch): deliver terminal fires instead of dropping them silently
WatchRunner._poll_watch committed active=False to the row BEFORE
calling _dispatch_result for a terminal fire, and the dispatch closure
registered by ChatSession.set_watch_runner enqueued each reminder with
a valid_until=is_watch_active predicate that re-read the row at drain
time. Since the runner already flipped active to 0, the predicate
returned False for every dispatched fire and NudgeQueue.drain silently
dropped the entry — the model never saw a watch result. Then a
subsequent action=cancel call hit list_watches_for_ws (filters
active==1), the now-inactive row was invisible, and the cancel
returned 'Watch "X" not found.' regardless of whether the watch had
actually run.

Reorder _poll_watch to dispatch before the row write, drop the
valid_until predicate from the watch closure (its only effect was the
bug above), and add a _terminal_dispatched guard on the runner so a
transient storage failure between dispatch and row-write doesn't
re-fire the reminder on the next tick. Add WatchRunner.forget_terminal_dispatched
and call it from the cancel path so an out-of-band deactivate (next_poll='')
doesn't leak the watch_id from the runner's pending-retry set indefinitely.

Cancel-by-name now routes through a new find_watch_by_name storage
method that ignores the active filter and prefers active rows over
newer-inactive same-name siblings. The session.py cancel branch
distinguishes 'already completed (auto-cancelled)' from 'not found'
so the model can tell apart 'this watch ran and finished' from
'no such watch.' Consolidate the two byte-identical _escape_like
/ _escape_ilike helpers in the storage backends into a single
turnstone.core.storage._utils.escape_like and apply it to the new
find_watch_by_name LIKE pattern so a model-supplied watch name
containing % or _ can't redirect a cancel to a sibling watch.

NudgeQueue.drain previously dropped predicate-failed entries without
logging anything, which is what hid this bug for so long. Drain now
emits nudge_queue.predicate_dropped: info for reason=predicate_false
(the normal lifecycle case — idle_children when every active child
finished between enqueue and drain), warning with exc_info for
reason=predicate_raised (a misbehaving predicate).

Tests: new test_poll_watch_terminal_fire_survives_drain (parametrized
stop_on_fired + max_polls_reached) drives the real WatchRunner._poll_watch
against a real tmp_db row and confirmed to fail against pristine main.
test_poll_watch_retry_deactivate_after_update_watch_failure exercises
the _terminal_dispatched retry-deactivate branch end to end.
test_cancel_clears_pending_terminal_dispatched_entry covers the cancel-
path leak case. test_find_by_name_prefers_active_over_newer_inactive
catches the ordering regression. test_find_by_name_treats_percent_as_literal
+ test_find_by_name_treats_underscore_as_literal pin the LIKE escape.
2026-05-18 19:39:29 -07:00
Patrick Buckley c1281b9721 chore: bump version to 1.5.16 2026-05-13 16:13:01 -07:00
Patrick Buckley 3124dbe52f fix(vendor): widen update-vendored-js sweep to catch shared_static/ + .py
The shared_static exclude in scripts/update-vendored-js.sh was meant to
skip self-references inside vendored libraries, but it also hid
shared_static/renderer.js — which loads the vendored libs and pinned
mermaid-11.14.0 across every renovate bump since #426. Tests under
tests/test_web_helpers.py were similarly invisible because the include
list omitted *.py.

Replace the broad shared_static exclude with the specific old-versioned
vendor directory (about to be rm -rf'd next anyway), and add *.py to the
include list. Bump renderer.js to mermaid-11.15.0 to repair the live
404, and refresh the test fixtures to current vendor versions so they
stop drifting.
2026-05-13 16:12:40 -07:00
renovate[bot] a8a1e738ca chore(deps): lock file maintenance 2026-05-13 15:50:14 -07:00
github-actions[bot] 27768abecc chore: download vendored JS files 2026-05-13 15:50:14 -07:00
renovate[bot] 1e5017293a chore(deps): update vendored js 2026-05-13 15:50:14 -07:00
renovate[bot] b2c688f3b1 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.14 2026-05-13 15:50:14 -07:00
renovate[bot] 09b1f07e18 chore(deps): update dependency vitest to v4.1.6 2026-05-13 15:50:14 -07:00
Patrick Buckley 1de8f6b4c7 chore: bump version to 1.5.15 2026-05-12 21:21:15 -07:00
Patrick Buckley 4f89c1c3f3 docs(changelog): release 1.5.15 notes
Fixes the 1.5.14 admin.js syntax error that left the console UI
non-functional whenever an MCP server row had consented users.
2026-05-12 21:21:08 -07:00
Patrick Buckley 6898860f18 fix(mcp): admin.js syntax error in bulk-revoke button (Phase 9)
Line 3446 used double-quote string delimiters with an embedded ">
that terminated the string mid-attribute, leaving "bulk-revoke (" as
bare tokens.  The rest of the surrounding block uses single-quote
delimiters; switch the broken line to match so the embedded > and "
sit safely inside the string.

The parse error wiped out every global in admin.js, so showAdmin and
the rest of the admin entry points were undefined — the console was
non-functional whenever an MCP server row had consented_users_count > 0.
2026-05-12 21:20:45 -07:00
Patrick Buckley 3d07cad272 chore: bump version to 1.5.14 2026-05-12 21:17:35 -07:00
Patrick Buckley 3737ddf89f docs(changelog): release 1.5.14 notes
Backports OAuth-MCP Phase 9 (#516) to the stable/1.5 track. Introduces
forward-only migrations 054_mcp_pending_consent and
055_mcp_user_tokens_server_index.
2026-05-12 21:17:26 -07:00
Patrick Buckley 464450b9e2 feat(mcp): admin status, deferred-consent persistence, operator docs (Phase 9) (#516)
* feat(mcp): admin status, deferred-consent persistence, operator docs (Phase 9)

Completes the OAuth-MCP build-out (Phases 0-8 shipped) by closing the
operator + deferred-consent gaps:

1. **Per-(user, server) deferred-consent persistence** — when a
   non-interactive run (scheduled / channel) hits ``mcp_consent_required``
   or ``mcp_insufficient_scope``, the sync pool dispatchers now upsert a
   row into a new ``mcp_pending_consent`` table.  The dashboard hydrates
   the gear-icon badge from this table on load, so users who weren't
   online to see the in-flight SSE prompt still surface the deferred
   work on next login.  Cleared automatically by the OAuth callback
   handler on consent completion; manual user dismiss via new DELETE
   endpoints.  Composite PK ``(user_id, server_name)`` collapses repeat
   occurrences for the same server — no NULLs-not-distinct trap.

2. **Admin status pill + bulk-revoke** — the MCP Servers admin row now
   shows ``consented_users_count`` for ``auth_type=oauth_user`` rows
   when ≥1, with a two-step-confirm ``bulk-revoke`` button that drops
   every user's token for the server via the existing
   ``delete_mcp_oauth_rows_by_server_name`` primitive.  Upstream RFC
   7009 revoke is intentionally NOT attempted in bulk (avoids N
   upstream HTTP calls per admin click); audit detail records
   ``upstream_revoke_outcome=bulk_admin_no_upstream``.  A "last
   refresh" pill (age + outcome) renders on each row, sourced from a
   new ``_last_refresh`` dict populated by ``_refresh_server`` on every
   call (both manual ``refresh_sync`` and the ``_cb_auto_reconnect``
   follow-up).

3. **ClientType.SCHEDULED** added to the prompts module + scheduler
   passes it through to ``create_workstream``.  ``ChatSession`` now
   computes ``_is_interactive_for_consent`` at construction (WEB / CLI
   are interactive; CHAT / SCHEDULED are not) and plumbs the flag
   through ``call_tool_sync`` / ``read_resource_sync`` /
   ``get_prompt_sync`` to the three sync dispatchers.  The wrap at the
   ``_is_structured_error`` gate routes consent codes to the new
   ``_record_pending_consent_best_effort`` helper for non-interactive
   callers only; interactive sessions stay on the in-flight SSE path
   Phase 8 ships unchanged.

4. **Operator docs** — ``docs/mcp-oauth.md`` (operator guide, parallel
   to ``docs/oidc.md``: ``auth_type`` choice, OAuth client setup,
   encryption-key rotation, troubleshooting matrix) and
   ``docs/operations/mcp-oauth-headless.md`` (one-paragraph runbook
   per ``feedback_runbook_trust_llm.md``: pre-consent recipe for
   scheduled / channel-driven runs).

Schema
- Migration 054_mcp_pending_consent.py — composite PK
  ``(user_id, server_name)``, ``occurrence_count`` + ``first_seen_at`` /
  ``last_seen_at`` for recency metadata, ``idx_mcp_pending_consent_user``
  for the badge-load query.  No FKs (matches the rest of the
  oauth_user schema).
- Migration 055_mcp_user_tokens_server_index.py — adds
  ``idx_mcp_user_tokens_server`` on ``(server_name, expires_at)`` so
  the admin pill's ``count_mcp_consented_users_*`` queries don't
  full-scan against the leading-``user_id`` composite PK.
- Cross-backend: works on SQLite + PostgreSQL via dialect-specific
  ``on_conflict_do_update`` (PG ``postgresql.insert`` / SQLite
  ``sqlalchemy.dialects.sqlite.insert``).  No ``NULLS NOT DISTINCT``
  needed — the simplified PK eliminates the cross-version trap.

Endpoints
- ``GET /v1/api/mcp/oauth/pending`` — list deferred-consent records for
  the authenticated user.  Install-level gate via cached
  ``any_oauth_user_mcp_servers`` short-circuits to ``{pending: 0}`` on
  installs with no oauth_user MCP servers — local-auth deployments
  exercise zero new storage queries on this path.  The gate result is
  cached on ``app.state`` with a 60s TTL to spare repeat dashboard
  loads.
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` — single dismiss.
  Returns 204 in both existed-and-deleted and never-existed cases
  (no cross-tenant existence leak); audits
  ``mcp_server.oauth.pending_consent_dismissed`` with
  ``mode=single`` + ``cleared=0|1`` so a session-hijack attacker
  scrubbing breadcrumbs leaves an audit trail.
- ``DELETE /v1/api/mcp/oauth/pending`` — bulk dismiss; audits
  ``mode=bulk`` + ``cleared=N``.
- ``POST /v1/api/admin/mcp-servers/{name}/bulk-revoke`` — admin
  bulk-revoke for the named server's per-user tokens.  Requires
  ``admin.mcp`` permission + 400s when the row isn't ``oauth_user``.

All four registered on both ``turnstone-server`` and
``turnstone-console`` (mirrors the Phase 8 ``/connections`` endpoint
shape).

Performance
- Admin list handler now uses a single ``GROUP BY`` bulk-count query
  (``count_mcp_consented_users_grouped_by_server``) wrapped in
  ``asyncio.to_thread`` rather than N per-row sync DB round-trips
  inside the async handler.  Skipped entirely when no row is
  oauth_user.

Frontend
- ``ui/static/app.js``: ``loadPendingConsents()`` hydrates the
  existing ``_pendingConsentServers`` set on dashboard init + after
  the user opens the settings modal.  Endpoint failures stay silent
  — the badge will be re-driven by the next in-flight tool error.
- ``console/static/admin.js``: ``consented_users_count`` pill +
  ``bulk-revoke`` button on each MCP row (only when ≥1 consented),
  two-step confirm matching the existing delete pattern.  ``last-
  refresh`` age + outcome pill in the per-row status cell, sourced
  from the freshest per-node entry in ``status[*].last_refresh_at`` /
  ``last_refresh_outcome``.  CSS for the pills in ``style.css``.

Tests
- ``test_mcp_pending_consent_storage`` — 13 tests covering upsert
  idempotency, list ordering, per-user isolation, single/bulk delete,
  count-by-server + grouped variant, install-level gate.
- ``test_mcp_pending_consent_dispatch`` — 9 tests, including the
  boundary-cross gate per ``feedback_tests_through_boundaries.md``:
  drives the real ``call_tool_sync`` → ``_dispatch_pool_sync`` →
  ``_is_structured_error`` → ``_record_pending_consent_best_effort``
  with a mocked classified-lookup so the structural plumb-through is
  verified end-to-end.  Includes a storage-failure test that pins
  the docstring's "envelope unchanged on storage failure" promise.
- ``test_mcp_pending_consent_endpoints`` — 11 tests: install gate,
  list-for-self, no-cross-user-leak, single/bulk delete, idempotent
  not-found, audit emission on single + bulk + cross-tenant dismiss.
- ``test_chat_session_interactivity_flag`` — 7 tests pinning the
  ``ClientType`` → ``_is_interactive_for_consent`` mapping against
  the module-level ``INTERACTIVE_CONSENT_CLIENT_TYPES`` frozenset.
- ``test_mcp_admin_bulk_revoke`` — 7 tests covering admin.mcp
  permission gate, 404 on missing, 400 on non-oauth_user, 200 with
  ``rows_deleted`` + ``consented_users_before``, audit row with
  ``upstream_revoke_outcome=bulk_admin_no_upstream``, cross-server
  isolation.
- ``test_mcp_oauth_handlers`` — 2 new callback tests pin the post-
  callback ``delete_mcp_pending_consent`` invocation: success-clears
  + storage-failure-still-redirects.
- 636 tests pass on the impacted surface (47 new + Phase 0-8 OAuth-MCP
  + session + prompts + storage admin).  ruff + mypy clean.

Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}`` — the
  flag flows only through the pool dispatchers, which only fire when
  the row resolves to ``oauth_user``.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) preserved on every
  AS / SDK / pool-loop await — no new awaits added to the hot path.
- Install-level gate on the badge endpoint: cached
  ``any_oauth_user_mcp_servers`` returns False on a row-less
  deployment → endpoint short-circuits without touching the pending-
  consent table; 60s TTL bounds the staleness window after admin
  flips ``auth_type``.
- Operator-actionable codes (key-unknown, url-insecure, *_forbidden)
  explicitly filtered out of persistence — they're outside the
  user-facing consent badge scope.
- Best-effort write: the structured-error envelope returned to the
  agent is identical whether the persistence write succeeds or fails
  (storage exception is logged with type name only — no chained
  context that could carry an ``httpx.Request`` bearer header).
- No ``exc_info=True`` on any new path that can chain a bearer-bearing
  ``httpx.Request``.
- Defensive parsing: ``_parse_pending_consent_envelope`` mirrors
  ``_is_structured_error``'s ``isinstance(decoded, dict)`` guard plus
  filters scope tokens through ``is_valid_scope_token`` capped at
  ``MAX_INSUFFICIENT_SCOPE_REPORTED`` — defense-in-depth even though
  production callers already validate upstream.
- Audit events on every dismiss endpoint so a session-control attacker
  scrubbing dashboard breadcrumbs still leaves a trail.

Cross-backend
- Tested on SQLite via the conftest backend fixture.
- PostgreSQL path uses ``postgresql.insert(...).on_conflict_do_update``
  parallel to the existing ``mcp_user_tokens`` upsert in Phase 3.

Deferred (not Phase 9 blockers)
- Multi-node pool eviction on bulk-revoke: only local-node sessions
  would be evicted if we built it, and there's no bulk-by-server
  primitive on MCPClientManager today; remote nodes will surface as
  a 401 on next dispatch which refreshes through the (now empty)
  token row.
- RFC 8693 / Azure OBO ``auth_type=oauth_token_exchange`` — captured
  in the design doc as a future architectural direction (~600 LOC +
  IdP-side admin work); requires OIDC token capture and per-MCP-server
  resource-trust configuration that v1 does not ship.

* docs(mcp): address Copilot review feedback on Phase 9

- Fix misleading admin.js comment that claimed the refresh pill rendered
  "<short-relative> <outcome>" — the pill actually renders only the short
  age, with outcome reflected via CSS class and tooltip.
- Replace broken feedback_secrets_not_in_env.md repo-root link in
  mcp-oauth.md with the inlined rationale (env-borne secrets reachable
  via shell tools / os.environ; TOML secrets are not).
2026-05-12 20:55:01 -07:00
Patrick Buckley 1fba80a9b5 chore: bump version to 1.5.13 2026-05-11 21:24:09 -07:00
Patrick Buckley 28a5914be0 docs(changelog): release 1.5.13 notes
Adds notes for the 14 patches cherry-picked to stable/1.5 since 1.5.12:
reactive PG LISTEN/NOTIFY node discovery + event-driven wait_for_workstream,
memory tool audit trail, task_agent skill personas, plus fixes for the
LLM-visible default alias bypass, mermaid streaming parse errors,
proxy-prefixed re-auth, dashboard appbar visibility, and the PG test
backend on the notify dispatcher suite.

Introduces forward-only migration 053_services_notify_trigger.
2026-05-11 21:23:53 -07:00
Patrick Buckley bb2515eddf fix(task_agent): address Copilot feedback on skill parameter
- Put ``skill`` back in the access-denial list in the tool
  description with a clarification — TASK_AGENT_TOOLS does not
  include the skill tool, so sub-agents cannot switch personas
  mid-task.  Removing the disclaimer entirely created an ambiguity
  the LLM could misread.

- Minimize the skill_data carried on the approval item dict to
  ``name`` / ``content`` / ``risk_level`` only.  ``get_skill_by_name``
  returns the full ~30-column prompt_templates row including
  ``scan_report``, ``installed_by``, ``source_url`` — none of those
  flow through ``_exec_task`` / ``_evaluate_intent``, and they
  shouldn't ride along any future audit serializer that reads the
  approval item shape.

- Regression test for ``skill=""``, whitespace-only, and ``\t\n``
  values — pins the documented "empty value is acceptable" contract
  at the ``(args.get("skill") or "").strip()`` chokepoint.
2026-05-11 21:18:48 -07:00
Patrick Buckley 19978bfd89 feat(task_agent): add optional skill parameter for per-call personas
The task_agent tool now accepts an optional ``skill=<name>`` argument
that loads the named skill's content as the sub-agent's persona,
substituting the hardcoded "# Task Agent" identity statement.  The
operating-guidance numbered list (one-shot, tool-use over narration,
no follow-up questions) is layered on top of every persona and always
applies — those are sub-agent semantics that a persona should ride on
top of, not replace.

Validation lives in ``_prepare_task`` so the approval surface tells
the operator what they're consenting to: the validated skill dict
(including content) rides on the item dict from prepare to exec to
defeat TOCTOU between consent and execution.  An unknown skill
returns a clean error item with a hint pointing at
``skill(action='search')``; a disabled skill returns a distinct error
so the LLM's recovery path can tell "not found" from "quarantined",
mirroring the enabled gate that ``_exec_skill(action='load')`` and
skill-search already apply.

High and critical skills now surface their risk tier on the approval
header (``, risk: critical``) and emit a
``task_agent.high_risk_skill`` warning — same signal ``_load_skills``
emits for session-level skills, so the operator sees the same flag
whether the skill is loaded session-wide or per-call.  ``_exec_task``
emits a ``task_agent.skill_invoked`` info log on the skill branch for
forensic traceability — the approval row captures the choice at
consent time, this log captures it at exec time so post-incident
search doesn't have to cross-walk approval and exec tables.

The ``_evaluate_intent`` func_args projection now includes the skill
name — without it, heuristic ``arg_pattern`` rules targeting a risky
persona name on ``task_agent`` silently no-op and the audit row loses
the choice.  Mirrors the long-standing ``spawn_workstream``
projection.
2026-05-11 21:18:48 -07:00
Patrick Buckley 8eec44d809 fix(ui): attach settings menu keydown synchronously
Caught by Copilot on PR #514.  openSettingsMenu sets _settingsMenu
synchronously, but the menu's keydown handler was registered inside
setTimeout(0).  The previous-commit guard in the global keydown
handler returns early when _settingsMenu is set (so dashboard isn't
hidden by Escape over the menu), which created a window where
Escape had no handler at all — the global skipped, the menu's own
listener wasn't ready yet, and the menu got stuck open until the
next interaction.

Attach keydown synchronously; keep mousedown + initial focus in
setTimeout (mousedown to avoid the opening click triggering its own
outside-click close, focus because the menu DOM needs a tick to
settle layout).
2026-05-11 21:18:48 -07:00
Patrick Buckley f1cf516eb6 fix(ui): keep appbar visible on dashboard, gear-icon dropdown menu
Two related changes that surfaced when the user pointed out the proxy's
node-picker pill was unreachable from the proxied dashboard view: the
dashboard overlay was covering the entire appbar.

  - Dashboard overlay now starts at top: 48px so the appbar (with the
    proxy-injected node picker) stays visible and interactive while the
    dashboard is open.  showDashboard no longer marks ui-header inert
    (tab-bar and split-root still are).  The dashboard's role downgrades
    from dialog+aria-modal to region — the appbar being reachable above
    it would otherwise contradict aria-modal's "ignore everything else"
    semantics.

  - Gear icon converts from a direct openSettingsPanel() click into a
    dropdown menu with two items: "MCP connections" (existing modal) and
    "Logout".  Reuses the .ws-tab-dropdown shell for visual consistency
    with the workstream tab chevron menu and the proxy node-picker.
    Logout uses .destructive styling to reduce misclick risk.

Bug fixes caught by the merged code-review pipeline:

  - Global Escape handler skips when _settingsMenu is open, otherwise it
    fires hideDashboard() before the menu's own handler — wiping the
    composer text + staged attachments out from under the user.
  - Menu-item click refocuses the trigger before close, so
    openSettingsPanel captures the gear (not <body>) as the eventual
    return-focus target.
  - ArrowUp keyboard cycling uses idx <= 0 ? len - 1 : idx - 1 instead
    of (idx - 1 + len) % len so the no-focus case wraps to the last
    item rather than the second-to-last.  Same fix backported to
    showTabDropdown which had the identical modulo bug.
  - Position clamps reordered: right-edge override now runs before the
    left-edge floor so a menu wider than the viewport still clamps to
    mx >= 4 instead of going negative.
  - openSettingsMenu caches _settingsMenuTrigger so closeSettingsMenu
    can reset ARIA without re-querying the gear by id.
  - aria-controls lifecycle wired both ways (set on open, removed on
    close).
2026-05-11 21:18:48 -07:00
Patrick Buckley 1df1e739ef fix(session): prevent LLM bypass of per-role plan/task model overrides
The LLM was passing ``task_agent(model="default")`` (and the same for
plan_agent) and routing to whichever backend the auto-created
``default`` alias was attached to at boot — flatspark in the verified
case (ws_id 7dde674) — silently bypassing the operator-configured
``model.task_alias`` / ``model.plan_alias`` (gh200).

Root fix:

- ``load_model_registry`` only synthesises the back-compat ``default``
  alias when neither DB nor ``[models.*]`` populate the registry.  The
  shim was only ever meant for single-CLI-model setups; with a multi-
  model DB it became a phantom routing target aliasing ``LLM_BASE_URL``.
- ``_render_agent_tool_descriptions`` filters ``default`` out of the
  LLM-visible alias list.  The English reading of "default" trips the
  model into picking it explicitly even when the description tells it
  to omit ``model=`` for the per-role default.

Defense-in-depth at the validator chokepoint
(``_validate_agent_model_override``): explicit rejection of
``alias == "default"`` (post-strip) with corrective guidance;
``default`` filtered out of the unknown-alias retry list so an LLM
probing with a bogus alias can't enumerate it back; the no-alternatives
wording is distinguished from the no-registry-configured wording.  The
render path also always rewrites tool descriptions instead of returning
early on filter-empty, so a reload that drops the registry to only
``default`` clears stale alias names left over from a prior render.
2026-05-11 21:18:47 -07:00
Patrick Buckley fbbb21012a feat(audit): emit memory tool save/update/delete events
Previously only the admin-console DELETE route emitted memory.delete
audit rows, so a long-running session whose memory was deleted via
the admin UI had no log trail showing what happened — masking
out-of-band deletes as apparent tool bugs.

The save branch now stamps memory.save (new row) or memory.update
(upsert); the delete branch does a lookup-then-delete-by-id pair so
the audit can record the resolved memory_id and type. All emissions
are best-effort: failures log at debug and swallow so an audit hiccup
never breaks the tool call itself. Reads (get/search/list) remain
un-audited.
2026-05-11 21:18:47 -07:00
Patrick Buckley 617de2488f fix(console): make proxy_api auth dispatch single-sourced
Copilot review on #511 flagged that the dispatch chain and the tests
both claimed to be in lockstep with one another, but only the comment
text said so — the parametrize list and the if/elif chain were two
independent hand-maintained copies, and the comments still referenced
the (long-reverted) ``_PROXY_AUTH_LOCAL_HANDLERS`` symbol.

Make the lockstep guarantee real by collapsing both copies onto one
``_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str]`` mapping
``(method, path)`` to handler-name strings.  ``proxy_api`` resolves
the name through ``globals()`` at call time so ``patch(...)`` in
tests still observes the override — a dict of function refs would
have captured the originals at module load (which is why the first
attempt at this dispatch broke the tests and got reverted).  Test
cases now derive directly from ``_PROXY_AUTH_LOCAL_HANDLERS.items()``,
so adding or removing an entry in the dispatch table flows through
to the parametrize list automatically and the two can't drift.
2026-05-11 21:18:47 -07:00
Patrick Buckley 1d5189bb8d fix(console): allow re-auth from inside the proxy-prefixed UI
When the user is on a proxied node page (``/node/{id}/...``) and the
JWT expires, the in-page login modal POSTs to ``/v1/api/auth/login``
which the proxy shim rewrites to ``/node/{id}/v1/api/auth/login``.
Two latent bugs both had to be fixed for the user to be able to
re-authenticate from inside the proxied UI:

1. ``is_public_path`` didn't recognise the ``/node/{id}/`` prefix
   over a public path, so the console's ``AuthMiddleware`` 401'd the
   login POST before any handler ran.  Extended via the existing
   ``_extract_proxied_path`` helper so a proxied public path stays
   public.

2. Even if the path had been public, ``proxy_api`` would have
   forwarded the request to the upstream node.  The upstream mints
   ``JWT_AUD_SERVER`` tokens; the console's ``AuthMiddleware``
   (expecting ``JWT_AUD_CONSOLE``) would reject those on the next
   proxied call, and ``_proxy_post`` drops ``Set-Cookie`` when
   forwarding anyway.  ``proxy_api`` now dispatches every entry in
   ``_PROXY_AUTH_LOCAL_PATHS`` (login, logout, setup, refresh,
   status, whoami, oidc/authorize, oidc/callback) to the console's
   own auth handlers, and short-circuits non-canonical methods on
   those paths with 405 instead of letting them slip through with
   the service-token fallback.

Tests parametrize across all eight local-dispatch entries so a future
refactor that drops a branch (or routes it through ``_proxy_post``)
fails loudly, plus a no-auth-header reproduction for the original
lockout and a 405 regression guard for the method-mismatch surface.
2026-05-11 21:18:47 -07:00
Patrick Buckley 89a282f86b fix(renderer): mermaid streaming parser errors + progressive hljs (#510)
* fix(renderer): mermaid streaming parser errors + progressive hljs

Live streaming was rendering mermaid diagrams with `Parse error,
got 'PS'` messages — bare `(`, `[`, `{` inside unquoted edge / node
labels re-entered Mermaid's shape parser. Two unrelated streaming-
specific issues in the renderer pile-up here; this commit addresses
both plus a follow-on UX improvement for code highlighting.

## Mermaid label autoquoter

`_normalizeMermaidSource` wraps two label forms that Mermaid rejects
when they contain bare shape-delimiter chars:

  1. Edge labels:  `|content|`  →  `|"content"|`
  2. Rectangle node labels:  `ID[content]`  →  `ID["content"]`

Shapes whose syntax already nests delimiters — cylinders `[(...)`,
subroutines `[[...]]`, trapezoids `[/.../]` `[\...\]`, circles
`((...))`, hexagons `{{...}}`, diamonds `{...}` — are intentionally
left alone (their inner delimiters are part of the shape syntax;
quoting would corrupt them). Labels already wrapped in `"..."` are
also left alone. The rewrite is idempotent and runs before the
mermaid SVG cache lookup so identical malformed input hits the
cache on re-render rather than re-quoting per tick.

## Markdown fence-pair regex

The old fence regex `/(```+)([^\s`]*)\n([\s\S]*?)\1/g` would, mid-
stream, pair an unclosed ```mermaid open with the OPENING backticks
of a later ```python fence as the "close", handing mermaid a
truncated source. New regex:

    /(```+)([^\s`]*)\n((?:(?!\1)[\s\S])*?)\1[ \t]*(?=\n|$)/g

Two constraints close the gap:

  - `(?!\1)` inside the content quantifier blocks the lazy matcher
    from extending across another N-backtick run. Smaller inner
    counts (e.g. 3-backtick inner inside a 4-backtick outer) still
    pass since `\1` is the open's actual count.
  - `[ \t]*(?=\n|$)` after `\1` forces the close to a line
    boundary, so ```python (open with a language tag) can't
    masquerade as a previous fence's close.

Together: an unclosed fence stays as plain markdown until its true
close arrives, so neither mermaid nor hljs ever sees a mid-stream
truncated source.

## Progressive hljs

Extracted `postRenderHljs` from `postRenderMarkdown` with a source-
keyed `_hljsCache` (FIFO, cap 64, keyed on `language:source`) and
wired it into `_streamingRenderApply`. Closed code fences are now
syntax-highlighted as they stream in, matching the progressive
mermaid pattern from #426. Per-tick cost stays cheap because the
cache returns the pre-tokenized HTML synchronously on hit; only
unique (language, source) pairs pay `hljs.highlightElement`.

## Internal cleanup from the review pipeline

  - `_cacheFifoEntry(cache, key, value, max)` replaces the duplicated
    `_cacheHljsEntry` and `_cacheMermaidEntry`. Single tested
    implementation across four caches (hljs, mermaid svg, mermaid
    error, mermaid normalize memo). The "don't evict on overwrite"
    invariant is pinned per-cache in tests.
  - `_mermaidNormalizeCache` memoizes raw textContent → normalized
    output so the per-rAF-tick autoquoter split + regex doesn't
    repeat for unchanged diagrams. Eviction shares
    `_MERMAID_CACHE_MAX` with the SVG cache it feeds.

## Tests

The fake DOM in tests/test_renderer_js.py grew a few capabilities
to drive these paths:
  - `classList` is now array-like (length + indexed access) so the
    hljs language-extraction loop works.
  - `textContent` setter mirrors the real-DOM side effect of
    entity-escaping into innerHTML, so `escapeHtml()` round-trips
    (otherwise every `renderMarkdown` returns empty `<p>` tags).
  - `querySelectorAll` handles both `pre code.language-mermaid`
    and `pre code[class*='language-']`.

Added: 6 fence-pairing regression cases, 9 hljs-progressive cases
(cache hit / distinct sources / language separation / NO_HIGHLIGHT
langs / terminal class / eviction / overwrite / postRenderMarkdown
wraps hljs / _streamingRenderApply invokes hljs), 11 autoquoter
cases including both diagram sources from the live screenshot
encoded verbatim as parametrized regressions, and 3 normalize-memo
cases (populates on first call, consulted before normalize via
sentinel pre-seed, distinct sources cache separately).

Total: 104 renderer tests pass (was 67).

* fix(renderer): apply Copilot review feedback on #510

Two doc / harness adjustments from the PR review — no behavior
change in production code.

- The `_mermaidNormalizeCache` comment claimed eviction "stays in
  lockstep with the SVG cache". That was misleading: the two
  caches key on different things (raw textContent vs normalized
  source) and evict independently. Updated the comment to describe
  what they actually share (the cap, for memory footprint) and
  what they don't (positional coupling), and to note that the memo
  deliberately survives `_initMermaid` since normalize output is
  theme-independent.

- The fake DOM in tests/test_renderer_js.py had `innerHTML` setter
  clear `children` but leave `_textContent` intact, so subsequent
  `textContent` reads could return stale data after an innerHTML
  mutation (real DOM invalidates textContent on innerHTML write).
  No current test triggered this, but it would mask future bugs
  that depend on innerHTML/textContent consistency. Setter now
  clears `_textContent`; the children-derived fallback in the
  getter returns `''` after the wholesale replace.

All 104 renderer tests still pass; ruff + mypy clean.
2026-05-11 21:18:47 -07:00
renovate[bot] 59c943b83a chore(deps): lock file maintenance 2026-05-11 21:18:47 -07:00
Patrick Buckley 14b7516b3f fix(notify): unbreak PG test backend on the notify dispatcher suite
CI's postgres-backend run failed 11 of the new notify tests from #505.
Three independent issues:

1. Migration 053's ``services_notify`` trigger lives only in the
   alembic chain, but the test fixture in conftest.py calls
   ``init_storage(..., run_migrations=False)`` for speed.  That path
   skips migrations and relies on ``metadata.create_all`` for the
   table tree.  Previous alembic-only DDL (migrations 041 / 048
   ``CREATE INDEX CONCURRENTLY`` on workstreams) is performance-only,
   so tests never depended on it.  053's trigger is the first
   behaviorally-required alembic-only DDL in the project — without it
   ``register_service`` doesn't fire NOTIFY and the trigger-filter
   tests time out.

   Fix: declare the trigger function + trigger in ``_schema.py`` and
   attach them via ``sa.event.listen(services, "after_create", ...)``
   DDL events, gated on ``dialect == "postgresql"``.  The same SQL
   constants are imported by migration 053 so there's a single source
   of truth.  Test fixture stays unchanged — ``create_all`` now
   installs the trigger on fresh PG test DBs.  Migration covers the
   upgrade-on-existing-DB path; the two are mutually exclusive given
   ``create_tables = not run_migrations`` in ``init_storage``.

2. NotifyDispatcher tests fired ``storage.notify(...)`` immediately
   after ``d.start()`` and hit a race: the listener thread is
   concurrently calling ``psycopg.connect(listen_url)`` + ``LISTEN
   <channel>`` over the network, so the notify can land before any
   session is listening on the channel and PG drops it (pg_notify
   only routes to sessions LISTEN'ing at COMMIT time).

   Fix: dispatcher gains a ``_listener_ready: threading.Event`` set
   inside ``_listener_loop`` after each successful ``storage.listen``
   open and cleared on disconnect, plus a public
   ``wait_until_ready(timeout)`` method.  Tests use a new
   ``_start_ready(d)`` helper that calls ``start()`` + asserts ready.
   Production callers don't need this (real reactive traffic arrives
   well after startup), but it's the right primitive for any future
   "start dispatcher, immediately send" call site too.

3. ``TestSqliteNotify`` is misnamed — its tests run against whichever
   backend the ``storage`` fixture provides (PG by default in CI).
   Two of its assertions were SQLite-specific:
   ``assert got.pid == 0`` only holds for the synthetic in-process
   path (PG carries real backend PIDs), and
   ``test_synthetic_sweep_emits_after_interval`` is fundamentally
   SQLite-only (no sweep on the PG path).

   Fix: drop the pid assertion (channel + payload are the
   backend-agnostic invariants), add an ``_is_sqlite`` fixture mirror
   of ``_is_postgres``, and gate the sweep test on it.  The sweep
   test also moves from monkey-patching ``stream._sweep_interval`` to
   passing the ``sweep_interval`` kwarg that ``SQLiteBackend.listen``
   now accepts (from the earlier Copilot review fix).

Validated locally against a fresh ``turnstone_test`` PG DB: 263
storage + console + notify tests pass on PG, 257 on SQLite, mypy +
ruff clean.
2026-05-11 21:18:47 -07:00
Patrick Buckley 5d0ec99449 feat(console): event-driven wait_for_workstream + idle cleanup via ChildEventBus
Retire two polling patterns in coord that have clean event sources.
PR 2 of 3 in the coord-completion stack; sits on top of PR #505
(reactive node discovery via PG LISTEN/NOTIFY).

`wait_for_workstream` (coord's block-wait tool) polled storage every
0.5 s in a worker thread regardless of whether anything had changed —
a 600 s wait incurred ~2400 round-trips. Now subscribes to a new
in-process `ChildEventBus` (`turnstone/core/child_event_bus.py`) and
blocks on `threading.Event.wait(min(remaining, WAIT_HEARTBEAT_INTERVAL))`:

- `CoordinatorAdapter` owns the bus; `_dispatch_child_event` calls
  `bus.notify(child_ws_id)` after each `_enqueue_on_ui` for the
  state-class branch (cluster_state, ws_closed, ws_rename,
  intent_verdict, approval_resolved, approve_request).
- Wait loop clears the Event BEFORE the storage snapshot to close
  the subscribe/check race; a notify between clear and the next
  `wait()` leaves the Event set so the loop re-reads without
  losing the wake-up.
- 2 s heartbeat cap preserves the existing `wait_progress` SSE
  cadence for the sidebar UI while cutting SSE traffic ~4x vs the
  pre-bus 500 ms cadence in the quiescent case.
- Worst-case completion latency is 2 s (vs pre-bus 0.5 s) because
  `set_state` buffers non-ERROR writes through `StateWriter`
  (async-flushed) while `emit_state` fans out immediately — a
  bus-driven wake can beat the flusher and read pre-transition
  state, then re-block until heartbeat. Deliberate trade-off; the
  SSE-traffic reduction outweighs the regression on the most
  common terminal transition.
- Defense-in-depth: ownership-filter `cleaned` to own-subtree
  before `register_waiter` so a foreign ws_id passed by an
  untrusted coord LLM (prompt injection) can't observe wake-up
  timing as a side channel. Predicate (`_row_in_own_subtree`)
  requires both `parent_ws_id == coord_ws_id` AND `user_id ==
  coord_user_id` parity — same gate strength as the existing
  `_is_own_subtree` mutating-op guard, so a corrupted /
  cross-tenant `parent_ws_id` alone can't satisfy it. Shared with
  `_snapshot_all` so the snapshot's `denied` shape stays in
  lockstep with the bus filter (Copilot review on #506).

Coord idle-cleanup thread polled the storage scan every
`check_every` seconds (~30 s on default 2 h timeout) even when no
coord was anywhere near idle. Now subscribes to
`SessionManager._state_subscribers` with a `tick_now` event and
blocks on `tick_now.wait(check_every)` — any state change wakes
the sweeper without waiting a full interval, AND the timeout still
fires the periodic sweep for the DB-orphan-only case. A
`min_sweep_interval=5 s` floor bounds DB-call traffic at ~0.2/s
under sustained activity so the loop can't tight-spin `close_idle`
at the rate of its own DB latency (6x improvement over the
pre-refactor fixed 30 s cadence under any activity, and prompt
state-change-driven wakes when below the floor).

`CoordinatorClient` constructor takes `child_event_bus` as a
required kwarg — there's no external SDK shape to preserve and
keeping it optional would silently mask a wiring bug in any future
caller. Tests construct their own `ChildEventBus()` per fixture.

Tests: 16 unit tests for `ChildEventBus` (register / unregister
symmetry, multi-waiter fan-out, multi-child waiter, subscribe/check
race, concurrent register / notify smoke); 7 new adapter tests
(bus notify fires for all 6 state-class events, drops for unknown
child / wrong ws_id); 7 new coord-client wait tests (subscribe-
after-terminal, notify wakes, unrelated notify doesn't wake,
heartbeat fires without notify, unregister on exit, multi-waiter
independence, cross-tenant denial via the user_id-parity filter);
8 idle-cleanup tests (initial sweep, heartbeat cadence, exception
swallowing, stop_event clean exit, state-change wake, subscriber
cleanup, mid-sweep wake, `min_sweep_interval` floor). All pass;
ruff + mypy clean. Full non-live suite: 6227 passed (+2 vs prior
baseline).
2026-05-11 21:18:47 -07:00
Patrick Buckley 6dfd1b5c18 feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher (#505)
* feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher

Add a console-side `NotifyDispatcher` that holds a dedicated PostgreSQL
`LISTEN` connection and fans wake-ups out to per-channel handlers on a
separate dispatch thread. Cluster collector subscribes to a new
`services` channel and runs node discovery reactively — new-node /
graceful-deregister visibility drops from up-to-60 s to ~500 ms on
Postgres, with the 60 s discovery loop retained as the backstop for
crash-shaped node loss (NOTIFY only fires on real writes).

Storage layer gains a uniform `notify` / `listen` API:
- PostgreSQL: real `pg_notify` / `LISTEN` on a dedicated session-mode
  connection that bypasses pgbouncer (mandatory: pgbouncer is required
  in transaction-pool mode per docs, which is incompatible with LISTEN).
- SQLite: in-process fan-out + synthetic-sweep fallback so consumer
  code is identical across backends.

`TURNSTONE_DB_LISTEN_URL` (or `[database] listen_url` in config.toml)
points the dispatcher's connection direct-to-Postgres. Defaults to the
main DB URL when unset.

Migration 053 installs the `services_notify` trigger; it filters
heartbeat-only UPDATEs in-trigger so the 30 s × N-nodes heartbeat tick
stays quiet, while INSERT, DELETE, and url/metadata-changing UPDATE
still fire.

Dispatcher detail:
- Two threads: listener (drains stream → bounded queue) and dispatch
  (invokes handlers under exception suppression). Same-channel notifies
  coalesce per dispatch batch so an N-node deploy burst is one
  `_discover_nodes` per channel.
- Reconnect uses exponential backoff (1 s → 30 s cap). After any
  successful reopen — whether the prior failure was a stream-poll error
  or a connect / initial-LISTEN error — one synthetic Notify with
  payload="reconcile" is enqueued per channel so handlers re-read on
  the same code path they use for real events.

Future consumers (ConfigStore live reload, scheduler immediate
dispatch, audit live-tail) plug in by adding their channel to the
dispatcher's construction list.

Tests: 22 dispatcher tests (incl. reconnect + coalescing under stub
storage), 7 SQLite notify-stream tests, 4 PG-gated trigger-filter
tests, 4 collector wire-in tests. All pass; ruff + mypy clean.

* fix(notify): address Copilot review on #505

- _sqlite.py: SQLiteBackend.listen() now de-dupes channel names via
  dict.fromkeys before constructing the stream — duplicates would
  otherwise register the queue twice and double-deliver each notify.
- _sqlite.py: SQLiteBackend.listen() gains a keyword-only sweep_interval
  parameter (defaults to _SQLITE_NOTIFY_SWEEP_INTERVAL) — matches what
  the comment at the constant already promised, and lets future
  consumers without their own polling timer pick a tighter cadence
  without reaching into private stream attributes.
- _sqlite.py: documented the `except queue.Empty: pass` end-of-drain
  termination so it's not mistaken for swallowing an unexpected error.
- _postgresql.py: docstring referenced :func:`_pg_listen_url` which
  was renamed to _resolve_pg_listen_url during PR development.
- notify_dispatcher.py: module docstring referenced a non-existent
  _bootstrap_console_subsystem; wire-in is at console/server.py::main.

Refuted (no change, false positives from github-code-quality bot):
- 4× "Statement has no effect" on Protocol-method `...` ellipsis bodies
  (idiomatic Python Protocol declaration, not dead code).
- 2× "Mixed import style" in tests — `import ... as nd_mod` is
  intentional to allow attribute assignment for monkey-patching the
  module's `_RECONNECT_BACKOFF_INITIAL` constant inside try/finally.
2026-05-11 21:18:47 -07:00
renovate[bot] 658c65aee8 chore(deps): lock file maintenance 2026-05-11 21:18:47 -07:00
Patrick Buckley 641ce8e7f6 docs(changelog): catch up 1.5.0 through 1.5.12 release notes
Converts [Unreleased] to [1.5.0] and adds individual sections for
1.5.1 – 1.5.12. Covers: MCP OAuth 2.1 + PKCE (Phases 1–8), OIDC
hardening, metacog NudgeQueue + wake trigger, SSE refresh-resume,
reasoning persistence (Phases 1–4), structured watch-result cards,
skills unlock, inline child approvals, Stage 3 Children primitive
lift, coordinator composer parity, node capability auto-detection,
progressive mermaid rendering, and the full schema migration list
for each release.

Updates the track list to stable/1.4, stable/1.5, and main.
2026-05-10 22:34:14 -07:00
Patrick Buckley d76c57e687 chore: bump version to 1.5.12 2026-05-10 18:28:46 -07:00
Patrick Buckley 58bf811a0e fix(server): always emit history event on /rewind to unblock edit-and-resend
Editing the first message in a workstream sends /rewind N where N is the
total user turns, leaving session.messages empty. The handler guarded the
history event with `if history:`, so only clear_ui was emitted. The
frontend dispatches the queued edit-and-resend from the history event
handler (app.js _pendingEditSend), so an empty history orphaned the
pending text and left the composer stuck in busy.

replayHistory already handles the empty case via showEmptyState(), so
emitting the event unconditionally is safe and unblocks the dispatch.
2026-05-10 18:27:23 -07:00
Patrick Buckley df942e375e feat(session): enriched backend error messages with provider + URL
A bare ``httpx.ReadTimeout`` previously surfaced as ``ReadTimeout: timed
out`` — no provider, no base URL, no model — leaving the user with no
signal to tell whether a model server hung, the URL was wrong, or the
model isn't loaded on the backend.

``ChatSession._format_backend_error`` now rewrites known boundary
exceptions (httpx ``ReadTimeout`` / ``ConnectError`` / etc. and OpenAI /
Anthropic SDK ``APITimeoutError`` / ``APIConnectionError`` /
``NotFoundError`` / ``AuthenticationError`` / ``RateLimitError``) into
operator-actionable text that names the provider, base URL (query
string stripped before ``sanitize_error_text`` redacts credentials),
and model.  Matching is by class name so the helper carries no SDK
imports.  Unrecognised exceptions fall through to the legacy
``f"{type(exc).__name__}: {exc}"`` shape, preserving existing grep
targets.
2026-05-10 18:27:14 -07:00
Patrick Buckley 84fd5dc859 chore: bump version to 1.5.11 2026-05-09 17:29:46 -07:00
Patrick Buckley ba8b1d9126 fix(session): AND-gate replay_reasoning_to_model with model capability
The Anthropic call sites in session.py passed the operator-side
`replay_reasoning_to_model` flag through without checking the
model's static `supports_reasoning_replay` capability. The OpenAI
Responses path AND-gated both flags in `_build_kwargs` so a model
without a reasoning lane (gpt-4o, etc.) silently skipped replay even
when the operator flag was set. The Anthropic path had no such gate.

For all current Claude entries this was a no-op asymmetry - every
`_ANTHROPIC_CAPABILITIES` row sets `supports_reasoning_replay=True`,
so `True AND op == op`. But:

- The capability flag was dead code on the Anthropic path
- A future Claude entry (or any Anthropic-shaped surface) shipping
  with the cap left at its False default would have replay fire
  anyway, against the cap declaration
- The asymmetry made `supports_reasoning_replay` an unreliable
  signal - readers couldn't tell if it gated anything per-provider

Move the AND-gate into `_resolve_replay_reasoning_to_model` via a
new optional `caps=` kwarg. When caps is provided, the resolver
returns `operator_on AND caps.supports_reasoning_replay`; when
omitted (back-compat for any caller not yet updated), it returns
the operator flag unchanged.

Thread caps through the three call sites: `_utility_completion`
(non-streaming), `_try_stream` (streaming, hoisted resolution out
of the retry loop since caps are attempt-invariant), and the
agent `_api_call` closure in `_run_agent`.

With the AND-gate now living at the session resolver, the redundant
in-provider gate in `OpenAIResponsesProvider._build_kwargs` is
removed. The provider now trusts the resolved bool it receives,
matching the AnthropicProvider shape and giving the cap a single
source of truth across providers. The two provider-level tests
that pinned the in-provider gate
(`test_include_omitted_when_capability_false`,
`test_include_omitted_by_default`) drop out; the session-level
boundary test
`TestSessionToOpenAIResponsesBoundaryIntegration::test_capability_false_omits_include_even_when_flag_true`
already covers the same end-to-end invariant.

Tests added:
- 4 resolver-level tests pinning the AND-gate semantics +
  back-compat when caps is omitted
- 1 wire-boundary integration test mirroring the OpenAI Responses
  `test_capability_false_omits_include_even_when_flag_true` -
  drives session._try_stream through the real AnthropicProvider
  with operator flag True + capability False and asserts the
  thinking block does NOT reach the SDK boundary

Existing `TestUtilityCompletionPassesFlag` test had its caps mock
upgraded from `SimpleNamespace` to a real `ModelCapabilities`
instance to satisfy the new attribute read and stay robust to
future capability fields.
2026-05-09 17:23:56 -07:00
Patrick Buckley 685b1e3d9b fix(console): preserve cs=None fallback in /v1/api/models placeholder
Copilot review feedback on #500.  The original
``list_available_models`` had an implicit cs=None branch where the
placeholder still advertised ``registry.default`` (filtered against
enabled rows) when ``app.state.config_store`` was None but
``coord_registry`` was bound — useful in the rare degraded state
where lifespan wired the registry but the ConfigStore failed to
initialise.  The PR #500 refactor accidentally dropped that branch:
the helper requires a config_store, so the cs=None case fell out as
"blank coordinator default".

Add an explicit ``elif coord_registry is not None`` branch that
mirrors the helper's tier 3 with the placeholder's enabled-rows
filter applied.  New test exercises this path by passing
``config_store=False`` to the test fixture.
2026-05-09 17:23:56 -07:00
Patrick Buckley 91300060fa fix(console): unify coordinator alias resolution across placeholder + factory
Previously /v1/api/models (home composer placeholder) and
console/session_factory.py walked separate two-/three-tier chains for
the coordinator alias.  session_factory was missing the
``model.default_alias`` tier, so admins who set the system default in
the Models tab would see it advertised but new coordinator sessions
would silently keep launching on ``registry.default``.

This commit:

- Extracts the chain into ``turnstone/console/coordinator_alias.py``.
  ``resolve_coordinator_alias`` returns the effective alias under a
  shared three-tier policy: explicit pin → ``model.default_alias`` →
  ``registry.default``.  Tier 2 is validated against
  ``registry.has_alias`` and falls through to tier 3 with a logged
  warning if unknown.  Tier 1 is intentionally passed through
  unvalidated so an explicit operator pin surfaces as 503 at
  ``registry.resolve`` rather than being silently swapped out.
- Wires both call sites through the helper.  The placeholder supplies
  an ``alias_filter`` that restricts every tier to enabled DB rows so
  the home composer never advertises a model the workstream picker
  can't actually offer; the session factory uses no filter (matches
  prior 503-on-typo behaviour for explicit pins).
- Adds direct integration tests for the session factory's chain
  (``tests/test_console_session_factory.py``) and updates the
  placeholder tests' fixture to provide a stub coord_registry, since
  the helper now requires one.
2026-05-09 17:23:56 -07:00
Patrick Buckley 688f047ce1 docs(console-ui): clarify coordinator placeholder fallback comment
Light-review followup on 389400c8.

The "mirrors session_factory.py:109-110" claim was inaccurate —
session_factory's chain is two tiers (coordinator.model_alias →
registry.default) and skips model.default_alias entirely.  The
placeholder handler extends that chain with model.default_alias as
tier 2 so admins who set the default in the Models tab see it
advertised in the home composer.  Comment now lists the three tiers
explicitly and flags the session_factory-vs-placeholder drift case
(where model.default_alias ≠ registry.default) as a separate issue
to track.

Also lifts the ``from types import SimpleNamespace`` import in the
test fixture to module level — minor readability cleanup.
2026-05-09 17:23:56 -07:00
Patrick Buckley 87893aa4ab fix(console-ui): align coordinator placeholder fallback with session_factory
Two Copilot-review followups on /v1/api/models default resolution.

- console/server.py: coordinator_default_alias now mirrors the full
  fallback chain in console/session_factory.py:109-110 — explicit
  coordinator.model_alias → model.default_alias → registry.default.
  The registry tier was missing, so the home composer placeholder went
  blank whenever an operator never set model.default_alias in the admin
  UI even though new coordinator sessions still launch on
  registry.default (loaded from config.toml [model].default by
  load_model_registry).  Two new tests cover the registry-default
  branch and the disabled-alias guard.
- console/static/app.js: _resolveModelLabel returns "" (not the bare
  alias) when the alias isn't found in the dropdown's model list, so
  callers can rely on the documented "fall back to neutral placeholder"
  contract.  Matches the existing doc comment.
2026-05-09 17:23:56 -07:00
Patrick Buckley 0988142303 feat(console-ui): home composer placeholders, toggle component, admin polish
Bundles the click-around polish on the console admin UX.

Home composer + schedule modals
- /v1/api/models now exposes coordinator_default_alias + judge_default_alias,
  resolved through the same chain console/session_factory.py uses.  Both the
  home composer's MODEL / JUDGE MODEL placeholders and the schedule create /
  edit modal model placeholders rewrite to "Default — alias (model)" once
  the API responds.  The `models_changed` SSE refresh keeps placeholders
  current as operators edit per-role assignments.
- Composer.setOptionPlaceholder added so callers can update just the first
  option's text without disturbing the rest of the choice list.

Admin → Models → Roles
- Channel adapter row added (channels.default_model_alias) — the migration
  to the Roles sub-tab missed it.  Key added to
  _MODEL_AFFECTING_SETTING_KEYS so edits fire the SSE refresh, and to the
  settings-tab roleKeys skip-list so it only renders in one place.
- Plan/Task agent rows now display "(inherit)" instead of the misleading
  "(default — <alias>)" — those roles cascade through plan_model →
  agent_model → session model, not a single concrete default.
- coordinator.reasoning_effort accepts "" (inherit), matching
  model.plan_effort / model.task_effort.
- Blank options in each role's MODEL select now match the "alias (model)"
  shape used by the other rows.

Toggle-switch component
- New .toggle-switch component (visually-hidden native checkbox + styled
  track + label).  40×22 hit target meets WCAG 2.5.5 (AAA), inset ring on
  the off state for ≥1.5:1 contrast against the modal surface.
- .toggle-stack groups toggles in a column with .toggle-group-divider for
  conceptual grouping (used in the Add Model modal between "Active" and the
  paired Reasoning toggles).
- .toggle--flush modifier zeroes the default top margin for toggles that
  sit flush against a heading or a dynamically-rendered row.

Sweep — every admin-modal boolean checkbox is now a toggle:
schedule (cs/es-autoapprove, es-enabled), policy (ep/epp-enabled),
tool-mode (ctm/etm-default), skill (csk/esk-auto-approve, csk/esk-enabled),
MCP (mcp-auto-approve, mcp-enabled), Add Model (Active, surface-persisted-
reasoning, replay-reasoning), judge bool settings (cancel_on_approval et
al.), and the user-roles-modal role assignment list.  The two
ogp-cred / eogp-cred inline credential checkboxes stay as compact inline
boxes since they sit beside text inputs in tight horizontal rows.

Add Model modal — the "Enabled" toggle promoted to "Active" and moved to
the very top of the form.  Tooltip explains it gates dropdown visibility
without removing the definition.

MCP authorization — the three radio buttons replaced with a vertical
.segmented-control option list.  Selected row paints --accent-dim plus a
filled .segmented-indicator; focus ring uses --accent so it stays visible
on the currently-selected option.

Role permissions modal — the 19 permission checkboxes are now
.toggle-switch.perm-toggle (monospace lowercase identifiers preserved).
The permissions are split into Scopes / Admin / Workstreams & Tools
sections under caps-styled section headers so the row-flow grid no longer
slices `admin.*` mid-column.

Judge bool toggles use a static "Enabled" caption rather than flipping
text on `.checked`; flipping lagged 50–300 ms behind the slider position
because the caption was sourced from the post-save reload.

CSS cleanup — dead `.admin-checkbox` / `.perm-checkbox` rules removed.
Specificity audit (scripts/css_specificity_audit.py) returns no conflicts
on any new component class.

Tests — 525 pass on the affected slices; new tests/test_console_available_
models.py pins each branch of the resolution chain in /v1/api/models so the
home composer placeholder stays correct as precedence rules evolve.
2026-05-09 17:23:56 -07:00
Patrick Buckley 40ecebf012 refactor(judge): require alias for judge.model, drop session-provider raw-model fallback
`IntentJudge.__init__` previously had a 3-way resolution chain: registered
alias → raw model id pinned onto the session provider → session model.  The
middle branch was a footgun documented in `console/session_factory.py:130-137`
— pinning the literal `judge.model` string onto the coordinator's session
provider silently broke every verdict whenever that provider didn't recognise
the model id (e.g. coordinator on Anthropic, `judge.model = "gpt-5-mini"` →
uniform `llm_fallback`).

Tightens to alias-only, matching `coordinator.model_alias` /
`model.plan_alias` / `model.task_alias`.  An unknown `config.model` now logs
a warning and inherits the session model — same path as empty.  Help text on
`judge.model` updated to clarify the contract.

Adds two regression tests in `TestModelAliasResolution` covering the
session-model inheritance for unknown values and the empty-model self-
consistency case.
2026-05-09 17:23:56 -07:00
Patrick Buckley c91869c7e5 fix(reasoning): synthesize reasoning_text alongside non-reasoning provider_blocks
GoogleProvider attaches raw tool_call dicts as ``provider_blocks`` on
the finish chunk for ``thought_signature`` round-trip
(``_google.py:_iter_stream``).  When the same turn streamed Gemini's
``reasoning_content`` as ``reasoning_delta`` chunks, the prior
synthesizer bailed out the moment ``provider_blocks`` was non-empty
— so the captured reasoning was visible live but lost on page reload.

Replace the early-return-if-non-empty check with a reasoning-bearing
type test (``thinking`` / ``redacted_thinking`` / ``reasoning`` /
``reasoning_text``).  When none of those types appear, append the
synthetic ``reasoning_text`` block to the existing list rather than
replacing it — preserving Google's tool-call fidelity blocks.

Also addresses two doc-accuracy review findings:
- ``LLMProvider.extract_reasoning_text`` docstring no longer claims
  OpenAI Chat / Responses are unwired (Phase 3+4 shipped extractors).
- Add the method to the Protocol methods table in
  ``docs/architecture.md`` (was missing alongside the class diagram).
2026-05-09 17:23:56 -07:00
Patrick Buckley 53b52092f9 fix(reasoning): per-block ANTHROPIC_VALID_BLOCK_TYPES filter + review fixes
The earlier all-or-nothing shape check on ``_provider_content`` discarded
every valid Anthropic block in a message the moment a single foreign
block (OpenAI ``reasoning``, Gemini thought parts, the synthetic
``reasoning_text`` from path-3 capture) appeared.  In the cross-model
resumption edge case that meant ``server_tool_use`` /
``web_search_tool_result`` blocks lost their ``encrypted_content``
silently, breaking web-search round-trip continuity on subsequent turns.

Replaced with a per-block walk: foreign blocks are dropped individually,
valid blocks ride the verbatim path, and an identity-preserving fast
path reuses the source list reference when nothing was filtered or
stripped (pinned by the ``is`` assertions in test_providers.py).

Also addresses validation-pass review findings:
- Document the single-tier vs three-tier ``surface_persisted_reasoning``
  resolution divergence between server.py:_build_history and
  session_routes.make_history_handler.
- Document why OpenAIResponsesProvider._convert_messages defaults
  ``replay_reasoning_to_model=False`` while Anthropic's defaults True.
- Document the ``source`` metadata field on synthetic ``reasoning_text``
  blocks as reserved-for-future-use, not dead code.
- Add edge tests for non-dict / missing-type-key blocks in
  _provider_content (defensive branches in the per-block walk).
2026-05-09 17:23:56 -07:00
Patrick Buckley 0d1a009a4c test(reasoning): skip wire-boundary tests when anthropic extra missing
CI test job installs `[test]` extras, which omits `anthropic`. The two
TestSessionToWireBoundaryIntegration cases drive the real
AnthropicProvider.create_streaming, which calls _ensure_anthropic() and
raises ImportError. Match the repo convention (test_channel_discord,
test_channel_slack, test_tls_*) by gating the helper with
pytest.importorskip("anthropic").
2026-05-09 17:23:56 -07:00
Patrick Buckley e8352bd8e5 fix(reasoning): apply Copilot review feedback + docs sync
PR #498 round-robin review surfaced 5 findings.  4 applied; 1 rejected
with rationale.

Applied

* **Copilot finding 5** (history_decoration.py:341): dispatcher
  inspected only ``provider_content[0]['type']``.  OpenAI Responses
  captures EVERY ``output_item.done`` event into ``provider_blocks``
  (not just reasoning) — in practice the order is
  ``[reasoning, message, ...]`` but the API doesn't guarantee that;
  a hypothetical ``[message, reasoning]`` ordering would silently
  drop the reasoning under an index-only check.  Now walks the list
  for the first block whose type is in ``_BLOCK_TYPE_PROVIDER_FACTORY``,
  then dispatches the WHOLE list to that provider's extractor.  Each
  provider's extractor already filters internally by its own block
  type, so passing the full list is correct.  Regression test added
  (``test_dispatcher_scans_past_unrecognized_first_blocks``).

* **Copilot finding 3** (migration 052 docstring): the previous
  review-fix wave used sed to rename ``persist_reasoning`` →
  ``surface_persisted_reasoning`` everywhere, which mangled a
  historical reference in the migration docstring ("The earlier name
  ``surface_persisted_reasoning`` was renamed...").  Restored to
  point at the actual pre-rename name (``persist_reasoning``).

* **Copilot finding 4** (sdk/typescript/src/events.ts:26):
  ``HistoryEvent`` JSDoc still referenced ``persist_reasoning`` —
  the sed rename only walked ``turnstone/`` and ``tests/``, missing
  the TypeScript SDK.  Updated to ``surface_persisted_reasoning``.
  Also widened the comment to cover all three reasoning-bearing
  block types (Anthropic ``thinking``, OpenAI Responses ``reasoning``,
  synthetic ``reasoning_text``) instead of mentioning only Anthropic.

* **github-code-quality finding** (session.py:1120): ``_resolve_server_type``
  had a bare ``except Exception: pass``.  Replaced with a
  ``log.debug(..., exc_info=True)`` + explanatory comment.  Behaviour
  unchanged (still returns ``""`` on any lookup failure); failures
  are now observable under DEBUG triage.

Rejected (with rationale)

* **github-code-quality finding** (_protocol.py:265):
  ``extract_reasoning_text``'s body is ``...`` per ``LLMProvider``
  Protocol convention.  Every method in the file uses ``...`` (PEP
  544 idiomatic Protocol style).  Changing only this one to
  ``raise NotImplementedError`` would be inconsistent with the rest
  of the file.  CodeQL's "statement has no effect" warning is
  technically correct for ``...`` as a standalone expression but
  ignores the documented Python Protocol convention.  No fix.

Docs sync

* docs/api-reference.md: ``history`` SSE event message-shape table
  gains the optional ``reasoning`` field.
* docs/architecture.md: ``ModelCapabilities`` row in the type table
  gains ``supports_reasoning_replay``; ``StreamChunk`` and
  ``CompletionResult`` rows gain the existing ``provider_blocks``
  field (was missing pre-PR).  New "Per-model reasoning persistence"
  subsection under the Models config section, documenting the two
  flags + capability gate + three reasoning paths + cross-provider
  shape filter.
* docs/settings.md: new "Reasoning persistence (per-model)"
  subsection with the two-flag table and capability-gate note.
* docs/diagrams/03-core-engine-classes.puml: ``LLMProvider`` interface
  adds ``extract_reasoning_text`` + the new ``replay_reasoning_to_model``
  kwarg; ``ModelCapabilities`` class adds ``supports_reasoning_replay``.
  PNG regenerated.

Lint + test gate

* ruff check + ruff format clean.
* mypy clean (191 source files).
* pytest -m 'not live' — 6116 passed (3 deselected), +1 net new test
  (``test_dispatcher_scans_past_unrecognized_first_blocks``).
2026-05-09 17:23:56 -07:00
Patrick Buckley de4cc568c4 fix(reasoning): apply full-stack review findings
Multi-stage /review on the full Phase 1+2+3+4 stack surfaced 9 findings
(0 critical, 3 major, 5 minor, 1 nit, 1 uncertain).  All applied.

Major

* perf-1 (session_routes.py:2402): make_history_handler ran sync
  storage.load_workstream_config inside async def history on the cold-
  workstream path, blocking the event loop on every dashboard /history
  request for non-resident workstreams.  Every other storage call in
  the same handler correctly used asyncio.to_thread.  Wrap the sync
  call in asyncio.to_thread (preserving the existing try/except so a
  DB failure still degrades to the conservative-default branch instead
  of bubbling out).

* q-2 (test_reasoning_audit_log_discipline.py): the security-sensitive
  test (reasoning text never lands at INFO+ severity) only covered the
  4 Phase 1 surfaces.  Phase 2 added the strip predicate in
  AnthropicProvider._convert_messages and Phase 3 added 3 more code
  paths that touch reasoning text — none guarded.  Added 4 parallel
  tests using the existing capture-and-walk infrastructure:
  OpenAIResponsesProvider.extract_reasoning_text,
  OpenAIChatCompletionsProvider.extract_reasoning_text,
  ChatSession._stream_response (drives the synth-block stamp via a
  fake reasoning-emitting stream), AnthropicProvider._convert_messages
  with replay_reasoning_to_model=False (drives the Phase 2 strip
  predicate).

* q-1 (model_registry.py:42): the persist_reasoning flag name implied
  storage-control but actually gates UI rehydration only — operators
  flipping it could reasonably expect "stop persisting reasoning" but
  storage of reasoning bytes happens in provider_data regardless.
  Renamed everywhere to surface_persisted_reasoning: ModelConfig
  field, migration 052 column (renaming in-place since 052 is not yet
  on main), schema, MODEL_DEFINITION_MUTABLE allowlist, _postgresql.py
  + _sqlite.py CRUD impls, _protocol.py create_model_definition
  signature, 3 console_schemas Pydantic models, console/server.py
  admin POST + PUT, model_registry row mapper, history_decoration.py
  helper parameter, server.py _build_history local var,
  session_routes.py make_history_handler local var, sdk/events.py
  HistoryEvent docstring, admin.js form id + override pill label,
  index.html form input id + UI label + tooltip, coordinator.js (none
  needed), and every test that referenced the old field name.  The
  admin tooltip now reads "Storage of reasoning bytes is unaffected
  by this flag — they ride in provider_data regardless" so the
  decoupling stays explicit at the operator surface.

Minor

* bug-1 (history_decoration.py:336): dispatcher discriminated on
  provider_content[0]["type"] only.  Anthropic's redacted_thinking
  blocks (sealed by the safety system) can appear before, after, or
  interleaved with regular thinking blocks per the API docs.  When a
  redacted block lands first, the dispatcher returned "" and the UI
  silently lost the surrounding thinking text.  Registered
  "redacted_thinking" as a second key in _BLOCK_TYPE_PROVIDER_FACTORY
  pointing at the same AnthropicProvider factory — the existing
  extractor's type=="thinking" filter already correctly skips redacted
  blocks while walking the full list.  Regression test added.

* q-3 (_protocol.py:155): replay_reasoning_to_model defaults split
  across 9 sites — operator-side defaults to False (matches DB
  server_default), provider-API defaults to True (back-compat with
  direct callers).  Original "pick False everywhere" fix would have
  silently flipped behaviour for any direct provider caller.  Instead
  documented the intentional bifurcation in the Protocol's
  create_streaming docstring.

* q-4+q-5 (_protocol.py:107 + 3 providers): MAX_REASONING_DISPLAY_BYTES
  was enforced via Python str slicing which counts code points, not
  UTF-8 bytes — 4-byte CJK/emoji glyphs would blow past the byte
  ceiling.  Renamed to MAX_REASONING_DISPLAY_CHARS to match actual
  behaviour.  Hoisted the 4-line truncation pattern into a shared
  _join_reasoning_with_cap helper in _protocol.py; each provider's
  extractor becomes a single line at the tail.

* q-6 (tests/_session_helpers.py): _NullUI + _make_session were
  duplicated verbatim between test_session_replay_reasoning.py and
  test_session_synth_reasoning_block.py.  Hoisted to a shared
  tests/_session_helpers.py module (importable, leading underscore so
  pytest doesn't try to collect it).  test_model_registry.py's
  _make_session has a different signature (registry/model_alias args
  + _FakeUI) and is not a candidate for sharing.

Nit

* q-7 (history_decoration.py:286): _make_provider_factory used a
  dict-as-cell workaround for closure read-only scope.  Replaced with
  the more idiomatic nonlocal pattern.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6115 passed (3 deselected).  Net +5 tests
  (4 audit-log discipline + 1 redacted_thinking dispatcher).

Refinements vs the dedupe output (caught during sanity rendering
the report)

* perf-1 fix preserved the try/except wrapper.  The original "wrap in
  to_thread" one-liner would have let an OperationalError bubble out
  instead of degrading to the fallback branch.

* q-3 fix explicitly documented the bifurcation rather than
  collapsing both sides to False.  "Pick False everywhere" would
  silently flip back-compat behaviour for direct provider callers.

* q-1 fix included the admin.js:5292 fallback site
  (m.persist_reasoning !== false) that the original threaded-change
  list missed.

* q-6 fix verified the third _make_session in test_model_registry.py
  is structurally different (different signature + different UI
  helper) and intentionally NOT a dedupe target.
2026-05-09 17:23:55 -07:00
Patrick Buckley b477c85ddc feat(reasoning): OpenAI Responses + Chat Completions capture/replay (Phase 3+4)
Wire reasoning capture and (where the API supports it) replay for the
two remaining provider paths.  Phase 3 was originally scoped as
"OpenAI Responses + Gemini" but a spike against the OpenAI SDK source
revealed that Gemini routes through the OpenAI-compatible endpoint
(``/v1beta/openai/``), which is structurally identical to vLLM /
llama.cpp / any other Chat-Completions-shaped local model.  Phase 3
and Phase 4 collapse into one feature with two distinct sub-paths:

* **Path 2 (OpenAI Responses)** — full capture+replay.  ``include=
  ["reasoning.encrypted_content"]`` on the request makes the API
  surface ``encrypted_content`` on reasoning items in
  ``provider_blocks``; ``_convert_messages`` round-trips them as
  ``ResponseReasoningItemParam`` input items on subsequent turns.
  Verified against the OpenAI Python SDK 2.33.0 source
  (``response_reasoning_item.py:31-62``,
  ``response_reasoning_item_param.py:33-37``,
  ``response_create_params.py:70-74``).  Even with ``store=False``,
  ``encrypted_content`` round-trips correctly per the SDK's own
  documentation.

* **Path 3 (Chat Completions / vLLM / llama.cpp / Gemini-compat)** —
  persist-only.  Canonical OpenAI Chat Completions has no reasoning
  field on the wire, but several local-model servers tack on
  ``delta.reasoning_content`` as Pydantic extras.  ``ChatSession.
  _maybe_synth_reasoning_block`` stamps a synthetic ``{type:
  "reasoning_text", text, source?}`` block onto ``_provider_content``
  at end-of-stream when no native ``provider_blocks`` were emitted but
  ``reasoning_parts`` accumulated text.  The ``source`` field carries
  ``server_compat.server_type`` (vllm, llama.cpp, sglang, …) for
  diagnostic value — informational only, doesn't gate behaviour.
  Reasoning text NEVER replays back to the model on this path; it
  rides ``_provider_content`` only for ``/history`` UI rehydration
  and gets stripped from the wire by the existing
  ``sanitize_messages`` underscore-prefix strip on every request.

What this change does

* ``ModelCapabilities.supports_reasoning_replay: bool = False`` added
  to the dataclass.  Set True on every OpenAI reasoning model
  (gpt-5* + o-series via the Responses API) and every Anthropic
  Claude entry (default + 6 model-specific).  Path-2 wire-build does
  ``replay_active = bool(replay_reasoning_to_model and caps.supports_
  reasoning_replay)`` so an operator who flips the flag on a
  non-reasoning model (gpt-4o via Responses) silently no-ops rather
  than emit a malformed ``include=`` request.

* ``OpenAIResponsesProvider`` gains:
  - ``_build_kwargs`` accepts ``replay_reasoning_to_model: bool``
    (threaded from ``create_streaming``/``create_completion``);
    adds ``include=["reasoning.encrypted_content"]`` when active.
  - ``_convert_messages`` accepts the same flag, captures
    ``_provider_content`` reasoning items pre-sanitization, and
    emits them as input items immediately before the assistant
    message they belong to.  Position is tracked by ASSISTANT
    ORDINAL (not raw index) — ``sanitize_messages`` drops orphan
    tool results and inserts synthesized error tool messages, but
    NEVER drops or duplicates assistant messages, so the n-th
    assistant in the original list is invariably the n-th in the
    sanitized list.  Index-based lookup would have silently
    misrouted reasoning attachments after any tool-message repair.
  - ``extract_reasoning_text`` walks ``type=="reasoning"`` items and
    returns ``summary[*].text`` + ``content[*].text`` concatenation.
  - ``_reasoning_item_for_input`` projects a stored item into
    ``ResponseReasoningItemParam`` shape (drops server-only
    ``status``).  Returns ``None`` when ``id`` is missing or non-
    string per the SDK ``Required[str]`` schema; caller skips
    appending, preventing malformed input items from reaching the API.

* ``OpenAIChatCompletionsProvider`` gains:
  - ``extract_reasoning_text`` walks synthetic
    ``type=="reasoning_text"`` blocks and returns the concatenated
    text directly (no underlying provider semantics — the synth
    block IS the surface).

* ``ChatSession`` gains:
  - ``_resolve_server_type(alias)`` reads ``server_compat.server_type``
    from the active model's capabilities dict.
  - ``_maybe_synth_reasoning_block(provider_blocks, reasoning_parts)``
    creates the synthetic ``reasoning_text`` block when no native
    blocks were emitted but reasoning was captured.  Wired at the
    end of ``_stream_response`` immediately before the
    ``_provider_content`` stamp.

* ``history_decoration.py`` dispatcher collapses three near-identical
  lazy-init singleton getters (one per recognised block type) into a
  single ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict + helper.  Adding a
  fourth provider becomes a one-line dict entry.

* Constants hoist: ``MAX_REASONING_DISPLAY_BYTES = 64 * 1024`` moved
  from three sibling provider modules into ``_protocol.py`` so a
  tuning change propagates uniformly to every provider's display path.

Cross-provider safety

The synthetic ``reasoning_text`` block type is intentionally NOT in
``ANTHROPIC_VALID_BLOCK_TYPES`` (Phase 2 constant).  Cross-model
resumption (operator switches from a local model to Anthropic mid-
workstream) falls through Phase 2's shape filter cleanly to the
text+tool_calls rebuild path rather than reaching Anthropic with a
malformed block.  Pinned by ``test_synthetic_block_falls_through_
anthropic_shape_filter``.

Same protection applies in reverse: OpenAI Responses
``type=="reasoning"`` items reaching Anthropic mid-workstream fail
the shape filter and rebuild from text+tool_calls.

Tests (49 net new tests)

* ``tests/test_provider_openai_responses_reasoning.py`` (21 tests):
  - Extractor unit tests: empty/none/no-reasoning/single/mixed/
    truncation/malformed/non-list (8).
  - ``_reasoning_item_for_input`` projection (4 tests including the
    new None-on-missing-id guard).
  - ``_build_kwargs`` include= gating: flag+capability/flag-false/
    capability-false/default-omits (4).
  - ``_convert_messages`` reasoning round-trip: emit-before-assistant/
    drop-on-replay-false/foreign-shape-skipped/default-replay-false (5).

* ``tests/test_session_synth_reasoning_block.py`` (23 tests):
  - ``_maybe_synth_reasoning_block`` direct unit tests (6).
  - Cross-provider safety regression — synthetic block falls through
    Anthropic shape filter (2).
  - ``OpenAIChatCompletionsProvider.extract_reasoning_text`` for the
    new synthetic block type (6).
  - ``_resolve_server_type`` direct unit tests (5).
  - ``_stream_response`` integration tests driving fake reasoning-
    emitting streams through the actual session method (3 tests
    — added in response to a code-review finding that pinned the
    wire-up at session.py needs an integration test).

* ``tests/test_session_replay_reasoning.py`` extended with 4
  ``TestSessionToOpenAIResponsesBoundaryIntegration`` tests driving
  ``session._try_stream`` -> real ``OpenAIResponsesProvider`` ->
  captured ``client.responses.create`` SDK boundary call.  Negative-
  tested: temporarily reverting the ``include=`` step in
  ``_build_kwargs`` makes ``test_replay_true_adds_include_to_
  responses_request`` fail; restoring makes it pass.

* ``tests/test_history_decoration.py`` extended with the new
  ``reasoning_text`` dispatcher branch test, and the Phase 1 stub
  test for the OpenAI Responses dispatcher branch was tightened
  (it now asserts real text extraction instead of the empty-string
  stub).

* ``tests/test_provider_anthropic_reasoning.py`` had its Phase 1
  ``OpenAIResponses returns "" for reasoning blocks`` stub test
  retitled and updated to assert the real Phase 3 behaviour.

Code-review pass

Multi-stage ``/review`` pipeline (4 finders + verify + dedupe) ran
on this diff.  6 findings (1 major, 3 minor, 2 nit), 0 critical, 0
security, 0 performance.  All applied:

* MAJOR (bug-1+bug-4+q-1): ``_convert_messages`` enumerate-index
  lookup was unsound under ``sanitize_messages`` length changes.
  Fixed by switching to assistant-ordinal-keyed lookup.
* MINOR (q-2+q-3): ``_MAX_REASONING_DISPLAY_BYTES`` duplicated
  across three provider modules + declared after first use.
  Fixed by hoisting to ``_protocol.py``.
* MINOR (q-4): three near-identical singleton getters in dispatcher.
  Fixed by collapsing to ``_BLOCK_TYPE_PROVIDER_FACTORY`` dict.
* MINOR (q-5): ``_maybe_synth_reasoning_block`` wire-up not pinned
  by integration test.  Fixed by adding three
  ``TestStreamResponseSynthBlockIntegration`` tests.
* NIT (bug-2): ``_reasoning_item_for_input`` fell back to ``id=""``;
  fixed to return ``None`` on missing/non-string id.
* NIT (q-6): four naming variants for the same concept; renamed
  ``_convert_messages`` kwarg to match the operator-flag name.
* REFUTED (bug-3): SDK distinguishes summary vs content as separate
  fields; no double-counting concern.

Briefing departures

The briefing's Phase 3 plan grouped Gemini with OpenAI Responses on
the assumption that Gemini reasoning had its own native shape (like
Anthropic's ``thinking``).  The spike confirmed Gemini-via-OpenAI-
compat is path-3 (Chat Completions shape, no native reasoning
items).  Phase 3+4 merger handles Gemini for free via the synthetic
``reasoning_text`` block — same mechanism used for vLLM and
llama.cpp.  Whether Gemini's specific endpoint actually emits
``reasoning_content`` deltas is server-dependent and not yet
empirically verified; capture is best-effort (server-emission-driven,
no flag gate).

The briefing's Phase 4 plan stamped reasoning as Anthropic-shaped
``thinking`` blocks ``{type: "thinking", thinking: <text>}``.  This
PR uses a distinct ``{type: "reasoning_text", text, source?}`` shape
to avoid a cross-model resumption hazard the briefing missed: an
unsigned synthetic Anthropic-shape block reaching Anthropic's wire
would 400 the API.  The distinct shape falls through Phase 2's shape
filter cleanly without needing signature validation in the filter.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6110 passed (3 deselected).  Phase 3+4
  added 49 net new tests.
2026-05-09 17:23:55 -07:00
Patrick Buckley 00bd80a658 feat(reasoning): wire-build shape filter + replay flag (Phase 2)
Make ``replay_reasoning_to_model=False`` actually suppress prior-turn
thinking blocks on the Anthropic wire (Phase 1 stored the operator
flag but the wire path always re-sent ``_provider_content``
verbatim).  As a side benefit, close a pre-existing latent bug where
foreign-shaped ``_provider_content`` (e.g. an OpenAI Responses
``type="reasoning"`` block reaching Anthropic on a mid-workstream
model switch, post-Phase-3) would have 400'd the API.

Why now: Phase 1 shipped the operator knob and UI rehydration but
the wire payload still always carried thinking blocks for
Anthropic-with-thinking turns.  Operators flipping replay=False saw
no behaviour change on the actual API call -- the flag only affected
``/history`` rendering.  Phase 2 closes that gap.

What this change does

* ``ANTHROPIC_VALID_BLOCK_TYPES`` (frozenset of 8 block types
  Anthropic's input boundary accepts) and
  ``ANTHROPIC_REASONING_BLOCK_TYPES`` (the strip subset) added at
  the top of ``_anthropic.py``.  The strip set is intentionally
  narrow: ``{"thinking", "redacted_thinking"}`` -- ``tool_use`` /
  ``server_tool_use`` / ``web_search_tool_result`` (which carry
  web-search ``encrypted_content``) MUST survive for round-trip
  continuity, and a regression test pins this.
* ``_convert_messages`` signature gains
  ``replay_reasoning_to_model: bool = True`` (back-compat default
  -- production call sites pass the resolved value explicitly).
  The verbatim ``_provider_content`` replay path is now wrapped by
  a shape-validity check using ``ANTHROPIC_VALID_BLOCK_TYPES``;
  foreign-shaped payloads fall through to the existing text+
  tool_calls rebuild path rather than reaching the API.  When
  shape is valid AND replay=False, a list comprehension drops
  thinking blocks from ``wire_blocks`` while preserving
  tool_use / web_search blocks.  When all blocks are stripped
  (message had only thinking, no text or tool_calls), the message
  also falls through to the rebuild path -- which silently skips
  if both content and tool_calls are empty (correct: stripped
  reasoning has nothing to replay).
* Orphan-tool detection still walks the ORIGINAL ``provider_content``
  (not ``wire_blocks``) so the strip cannot accidentally lose the
  source-of-truth tool_use IDs.  The implementation comment pins
  this invariant.
* Protocol surface grows the kwarg on both ``create_streaming`` and
  ``create_completion``.  ``OpenAIChatCompletionsProvider``,
  ``OpenAIResponsesProvider``, and ``GoogleProvider`` (via
  inheritance) accept the kwarg and ignore it -- they have no
  first-class reasoning shape on the wire today.  Phase 3 will use
  it on the OpenAI Responses adapter to gate
  ``include=["reasoning.encrypted_content"]``.
* ``ChatSession._resolve_replay_reasoning_to_model(alias)`` reads
  ``ModelConfig.replay_reasoning_to_model`` from the registry,
  defaulting to ``False`` on lookup failure (the conservative
  miss-fallback: replaying reasoning text against an unknown
  operator preference is worse than missing the strip).  Threaded
  into the three production call sites:
  ``ChatSession._try_stream`` (streaming), ``_utility_completion``
  (title gen / compaction / extraction), and the agent provider
  call site (plan / task agents).

Token calibration deferred to Phase 4

The briefing's optional Phase 2 step (extending ``_msg_text_chars``
to count ``_provider_content`` bytes that survive the strip)
required either invasive flag-threading through every call site
of the static method or a lossy approximation that picked the wrong
direction for the default case.  Per the briefing's ``pick a
phase'' guidance, this is bumped to Phase 4.  The pre-existing
silent under-count on Anthropic-thinking turns persists when
replay=True.  Strip-when-False naturally fixes the under-count by
keeping the bytes off the wire entirely; the residual case is the
opt-in replay path.

Tests (28 new, all driving through real boundary objects)

* ``tests/test_provider_anthropic_replay.py`` (19 tests):
  - Strip vs preserve under both flag values (3 tests including
    redacted_thinking).
  - Default-kwarg back-compat preserves verbatim replay (1 test).
  - Web-search tool_use + server_tool_use + web_search_tool_result
    survive strip with encrypted_content intact (2 tests, edge 14).
  - Orphan-tool synthesis after strip -- pins the
    ``provider_content`` source-of-truth read at lines 397-433
    (1 test).
  - Foreign-shape fallthrough: OpenAI ``type="reasoning"`` block
    rebuilds via text+tool_calls (1 test).
  - Mixed-shape fallthrough: even one foreign block forces
    rebuild (1 test).
  - Empty / None / non-list ``_provider_content`` fallthrough
    (3 tests).
  - Legacy Anthropic-thinking row pre-Phase-2 stays in verbatim
    path -- no regression on existing conversations (2 tests).
  - All-blocks-stripped fallthrough behaviour: rebuild from text
    if available, silently skip if not (2 tests).
  - Constants pinning: strip set is narrow, valid set includes
    web search, strip is subset of valid (3 tests).
* ``tests/test_session_replay_reasoning.py`` (12 tests):
  - Resolver: 6 tests covering miss / default / set / explicit /
    fallback alias / exception.
  - Streaming call site: 3 tests pinning the kwarg propagates
    through ``_try_stream`` to a stub provider.
  - Non-streaming call site: 1 test pinning
    ``_utility_completion`` propagates the flag.
  - End-to-end boundary integration: 2 tests driving
    ``_try_stream`` -> real ``AnthropicProvider`` -> captured
    Anthropic SDK ``client.messages.stream`` boundary, asserting
    on the ACTUAL wire payload shape.  Negative-tested:
    temporarily reverting the kwarg-thread at
    ``_anthropic.py:create_streaming`` makes the wire test fail
    with ``Strip predicate did not fire at wire boundary``;
    restoring makes it pass.

The boundary integration tests were added in response to a code
review finding that the bare-stub call-site tests would not catch
a regression where the provider stops reading the kwarg or
``_convert_messages`` silently drops the strip.  The integration
tests close that gap by inspecting what reaches the (mocked) SDK,
not just what the provider was called with.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6061 passed (3 deselected).  Phase 2
  added 28 net new tests.
2026-05-09 17:23:55 -07:00
Patrick Buckley 47df9d23c5 feat(reasoning): persist reasoning text on history payload (Phase 1)
Surface stored Anthropic thinking blocks on /history responses so
refreshing the page rehydrates the reasoning bubble. Wire payloads
unchanged. Per-model operator knobs added to model_definitions for
both UI rehydration and (Phase 2) wire-build replay.

Why now: reasoning is already round-tripped via _provider_content for
Anthropic-with-thinking turns, but never surfaces on the history wire,
so a tab reload showed only the final answer with no rationale.
Operators also have no per-model lever to opt out of UI display or to
opt in to replay-to-model on subsequent calls.

What this change does

* Migration 052 adds two boolean columns to model_definitions:
  persist_reasoning (default 1) controls UI rehydration; replay_
  reasoning_to_model (default 0) reserved for Phase 2's wire-build
  shape filter. Mirrors the enabled column pattern (NOT NULL +
  integer server_default).
* LLMProvider Protocol gains extract_reasoning_text(provider_blocks)
  with concrete impls on AnthropicProvider (walks type=='thinking'
  blocks, joins with newline, caps at 64 KiB) and no-op stubs on
  OpenAIChatCompletionsProvider + OpenAIResponsesProvider. Google
  inherits the no-op via OpenAIChat. Phase 3 will wire the OpenAI
  Responses extractor once include=['reasoning.encrypted_content']
  is requested.
* turnstone.core.history_decoration gains a structural dispatcher
  extract_reasoning_text_from_provider_content keyed off the first
  block's type field (Anthropic 'thinking' / OpenAI Responses
  'reasoning' / Gemini 'thought' are non-overlapping by API design).
  Both history surfaces use it: _build_history calls the dispatcher
  directly (the SSE-replay path builds entry dicts from scratch),
  and the lifted make_history_handler runs the list-helper variant
  in the existing to_thread block.
* make_history_handler resolves persist_reasoning via three tiers:
  live session -> workstream_config.model_alias (the same key
  SessionManager uses to rehydrate the original model after process
  restart) -> conservative True default. Operator flag-flip takes
  effect uniformly on both warm and cold workstreams.
* Frontend: app.js replayHistory and coordinator.js role==='assistant'
  branch each call the existing reasoning-bubble construction (for
  app.js, the document.createElement pattern from the live SSE
  handler; for coord, the appendMsg('reasoning') helper) when
  msg.reasoning is non-empty. Reasoning bubbles render before the
  content bubble, matching live SSE order.
* Admin UI: two checkboxes ('Persist reasoning', 'Replay reasoning
  to model') in the model edit modal, plus override-pill display in
  the model row when set to non-default values.

What is intentionally out of scope

* Phase 2 -- ANTHROPIC_VALID_BLOCK_TYPES shape filter at
  _anthropic.py:312-316, _convert_messages replay_reasoning_to_model
  parameter, thinking-strip branch, _msg_text_chars token-calibration
  extension. The replay flag is stored but not consumed on the wire.
* Phase 3 -- OpenAI Responses include=['reasoning.encrypted_content'],
  Gemini include_thoughts spike, ModelCapabilities.supports_
  reasoning_replay.
* Phase 4 -- Local-model / chat-template reasoning persistence
  (session.py:3486 reasoning_parts accumulator).

Tests

* AnthropicProvider.extract_reasoning_text -- 13 unit tests covering
  None / empty / mixed / multi-block / cap / malformed / non-list
  inputs plus other-provider no-op verification (real provider
  instances, no mocks).
* extract_reasoning_for_history -- 10 dispatcher tests including
  block-type discriminator routing (thinking vs reasoning vs
  unknown), strip-when-flag-false, empty / non-dict guards, and
  cross-role isolation.
* _build_history -- 6 boundary tests through the real Anthropic
  extractor with stub sessions, including the registry-lookup
  failure default-True branch.
* make_history_handler -- 5 round-trip tests through real storage:
  the storage layer's reconstruct_messages decodes provider_data
  into _provider_content, and the helper extracts through the real
  AnthropicProvider. Includes the live-session flag honoring path,
  the cold-workstream workstream_config lookup path, and the
  no-alias default-True fallback path.
* Audit-log discipline -- 4 structural mock-and-assert tests that
  capture every Logger.info / warning / error call across the
  pipeline (extractor, dispatcher, list-helper, _build_history)
  and assert no captured payload contains a marker reasoning string.
* model_definitions storage -- 6 round-trip tests: default flags,
  explicit create with both flags, individual update of each flag,
  and list-includes-flags assertion.
* model_registry -- 4 tests: dataclass defaults, dataclass with
  explicit flags, DB-row-mapping with both flags, and pre-052
  legacy-row default-fallback.

Edge cases pinned by the test suite

* Pre-052 DB rows missing the new columns degrade to dataclass
  defaults (test_db_reasoning_flags_default_when_absent).
* Live session in memory has its flag honored (test_history_handler_
  with_persist_flag_false_via_live_session).
* Cold workstream resolves the flag via workstream_config +
  app.state.registry (test_history_handler_cold_workstream_resolves_
  via_workstream_config) -- this closes the gap where a process
  restart would have silently un-honored an operator flag-flip.
* Cold workstream without persisted model_alias falls through to
  default True (test_history_handler_cold_workstream_no_alias_
  defaults_true).
* Foreign / unknown / missing block types degrade silently to no
  reasoning field rather than misroute or crash.

Lint + test gate

* ruff check + ruff format -- clean.
* mypy -- no issues across all 191 source files.
* pytest -m 'not live' -- 6030 passed (3 deselected).
2026-05-09 17:23:55 -07:00
Patrick Buckley 16fc7efce2 style(sse): align comments with always-advance seq invariant
Doc-debt cleanup flagged by /review on 9dc29db7. The cap+seq fix
flipped the seq-advance rule but left two doc sites describing the
old "incremented only on actual append" shape — exactly the buggy
invariant the previous commit removed. Future readers trusting the
stale docs would be one wrong assumption away from re-introducing
the silent-drop bug.

Updates the field-init comment block and the docstring on
register_listener_with_in_progress_snapshot (which sits at the
snap_seq capture site, so its contract is consumer-facing).

Also drops the now-dead `seq: int = 0` initializer in
on_reasoning_token and on_content_token — under the new shape, the
unconditional `seq = self._ws_inflight_seq` inside the lock makes
the initializer unreachable. Was load-bearing under the old
else-branch; harmless now but signals "some path leaves seq at 0"
to a reader.
2026-05-09 17:23:55 -07:00
Patrick Buckley e8eca2ec9b fix(sse): always advance _ws_inflight_seq on emit, even past cap
Copilot caught a real bug in the cap+seq interaction: the previous
shape only advanced ``_ws_inflight_seq`` when the buffer actually
appended, on the theory that "every _seq corresponds to a buffered
fragment" was a useful invariant. It wasn't — once the buffer hit
its cap, seq stalled at the high-water-pre-cap, so a subscriber that
registered AFTER the cap was hit would capture
``snap_seq == stalled_seq``, and every subsequent live token (also
tagged with the stalled seq) would be filter-dropped by the events
handler's ``seq <= snap_seq`` dedup. Silent loss of the entire
post-cap stream for refresh-past-cap tabs.

Fix: advance seq on every emit, regardless of buffer cap. The cap
is a buffer-size limit, not a stop-streaming signal. Past-cap tokens
are absent from the snapshot's text payload (the buffer was
truncated at cap) but the live stream past them is now correctly
delivered — refresh-after-cap renders snapshot-up-to-cap then live
tokens past it, with a visual gap equal to the past-cap chunk and
no silent drop of subsequent tokens.

Test ``test_inflight_seq_increments_only_on_actual_append`` enforced
the buggy invariant and is renamed/flipped to
``test_inflight_seq_advances_on_every_emit_even_at_cap``. Added
``test_subscriber_after_cap_hit_receives_subsequent_tokens`` (and
the reasoning equivalent) as direct regressions for the
silent-token-loss scenario.
2026-05-09 17:23:55 -07:00
Patrick Buckley 57563b0c12 docs(sse): document state_change + in_progress_snapshot events
Updates the docs that describe the per-workstream SSE event stream and
the SessionUI lifecycle to match the refresh-resume changes:

- api-reference.md: documented the `state_change` event (previously
  undocumented despite already being a live event) and the new
  `in_progress_snapshot` event; rewrote the multi-consumer fan-out
  paragraph to mention the kind-specific replay tail (state_change +
  optional in_progress_snapshot) so the "no catch-up needed" claim
  is no longer misleading.
- architecture.md: bumped the SessionUI Protocol stub to 16 methods
  (added `on_turn_start` / `on_turn_committed`) and pointed at the
  in_progress_snapshot section in the API reference.
- sdk.md: added rows for `state_change`, `in_progress_snapshot`, and
  `approval_resolved` (preexisting gap) to the per-workstream event
  table.
- coordinator-api-tour.md: added an `in_progress_snapshot` row to the
  event table and rewrote the reconnection-contract paragraph to
  cover mid-stream content/reasoning restoration.
- diagrams/04-conversation-turn.puml: added `on_turn_start()` before
  the thinking-start emit and `on_turn_committed()` immediately after
  `messages.append(assistant_msg)`, with notes explaining the inflight-
  buffer reset semantics. PNG regenerated.
2026-05-09 17:23:55 -07:00
Patrick Buckley 43e622840b feat(sse): refresh-resume for mid-stream page reloads
Refreshing a coordinator or interactive workstream pane while the LLM
is mid-stream now restores the partial assistant text + reasoning
immediately and flips the composer back to stop-mode, instead of
showing nothing until the response completes.

Per-turn inflight buffers (`_ws_inflight_content`, `_ws_inflight_reasoning`,
`_ws_inflight_seq`) on `SessionUIBase` are kept separate from the
existing multi-turn `_ws_turn_content` buffer that drives the
dashboard's IDLE-piggyback payload. New `on_turn_start` (top of
send-loop, defensive) and `on_turn_committed` (right after
`messages.append(assistant_msg)`, primary) lifecycle hooks reset
inflight at turn boundaries. The seq counter is monotonic across
turns so a long-lived subscriber's `snap_seq` cutoff stays valid for
the lifetime of the connection — resetting per-turn would silently
drop turn N+1's first M tokens (M = whatever was streamed pre-snapshot
in turn N).

`snapshot_and_consume_state_payload` also drains inflight at idle/error
so cancel and exception paths don't leak stale text. New
`register_listener_with_in_progress_snapshot` atomically registers a
listener and snapshots the inflight buffers; `make_events_handler`
emits a `state_change` event (so the JS busy machine flips to
stop-mode) followed by a one-shot `in_progress_snapshot` after the
kind-specific replay, then strips the internal `_seq` field from
yielded live events while filtering against `snap_seq`. A per-listener
shallow `dict` copy in the live drain prevents the multi-tab race
where one listener's `del event["_seq"]` would corrupt another
listener's filter view.

`_synthesize_cancelled_results` now emits synthetic `on_tool_result`
events for each cancelled tool so live coord tabs can drop the
newly-additive `coord-tool-batch--running` indicator cleanly. The
indicator now coexists with `--auto`/`--approved` (applied on
`tool_info` and `approval_resolved` approved; removed when every row
in the batch has a result), making live tool execution visually
parallel to the replay-time orphan rendering.

Frontend handlers in `app.js` (interactive) and `coordinator.js` (coord)
absorb EventSource auto-reconnect re-replays via a length-based
prefix check on the in-progress buffer. New `InProgressSnapshotEvent`
+ `StateChangeEvent` dataclasses in the Python and TypeScript SDKs
with type guards.

`_MAX_TURN_CONTENT_CHARS` lifted 256 KiB → 512 KiB (single constant
for both buffers — headroom for current commercial models).

Regression tests cover race-free composition under concurrent writers,
seq-filter dedup invariants, the cross-turn seq monotonic invariant,
idle/error inflight drain, synthesized `on_tool_result` on cancel
(including UI-hook failure isolation), and the multi-listener
shared-dict invariant.
2026-05-09 17:23:55 -07:00
210 changed files with 22926 additions and 4785 deletions
+649 -3
View File
@@ -8,13 +8,659 @@ version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
Three release tracks are maintained:
- **`stable/1.0`** — patch-only (`v1.0.x`)
- **`stable/1.3`** — patch-only (`v1.3.x`)
- **`stable/1.4`** — patch-only (`v1.4.x`)
- **`main`** — experimental (`v1.5.0aN`)
- **`stable/1.5`** — patch-only (`v1.5.x`)
- **`main`** — experimental (next major)
## [Unreleased]
## [1.5.18]
Backports the `turnstone-admin` config-loading alignment from `main`
plus the accompanying `load_config` permission-warning hardening. No
schema changes.
### Added
- **`turnstone-admin` reads `config.toml`** — the admin CLI now honors
the same `[database]` section that `turnstone-server` does, with the
same precedence (`CLI / config.toml > TURNSTONE_DB_* env > defaults`).
Operators with DB credentials in `config.toml` no longer need to
re-export `TURNSTONE_DB_URL` before every admin invocation. Newly
plumbed through to `init_storage`: `pool_size`, `sslmode`,
`sslrootcert`, `sslcert`, `sslkey` — previously the admin CLI
silently dropped these. A new `--config PATH` flag mirrors the
one already on `turnstone-server`.
### Security
- **Permissive `config.toml` now warns** — `turnstone.core.config.load_config`
logs a single warning when the resolved config file is group- or
world-readable (any bit in `0o077`). DB password and TLS key paths
live in `[database]`; operators usually want the file at `0600`.
## [1.5.17]
Backports a clutch of coordinator-tool clarity fixes plus a watch-delivery
correctness fix from `main` to the `stable/1.5` track, plus a previously-
latent intent-verdicts persistence bug exposed by the new heuristic-verdict
INSERT paths. No schema changes.
### Fixed
- **`intent_verdicts` PK collisions on every llm_fallback delivery** —
async LLM-tier "llm_fallback" verdicts (`turnstone/core/judge.py`
`_deliver_fallbacks` and the in-loop fallback path) deliberately
reuse the heuristic verdict's `verdict_id` so the row gets
"upgraded in place" from `tier="heuristic"``tier="llm_fallback"`
when the LLM judge times out, is cancelled, or returns no content.
The consumer `_persist_intent_verdict` was doing a plain INSERT,
hitting the `intent_verdicts_pkey` constraint on every fallback
delivery; Postgres logged the duplicate-key error, the application
try/except swallowed it at `log.debug`, and the row never actually
got upgraded — the LLM judge's annotation
(`"(LLM judge did not return a verdict)"`) was lost. The collision
rate exploded on this release because the new heuristic-INSERT
paths in the auto-approve early-return branches of `approve_tools`
(introduced below) leave no gap for the fallback to land cleanly
into. Fix: new `upsert_intent_verdict` storage method using
`ON CONFLICT (verdict_id) DO UPDATE` that updates only `tier`,
`reasoning`, `judge_model` — the three fields that genuinely
change between heuristic and llm_fallback. Every other column
(identity, carried-verbatim, and `user_decision`) is excluded;
`user_decision` in particular would otherwise be clobbered back
to `"pending"` when a fallback arrives after the operator has
already resolved the approval. The bulk-INSERT path stays as
plain INSERT — fresh UUIDs in `judge.evaluate` make in-turn dups
impossible; the inverse race (fallback wins before bulk lands) is
reachable but unchanged in observable behavior by this fix,
documented at the bulk site for a future hardening pass.
- **Coordinator LLM re-spawn loops on large fan-outs** — the spawn-tool
return JSON used `ws_id` as its key, which primed the model's recency
bias to feed the spawn result straight back into another
`spawn_workstream(ws_id=...)` call instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this cascaded
into self-inflicted re-spawn loops. The LLM-facing tool result now emits
`child_ws_id` (the storage column / HTTP API contract is unchanged); the
field name is already an existing project term so the rename aligns
rather than introduces new vocabulary. Also handles the silent
upstream-omits-ws_id success-shape edge that previously emitted
`{"child_ws_id": null}` to the LLM — now surfaces a tool error so the
model retries rather than chasing a null id.
- **`inspect_workstream` blowing the coordinator context budget** — a
coord doing a fan-out wave against tool-heavy children could land
>100 KB of raw output per inspect call, and the previous safety net
(`_truncate_output`'s head+tail strategy) silently dropped *middle*
messages — exactly the wrong shape for understanding a child's
trajectory (the FIRST sets the brief, the LAST shows the conclusion,
the middle is the connective tissue). Output now goes through a
three-tier degradation ladder mirroring the search tool's
`_format_search_results`: `_tier="full"` (every message verbatim) →
`_tier="compact"` (per-message head/tail-snipped content + snipped
`tool_calls.arguments`, falling through a `(20,30)` / `(10,20)` /
`(5,10)` message-list trim ladder) → `_tier="skeleton"` (counts, role
distribution, last-assistant preview). Budget 32 KiB matches the
search tool's; the chosen tier is annotated on the response so the
model can recall with a tighter `message_limit` if signal was lost.
- **Auto-approved verdicts indistinguishable from pending review** —
`intent_verdict` rows for auto-approved tool calls landed with
`user_decision=""`, which read identically to "still waiting for the
operator" in the audit trail and led to a real misdiagnosis incident.
The column now carries an explicit vocabulary at insert: `pending` /
`approved` / `denied` / `timeout` / `policy` / `blanket` / `skill` /
`always` / `auto_approve_tools`. The auto-approve early-return
branches in `approve_tools` now persist heuristic verdicts stamped
with their reason (previously dropped on the floor), and late LLM-tier
verdicts that arrive for an already-auto-approved call_id are stamped
via a TTL-pruned lookup map — so the audit row carries the
auto-approve reason even when the LLM judge daemon completes after
the synchronous approval cycle finished. `resolve_approval` gains a
`timeout` kwarg writing `"timeout"` (the previous shape collapsed
passive timeouts and active denials into the same column).
- **`list_skills` empty `allowed_tools` misread as "no tool access"** —
the response previously emitted `"allowed_tools": []` for every skill
that hadn't declared an auto-approve allowlist, which a coordinator
model read as "this skill can't use any tools" (real misdiagnosis: a
code-review child appeared to have been spawned with zero tool
access). The field is now omitted entirely when empty — absence
carries the unambiguous meaning "no tool is pre-approved for this
skill", presence (non-empty list) keeps the standard Claude Code
skill-spec shape. The tool description rewrite makes the
auto-approve-allowlist semantics explicit so a future reader doesn't
re-derive the gating misread.
- **Watch terminal-fires silently dropped on backpressure** —
delivery now routes terminal events through the same path as
normal fires instead of being filtered out when the consumer was
saturated.
### Documentation
- **Storage `LIKE_ESCAPE` contract** — clarify that callers passing
`.like(escape=...)` must use the same escape character that the
storage helper assumes; previous wording let a reader pass a
different escape and silently produce no matches.
## [1.5.15]
### Fixed
- **Admin console blank-page on MCP server rows with consented users** — a
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
delimiters on the bulk-revoke button HTML literal, but the literal embeds
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
into bare tokens, and the resulting `SyntaxError` wiped out every global
in `admin.js``showAdmin` and all other admin entry points became
undefined, so the console UI was non-functional whenever the rendered MCP
server list contained at least one row with `consented_users_count > 0`.
Switch the literal to single-quote delimiters to match the surrounding
block.
## [1.5.14]
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
### Added
- **OAuth-MCP Phase 9 — admin status, deferred-consent persistence, operator
docs** — completes the per-(user, server) OAuth-MCP build-out. The sync pool
dispatchers now upsert into a new `mcp_pending_consent` table on
`mcp_consent_required` / `mcp_insufficient_scope`, so a non-interactive run
(scheduled / channel) that hits an unconsented server surfaces the deferred
prompt to the user on their next dashboard load via the gear-icon badge —
rows are cleared automatically by the OAuth callback handler on consent
completion, or via new DELETE endpoints for manual dismiss. The MCP Servers
admin row gains a `consented_users_count` pill and a two-step-confirm
bulk-revoke button for `auth_type=oauth_user` servers (upstream RFC 7009
revoke is intentionally not attempted in bulk to avoid N synchronous
round-trips against the provider). Operator-facing docs land at
`docs/mcp-oauth.md` and `docs/operations/mcp-oauth-headless.md`.
Introduces forward-only migrations `054_mcp_pending_consent` and
`055_mcp_user_tokens_server_index`.
## [1.5.13]
This release introduces one forward-only schema migration:
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
the dispatcher uses in-process fan-out).
### Added
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
per-channel handlers on a separate dispatch thread. The cluster collector
subscribes to a new `services` channel and reacts to node register /
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
loop; the 60 s loop is retained as the backstop for crash-shaped loss
(NOTIFY only fires on real writes). The storage layer also gains a uniform
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
`[database] listen_url` in `config.toml`) points the dispatcher at a
direct-to-Postgres URL; defaults to the main DB URL when unset.
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
whenever a child state change is dispatched to the UI, and the wait loop
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
storage ~2400 times now wakes only on real state transitions, with ~4× lower
SSE traffic in the quiescent case.
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
route previously emitted only `memory.delete`, so tool-initiated mutations
had no audit footprint). All emissions are best-effort and never break the
tool call itself.
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
an optional `skill=<name>` argument that loads the named skill's content as
the sub-agent's persona in place of the hardcoded identity statement. The
fixed operating-guidance block (one-shot, tool-use over narration,
no follow-up questions) is still layered on top of every persona. High- and
critical-risk skills surface their risk tier in the approval header and
emit a `task_agent.high_risk_skill` warning, matching the existing
session-load gate.
### Fixed
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
back-compat `default` alias auto-synthesised by `load_model_registry`
remained visible to the model even when an operator had configured
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
routed to whichever backend the synthesised alias was attached to at boot
instead of the configured per-role default. The synthesised alias is now
only added when neither the DB nor `[models.*]` populates the registry,
filtered out of the LLM-visible alias list, and explicitly rejected at the
validator chokepoint as defense-in-depth.
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
node labels were re-entering the shape parser and producing
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
affected label forms (`|content|` and `ID[content]`) before the SVG cache
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
highlights code blocks progressively as they stream rather than only after
completion.
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
POST went to `/v1/api/auth/login` and was rewritten to
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
public path, and `proxy_api` would have forwarded the login request to the
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
handlers. The dispatch table is a single `(method, path) → handler` mapping
so the test parametrize list can't drift from the implementation.
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
overlay was covering the entire appbar, hiding the proxy-injected node
picker. The overlay now starts at `top: 48px` and the dashboard's role
downgrades from `dialog+aria-modal` to `region` so the appbar above it
remains reachable. The gear icon converts from a direct settings-panel
click into a dropdown with "MCP connections" and "Logout" (the latter with
`.destructive` styling). The settings-menu keydown handler is now attached
synchronously so `Escape` can't fall through the brief window between the
menu opening and its listeners being installed.
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
`services_notify` trigger lives only in the alembic chain, but the test
fixture creates tables via `metadata.create_all`. The trigger function +
trigger are now declared in `_schema.py` and attached via
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
PostgreSQL dialect, with the same SQL constants imported by migration 053
so there's a single source of truth.
## [1.5.12]
### Added
- **Enriched backend error messages** — provider name and attempted URL are now
included in session error responses, so operators can triage connectivity
failures without enabling debug logging.
### Fixed
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
had no messages remaining after a rewind the history event was skipped,
leaving connected UIs with stale content and blocking edit-and-resend flows.
## [1.5.11]
This release introduces one forward-only schema migration:
`052_model_reasoning_persistence``surface_persisted_reasoning` and
`replay_reasoning_to_model` flag columns on `model_definitions`.
### Added
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
restore) now receive an `in_progress_snapshot` event carrying the buffered
partial response, so the UI can resume rendering the in-flight turn without
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
counter so a reconnecting client can skip events it already saw.
- **Reasoning persistence** (Phases 14) — model reasoning text can now be
persisted to conversation history and optionally replayed to the model on
subsequent turns. Phase 1 persists reasoning text on the history payload.
Phase 2 wires a build-time shape filter and a per-model
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
new per-model capability flags (`surface_persisted_reasoning`,
`replay_reasoning_to_model`) both default `False` on unknown and
local-server models.
- **Console home composer: placeholders + toggle** — the console landing-page
composer now shows context-aware placeholder text and a toggle component for
advanced options; an admin polish pass tightened spacing and focus behaviour
across the form.
### Changed
- **`judge.model` now requires a named alias** — raw provider model IDs on
`judge.model` in config are no longer accepted; the judge must reference an
alias registered in the model registry. The session-provider raw-model
fallback is removed. Existing configs using an unregistered model ID need a
corresponding alias entry.
### Fixed
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
flag for a model that does not declare reasoning-replay support now silently
no-ops instead of forwarding reasoning blocks and triggering a provider error.
- **Coordinator alias resolution unified across placeholder + factory** — a
placeholder coordinator and the real coordinator factory could previously
resolve to different model aliases, producing a visible mismatch in the model
display. Both paths now share the same resolution logic.
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
under-initialised coordinator state no longer 500s when the models endpoint
is hit before the coordinator subsystem is fully bootstrapped.
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
counter that broke `state_change` / `in_progress_snapshot` ordering on
reconnect.
- **Reasoning persistence shape + replay fixes** — per-block
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
synthesised alongside non-reasoning `provider_blocks` so both appear
together in the history payload.
## [1.5.10]
This release introduces one forward-only schema migration:
`051_skill_notify_on_complete_array_default` — backfills
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
### Added
- **Skills unlock action** — operators can unlock an installed skill to allow
local customisation. Once unlocked, the skill's resource content, system
prompt additions, and notify configuration are editable through the admin UI.
Skills shipped as part of a bundle remain locked (read-only) until explicitly
unlocked; the unlock is logged to the audit trail. A lock icon in the
top-right of the Skills detail pane doubles as the unlock trigger.
### Fixed
- **`skills.sh` install endpoint** — the install script was targeting an
endpoint removed in an earlier refactor; switched to `/api/download`.
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
(object) instead of `[]` (array), causing notify configurations to be
rejected at schema validation.
- **Skills admin UI modal errors** — `.is-visible` class used consistently
instead of inline `style.display`; stale error text is cleared on submit;
designer-review lock-icon UX applied.
## [1.5.9]
### Fixed
- **`repair=False` on all display-read `load_messages` call sites** —
passing `repair=True` on display paths was silently mutating the stored
message list, causing divergence between what the UI showed and what the
model received on the next turn.
## [1.5.8]
This release introduces two forward-only schema migrations:
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
`050_conversations_source_and_reminders``_source` and `_reminders` columns
on `conversations`.
### Added
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
configured with a client ID and secret through the admin UI. The full token
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
stored encrypted at rest using a key derived from the JWT secret. The consent
flow runs in-browser via a provider redirect. Rolled out in phases:
- Minimum admin form and OAuth schema (`21663d15`).
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
- SDK 401/403 introspection via httpx response hook (`bde09134`).
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
their OAuth token is permitted to call (`cfc8a6c8`).
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
use of an OAuth-gated server and can revoke consent from their profile;
admins see per-server consent counts in the MCP Servers tab (`61051339`).
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
that delivers at most one nudge per turn, preventing multi-channel pile-ups
that inflate context. Observable changes:
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
through to the system message so the model can reason about recency.
- Coordinator idle-children observer: a coordinator with no in-flight
children for longer than the configured idle threshold receives a nudge.
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
rather than the previous `_watch_pending` list, giving them the same
delivery guarantees and priority handling as other advisories.
- **Structured watch-result card** — the UI renders watch results as a styled
card with a system-nudge marker, distinct from the assistant message body.
On history replay, system-nudge turns are visually distinguished from normal
assistant turns.
- **Side-channel persistence** — `_source` and `_reminders` side-channel
fields are persisted to the `conversations` storage table and restored on
session resume, so metacognitive context survives process restarts. A
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
### Fixed
- **Replay consistency** — queued user messages captured mid-loop are now
persisted and replayed in the correct order on a subsequent `events`
subscription. Coordinator history replay fixed: blank assistant cards and
out-of-order tool results on the coordinator tree no longer occur when the
coordinator has mixed queued + delivered messages.
- **Session reminder preservation on fork + resume** — `_source` and
`_reminders` are carried through workstream fork and restored from storage
on resume.
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
columns; `_source` and `_reminders` now strip NUL bytes on write.
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
now committed atomically on first model add; startup teardown is offloaded
to avoid blocking the event loop.
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
now replaces the pool entry with a fresh session; the carrier token is
owned by the pool entry to prevent a race between the 401 handler and a
concurrent request.
- **OIDC hardening** — multiple security and correctness fixes:
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
callback robustness — typed exceptions, shape checks, log sanitization, JS
race (bug-46, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
stranded-user self-heal on role-mapping failure (cumulative bug-1).
## [1.5.7]
### Added
- **Inline node picker** — a compact node-switcher dropdown in the console
header replaces the "← Back to console" banner, so operators can switch
between nodes without a full navigation.
### Fixed
- **Queued user messages injected mid-loop** — messages queued while a
generation was in progress were not being delivered at the correct seam and
could be dropped or reordered when the worker consumed the queue.
- **Search tool output bounded** — pathological inputs (very long lines with
no whitespace) could produce search results exceeding the context budget.
Output is now clamped before reaching the message.
## [1.5.6]
### Added
- **`api_surface` toggle** — model definitions gain an `api_surface` field
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
the provider client uses. Enables Mistral Medium reasoning via the Responses
surface; Chat Completions remains the default for all other models.
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
includes a `healthy_aliases` list per node, so the coordinator and operators
can see which model aliases are currently reachable without a separate
per-model health probe.
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
configure per-kind reasoning effort and alias overrides from the UI rather
than editing `config.toml`. Live-refresh dropdowns update in place when
model definitions change.
### Fixed
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
query-aware candidate-set selection, dramatically improving recall for
queries whose terms span multiple stored entries.
- **Workstream model + config preserved on rehydrate** — reopening a closed
workstream no longer overwrites the model alias and per-workstream config
with session defaults.
- **Console home composer: attachments + user-message pills** — multipart
attachments in the home composer were not forwarded correctly; user-message
pills in the coordinator chat pane were missing.
## [1.5.5]
### Fixed
- **Saved-workstream tool result rendering** — tool results in closed
workstreams were not rendering on history replay. Audit-trail decoration for
tool calls is now applied on the replay path.
## [1.5.4]
### Added
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
first-class citizens in the cluster event bus. `child_ws_state` events are
pushed through the cluster SSE stream so the console tree view updates in
real time without polling. `list_children` and `get_child` primitives on
`SessionManager` provide a consistent cross-node view of the coordinator's
spawn tree.
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
in the console admin panel now supports checkbox multi-select with a
bulk-delete action.
## [1.5.3]
This release introduces one forward-only schema migration:
`048_workstream_reaper_index` — partial composite index on `workstreams` for
the orphan-reaper query.
### Fixed
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
`close_idle` pass now scopes the DB-orphan reaper by
`services.last_heartbeat` so workstreams belonging to a live node are not
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
storage primitives added; a partial composite index keeps the reaper scan
cheap.
- **Coordinator pool idle cleanup** — a periodic task on the console now
closes coordinator pool entries whose session has gone idle past the
configurable threshold, preventing pool exhaustion on long-running consoles.
## [1.5.2]
### Added
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
nudges are rendered as a distinct styled bubble rather than being injected
inline into the assistant message, making it easier to distinguish model
output from metacognitive annotations. The CLI REPL gains matching
`on_user_reminder` / `on_tool_reminder` callbacks.
### Fixed
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
now fires correctly on the third repetition; a write-success-clear that
reset the counter after a successful tool call (preventing streaks across
mixed-outcome sequences) was removed.
- **Metacog reminders isolated to side-channel** — reminder text no longer
appears in the user content turn; it flows through a dedicated side-channel
the session injects into the system context, preventing the model from
attributing it to the user.
## [1.5.1]
### Added
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
now receive the child's pending approval detail in `child_ws_state` events,
enabling the coordinator to surface approval prompts without a separate poll.
### Fixed
- **Coordinator registry auto-refresh** — the console coordinator registry now
refreshes when model definitions change, so a newly added alias is visible
to coordinators without restarting.
- **Coordinator fan-out default** — coordinators now fan out to independent
child workstreams by default instead of serialising them, matching the
documented contract for parallel-work patterns.
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
summaries and tool results from child workstreams were silently truncated at
the previous 4 KiB cap.
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
polling now runs on a dedicated 200-thread executor, matching interactive's
`sse_executor`, so coordinator long-poll blocking no longer contends with
storage and routing workers on the default pool.
## [1.5.0]
User-visible additions: a unified workstream HTTP surface (interactive and
coordinator under one URL family), inline child approvals, coordinator
composer parity, progressive rendering, OIDC authentication, MCP OAuth
foundations, and a redesigned UI built on the Design System v1 token layer.
This release removes the pre-1.5 body-keyed and query-keyed URL family.
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
This release introduces the following forward-only schema migrations that the
server applies automatically on first startup. All are additive; no data loss.
- `039_workstream_kind``kind` + `parent_ws_id` columns on `workstreams`.
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
`admin.cluster.inspect` to the builtin-admin role.
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
introduced by 039.
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
builtin-admin.
- `043_skill_description_required` — backfills empty `description` rows in
`prompt_templates`.
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
(`interactive` / `coordinator` / `any`).
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status`
`risk_level`.
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
by rendezvous routing in 1.4.
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
rows removed from the coordinator in 1.5.0a4.
### Added
- **Inline child approvals** — pending tool approvals on coordinator child
workstreams surface directly in the coordinator tree view. A risk pill shows
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
buttons appear inline so operators do not need to navigate to the child's
workstream. `pending_approval_detail` is exposed on
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
so all connected clients render approval prompts simultaneously. LLM judge
verdicts are cached client-side and replayed on SSE reconnect.
- **Coordinator composer parity** — the coordinator composer now supports
Stop, Send-to-queue, and Attach (file upload), matching the interactive
workstream composer feature set.
- **Per-call model and judge override on coordinator composer** — operators
can override the model alias and judge model for a single coordinator send
from the composer, without changing the node-wide or role-wide defaults. Bad
aliases return a corrective error listing available choices.
- **Coordinator status bar + richer history replay** — each coordinator
workstream gains a per-coordinator status bar showing active children, token
spend, and generation state. History replay in the coordinator panel is
extended to include tool results and thinking blocks.
- **Coordinator child error surfacing + memory tool** — child workstream
errors are surfaced as distinct error rows in the coordinator tree view
rather than disappearing silently. The coordinator gains access to a
`memory` tool (same interface as interactive) for retrieving stored facts.
- **Coordinator inline tool-batch construct** — the coordinator tool approval
UI replaces the separate approval dock with an inline batch construct that
groups all pending tool calls for a given turn into a single review card.
- **Node capability auto-detection** — nodes report kernel-level capabilities
(available memory, CPU count, accelerator presence) via
`/v1/api/node/capabilities` at startup, enabling the console to filter model
aliases offered to coordinators routing to that node.
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
a `SKILL.md` file's content into the modal auto-populates the name,
description, and configuration fields.
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
soon as a complete diagram block is detected in the stream rather than
waiting for the full response; the diagram re-renders in place as the model
extends it.
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
math delimiters are now recognised alongside the existing `$$` fences.
### Removed (BREAKING — 1.5.0)
- **Legacy body-keyed and query-keyed URL family for the workstream
+1 -1
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.14 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
+46 -1
View File
@@ -281,6 +281,7 @@ Each message in the `messages` array has:
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
Each entry in `tool_calls`:
@@ -325,6 +326,44 @@ finalize any in-progress assistant message.
{"type": "stream_end"}
```
**`state_change`** -- the worker thread transitioned to a new state. Drives
the client's busy-mode (composer in send vs. stop, spinner indicators,
auto-focus on idle). Sent live during normal operation AND on every fresh
SSE subscribe (so a mid-stream page refresh restores the correct composer
state without waiting for the next live transition).
```json
{"type": "state_change", "state": "running"}
```
| Field | Type | Description |
|----------|--------|----------------------------------------------------------------------|
| `state` | string | One of `"running"`, `"thinking"`, `"attention"`, `"idle"`, `"error"` |
**`in_progress_snapshot`** -- one-shot replay of the in-progress turn's
content + reasoning text-so-far when this client connects mid-stream.
Lets a refreshing browser tab restore partial assistant text immediately
instead of waiting for the response to complete. Yielded once after the
kind-specific replay phase (history + pending), only when at least one
of `content` / `reasoning` is non-empty. Both halves render into the same
assistant bubble the live `content` / `reasoning` events would target;
clients should treat the snapshot as idempotent (skip overwrite if the
current local buffer is already a superset prefix — covers EventSource
auto-reconnect re-replays).
```json
{
"type": "in_progress_snapshot",
"content": "Here is the answer so far: it depends on ",
"reasoning": "The user is asking about a comparison; let me think about..."
}
```
| Field | Type | Description |
|--------------|--------|------------------------------------------------------------|
| `content` | string | Joined assistant content text accumulated this turn |
| `reasoning` | string | Joined reasoning / chain-of-thought text accumulated |
**`tool_info`** -- one or more tool calls that were auto-approved (no user
action required).
@@ -522,7 +561,13 @@ Each SSE connection to a workstream receives its own delivery queue. Events
produced by the worker thread are fanned out to all registered listener queues,
so multiple consumers (browser, console proxy, SDK) can connect
simultaneously and each receives every event. On reconnect the client receives
a full history replay, so no catch-up mechanism is needed.
the kind-specific replay (`connected` + `status` + `history` + pending
approval / plan for interactive; `connected` + `status` + pending for coord)
followed by a `state_change` carrying the current worker state and an
optional `in_progress_snapshot` carrying any partial content / reasoning
buffered for the in-progress turn — so a mid-stream refresh restores both
the busy-mode UI and the partial assistant text without waiting for the
response to complete.
---
+45 -5
View File
@@ -91,7 +91,7 @@ turnstone/
discord/ Discord adapter (bot, cog, views, streaming, config)
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
katex-0.16.47/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
ui/
colors.py ANSI color constants with NO_COLOR support
markdown.py Streaming terminal markdown renderer (line-buffered)
@@ -231,11 +231,13 @@ The engine emits state changes via `_emit_state()` which calls
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 16
methods. Every frontend must implement all of them.
```python
class SessionUI(Protocol):
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -252,6 +254,14 @@ class SessionUI(Protocol):
def on_rename(self, name: str) -> None: ... # propagate alias to tab/UI label
```
`on_turn_start` fires at the top of each iteration of the send-loop;
`on_turn_committed` fires immediately after `messages.append(assistant_msg)`.
`SessionUIBase` uses both to reset the per-turn inflight buffers
(`_ws_inflight_content` / `_ws_inflight_reasoning` / `_ws_inflight_seq`)
that fuel the SSE refresh-resume `in_progress_snapshot` event — see
the per-workstream events stream in
[`docs/api-reference.md`](api-reference.md#get-v1apiworkstreamsws_idevents).
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
### Three Implementations
@@ -619,14 +629,15 @@ LLMProvider (protocol)
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
| `retryable_error_names` | Exception class names that trigger retry |
| `extract_reasoning_text()` | Walk stored `provider_blocks`, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic `thinking`, OpenAI Responses `reasoning`, OpenAI Chat synthetic `reasoning_text`) |
**Normalized data types:**
| Type | Fields |
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
@@ -714,6 +725,35 @@ and `"openai-compatible"`.
`max_tokens`, and `reasoning_effort` to override the global defaults from
ConfigStore. When unset (`NULL`), the global default is used.
**Per-model reasoning persistence:** Two booleans on `model_definitions`
(migration 052) control how reasoning text round-trips:
* `surface_persisted_reasoning` (default `True`) — gates whether stored
reasoning text is surfaced on `/history` payloads for UI rehydration.
**Storage of reasoning bytes happens regardless of this flag** — they
ride in `provider_data` independently. Phase-1 admin UI label "Surface
persisted reasoning."
* `replay_reasoning_to_model` (default `False`) — gates whether stored
reasoning blocks are sent back to the provider on subsequent turns.
Capability-gated: `ModelCapabilities.supports_reasoning_replay` must
also be `True` for the wire path to actually replay (canonical OpenAI
gpt-5*/o-series and Anthropic Claude entries set it; unknown / local-
server models default to `False`).
Three reasoning paths are recognised:
| Path | Provider | Capture | Persist | Replay |
|------|----------|---------|---------|--------|
| 1 | Anthropic Messages API | `thinking_delta` | `provider_blocks` (`type="thinking"`) | Verbatim via `_provider_content` |
| 2 | OpenAI Responses (gpt-5*, o-series) | `response.reasoning_text.delta` events | `provider_blocks` (`type="reasoning"`) — only when `include=["reasoning.encrypted_content"]` | `ResponseReasoningItemParam` input items |
| 3 | OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat) | `delta.reasoning_content` Pydantic extras | Synthetic `{type: "reasoning_text", text, source}` block stamped at end-of-stream | None — no API surface for replay on Chat Completions |
Cross-provider safety is enforced by `ANTHROPIC_VALID_BLOCK_TYPES` (a
shape filter in `_anthropic.py:_convert_messages`): foreign blocks
(OpenAI `reasoning`, synthetic `reasoning_text`) fall through to the
text+tool_calls rebuild path rather than reaching Anthropic's input
boundary as malformed content.
```toml
[models.local]
base_url = "http://localhost:8000/v1"
+9 -2
View File
@@ -110,11 +110,18 @@ owns it; the node is just currently unreachable.
### Example — `spawn_batch`
This is the coordinator-tool result shape (the JSON the LLM receives),
not an HTTP API response — the table above keys it under "model tool"
to distinguish it from the `/v1/api/...` endpoints in the same table.
The underlying HTTP spawn endpoint still returns `ws_id`; the tool
result re-keys it to `child_ws_id` to defuse a coordinator-LLM recency
bias (see `docs/coordinator-skills.md`).
```json
{
"results": {
"0": {"ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
"0": {"child_ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"child_ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
},
"denied": [
{"idx": 1, "reason": "skill not found: nonexistent-skill"}
+9 -4
View File
@@ -115,7 +115,8 @@ with a `type` field. The recurring shapes a UI has to handle:
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `rename` | Session's display name changed | `name` |
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
@@ -130,9 +131,13 @@ with a `type` field. The recurring shapes a UI has to handle:
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved) and any in-flight `wait_*` / `batch_*`
indicator — so a tab refresh mid-approval doesn't strand the
operator.
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
---
+12 -5
View File
@@ -169,14 +169,21 @@ validates ws_id against `parent_ws_id=coord_ws_id` AND
the wait into reporting "complete".
Pattern: capture each spawn result in the next tool call's input.
The JSON tool-result carries `{"ws_id": "...", "name": "...",
The JSON tool-result carries `{"child_ws_id": "...", "name": "...",
"node_id": "...", "routing_strategy": "..."}`; the model should
extract the ws_id and pass it to `inspect_workstream` /
`wait_for_workstream` / `send_to_workstream` / `close_workstream`
verbatim.
extract the `child_ws_id` and pass it as `ws_id` (or in the `ws_ids`
list) to `inspect_workstream` / `wait_for_workstream` /
`send_to_workstream` / `close_workstream` verbatim. The asymmetry
— spawn returns `child_ws_id` but the other tools accept `ws_id` /
`ws_ids` — is intentional: it defuses a coordinator-LLM recency
bias where seeing `ws_id` in a spawn return primed re-spawn loops
instead of progression to the wait phase.
A UI that wants human-readable identifiers should render the `name`
field and keep the ws_id as the click-through key.
field and keep the workstream id as the click-through key — note
that the id *value* is the same regardless of whether it arrived
under the `child_ws_id` key (spawn return) or the `ws_id` key
(every other tool's input/output); only the field name differs.
---
+4 -2
View File
@@ -69,9 +69,10 @@ class "NullUI" as NullUI {
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ...) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ...) → CompletionResult
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
+ convert_tools(tools) → list[dict]
+ extract_reasoning_text(provider_blocks) → str
+ retryable_error_names: frozenset[str] {property}
--
core/providers/_protocol.py
@@ -126,6 +127,7 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_web_search: bool
+ supports_tool_search: bool
+ supports_vision: bool
+ supports_reasoning_replay: bool
}
' ChatSession
+16
View File
@@ -24,6 +24,14 @@ CS -> DB : save_message(ws_id, "user", input)
group loop [while tool_calls present]
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
@@ -73,6 +81,14 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:25b5448bbb7da8ddafe4f65c6c5e6cbcaa9cb9f31746ca46d3a2241bc47b1956
size 259687
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3aa8d972bba40d78152f9f0c762b9f5ec616d8052c45fa52b7dd1c679ed81d61
size 325245
oid sha256:c14dfbb2db8dcb22cd332b2cf0e53ba75141adb213dae47dd9dbfd389ed482fe
size 354799
+1
View File
@@ -84,6 +84,7 @@ Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
|----------|---------|-------------|
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_LISTEN_URL` | (falls back to `TURNSTONE_DB_URL`) | Direct-to-PostgreSQL URL for the console's dedicated `LISTEN` connection. Set this when `TURNSTONE_DB_URL` points at PgBouncer in transaction pooling mode — LISTEN is session state and the transaction-pooled connection can't hold it. See [pgbouncer.md](pgbouncer.md). |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
+117
View File
@@ -0,0 +1,117 @@
# MCP OAuth — per-user authorization for MCP servers
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
---
## When to use which `auth_type`
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
| `auth_type` | What it means | When to use |
|---|---|---|
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
## Prerequisites for `auth_type=oauth_user`
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata** *or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
3. **OAuth client registration**. Two paths:
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
---
## Configuration
### Per-server fields (admin UI)
| Field | Required | Description |
|---|---|---|
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
| Multitenant Authorization | Yes | `none` / `static` / `oauth_user` (recommended). |
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
| Client Registration | Yes (oauth_user) | `preregistered` or `dynamic`. |
| Client ID | Yes (preregistered) | OAuth 2.0 client ID. Stored unencrypted. |
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
### Encryption key
```toml
[security]
mcp_token_encryption_key = "base64-fernet-key"
# For rotation, list the keys in priority order — first is used for new
# writes, all are tried for reads.
# mcp_token_encryption_keys = ["new-key", "old-key"]
```
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
---
## Admin status indicators
The MCP Servers admin tab shows per-server status pills (Phase 9):
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
---
## Auth-type transitions
| From | To | What happens |
|---|---|---|
| `none` / `static``oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
+29
View File
@@ -0,0 +1,29 @@
# MCP OAuth in headless / scheduled / channel-driven runs
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
**Affected surfaces**:
- Scheduled workstreams (`turnstone-console` task scheduler).
- Discord adapter runs.
- Slack adapter runs.
- Any future channel adapter without an interactive browser session.
**What happens when consent is missing**:
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
**Recovery**:
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
**Pre-consent recipe**:
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
1. Open the dashboard.
2. Open the settings modal (gear icon).
3. Click Connect on each MCP server the schedule will use.
4. Confirm consent in the popup.
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
+24
View File
@@ -199,4 +199,28 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
not use server-side prepared statements by default, so this is not an
issue.
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
transaction pooling assigns a real server connection only for the
duration of each transaction, then returns it to the pool. PostgreSQL
`LISTEN` is session state — a transaction-pooled client can't hold the
multi-statement session a long-lived `LISTEN` needs. The console's
`NotifyDispatcher` (reactive node discovery via the `services` channel)
therefore opens a **dedicated, direct-to-Postgres** connection that
bypasses PgBouncer.
Configure via `config.toml` `[database] listen_url` (preferred —
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
(config.toml wins when both are set). Defaults to the main DB URL when
unset.
| Setting | Behaviour |
|---|---|
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
Set this whenever PgBouncer is in transaction mode (the recommended
setting per this doc). The override only adds one long-lived PG
connection per console process — sized into the cluster's
`max_connections` budget alongside the pool.
See also: [Docker deployment](docker.md) · [Security](security.md)
+3
View File
@@ -138,6 +138,9 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `error` | `ErrorEvent` | `message` |
| `info` | `InfoEvent` | `message` |
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `cancelled` | `CancelledEvent` | — |
**Global events** (from `stream_global_events()`):
+15
View File
@@ -59,6 +59,21 @@ from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Reasoning persistence (per-model)
Two boolean flags on `model_definitions` (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|------|---------|--------|
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
provider-side mechanics (Anthropic `thinking`, OpenAI Responses
`reasoning` + `include=["reasoning.encrypted_content"]`, synthetic
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
### Plan / task agent overrides
`plan_agent` and `task_agent` sub-sessions resolve independently from the
+3 -3
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.5.10"
version = "1.5.18"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -82,9 +82,9 @@ include = [
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.16.45/**/*",
"turnstone/shared_static/katex-0.16.47/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.14.0/**/*",
"turnstone/shared_static/mermaid-11.15.0/**/*",
"turnstone/shared_static/hls-1.6.16/**/*",
"turnstone/sdk/py.typed",
"turnstone/deploy/*.yaml",
+6 -3
View File
@@ -50,11 +50,14 @@ update_refs() {
local old_pattern="$1" # e.g. katex-0.16.38
local new_pattern="$2" # e.g. katex-0.16.39
# Find all files with version references (excludes vendored JS and worktrees)
# Find all files with version references. Excludes the old versioned vendor
# directory itself (about to be rm -rf'd anyway) so we don't bother rewriting
# self-references inside it — but does NOT exclude all of shared_static/,
# because shared_static/renderer.js loads the vendored libs and needs the bump.
local files
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' \
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' --include='*.py' \
-F "$old_pattern" . \
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir='shared_static' \
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir="$old_pattern" \
2>/dev/null || true)
for f in $files; do
sed -i "s|${old_pattern}|${new_pattern}|g" "$f"
+127 -127
View File
@@ -74,9 +74,9 @@
}
},
"node_modules/@oxc-project/types": {
"version": "0.127.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz",
"integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==",
"version": "0.130.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.130.0.tgz",
"integrity": "sha512-ibD2usx9JRu7f5pu2tMKMI4cpA4NgXJQoYRP4pQ7Pxmn1l6k/53qWtQWZayhYy3X4QZkt90Ot+mJEaeXouio6Q==",
"dev": true,
"license": "MIT",
"funding": {
@@ -84,9 +84,9 @@
}
},
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.1.tgz",
"integrity": "sha512-fJI3I0r3C3Oj/zdBCpaCmBRZYf07xpaq4yCfDDoSFm+beWNzbIl26puW8RraUdugoJw/95zerNOn6jasAhzSmg==",
"cpu": [
"arm64"
],
@@ -101,9 +101,9 @@
}
},
"node_modules/@rolldown/binding-darwin-arm64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.1.tgz",
"integrity": "sha512-cKnAhWEsV7TPcA/5EAteDp6KcJZBQ2G+BqE7zayMMi7kMvwRsbv7WT9aOnn0WNl4SKEIf43vjS31iUPu80nzXg==",
"cpu": [
"arm64"
],
@@ -118,9 +118,9 @@
}
},
"node_modules/@rolldown/binding-darwin-x64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz",
"integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.1.tgz",
"integrity": "sha512-YKrVwQjIRBPo+5G/u03wGjbdy4q7pyzCe93DK9VJ7zkVmeg8LJ7GbgsiHWdR4xSoe4CAXRD7Bcjgbtr64bkXNg==",
"cpu": [
"x64"
],
@@ -135,9 +135,9 @@
}
},
"node_modules/@rolldown/binding-freebsd-x64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz",
"integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.1.tgz",
"integrity": "sha512-z/oBsREo46SsFqBwYtFe0kpJeBijAT48O/WXLI4suiCLBkr03RTtTJMCzSdDd2znlh8VJizL09XVkQgk8IZonw==",
"cpu": [
"x64"
],
@@ -152,9 +152,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz",
"integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.1.tgz",
"integrity": "sha512-ik8q7GM11zxvYxFc2PeDcT6TBvhCQMaUxfph/M5l9sKuTs/Sjg3L+Byw0F7w0ZVLBZmx30P+gG0ECzzN+MFcmQ==",
"cpu": [
"arm"
],
@@ -169,9 +169,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.1.tgz",
"integrity": "sha512-QoSx2EkyrrdZ6kcyE8stqZ62t0Yra8Fs5ia9lOxJrh6TMQJK7gQKmscdTHf7pOXKREKrVwOtJcQG3qVSfc866A==",
"cpu": [
"arm64"
],
@@ -189,9 +189,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-musl": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz",
"integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.1.tgz",
"integrity": "sha512-uwNwFpwKeNiZawfAWBgg0VIztPTV3ihhh1vV334h9ivnNLorxnQMU6Fz8wG1Zb4Qh9LC1/MkcyT3YlDXG3Rsgg==",
"cpu": [
"arm64"
],
@@ -209,9 +209,9 @@
}
},
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.1.tgz",
"integrity": "sha512-zY1bul7OWr7DFBiJ++wofXvnr8B45ce3QsQUhKrIhXsygAh7bTkwyeM1bi1a2g5C/yC/N8TZyGDEoMfm/l9mpg==",
"cpu": [
"ppc64"
],
@@ -229,9 +229,9 @@
}
},
"node_modules/@rolldown/binding-linux-s390x-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.1.tgz",
"integrity": "sha512-0frlsT/f4Ft6I7SMESTKnF3cZsdicQn1dCMkF/jT9wDLE+gGoiQfv1nmT9e+s7s/fekvvy6tZM2jHvI2tkbJDQ==",
"cpu": [
"s390x"
],
@@ -249,9 +249,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-gnu": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz",
"integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.1.tgz",
"integrity": "sha512-XABVmGp9Tg0WspTVvwduTc4fpqy6JnAUrSQe6OuyqD/03nI7r0O9OWUkMIwFrjKAIqolvqoA4ZrJppgwE0Gxmw==",
"cpu": [
"x64"
],
@@ -269,9 +269,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-musl": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz",
"integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.1.tgz",
"integrity": "sha512-bV4fzswuzVcKD90o/VM6QqKxnxlDq0g2BISDLNVmxrnhpv1DDbyPhCIjYfvzYLV+MvkKKnQt2Q6AO86SEBULUQ==",
"cpu": [
"x64"
],
@@ -289,9 +289,9 @@
}
},
"node_modules/@rolldown/binding-openharmony-arm64": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz",
"integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.1.tgz",
"integrity": "sha512-/Mh0Zhq3OP7fVs0kcQHZP6lZEthMGTaSf8UBQYSFEZDWGXXlEC+nJ6EqenaK2t4LBXMe3A+K/G2BVXXdtOr4PQ==",
"cpu": [
"arm64"
],
@@ -306,9 +306,9 @@
}
},
"node_modules/@rolldown/binding-wasm32-wasi": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz",
"integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.1.tgz",
"integrity": "sha512-+1xc9X45l8ufsBAm6Gjvx2qDRIY9lTVt0cgWNcJ+1gdhXvkbxePA60yRTwSTuXL09CMhyJmjpV7E3NoyxbqFQQ==",
"cpu": [
"wasm32"
],
@@ -325,9 +325,9 @@
}
},
"node_modules/@rolldown/binding-win32-arm64-msvc": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz",
"integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.1.tgz",
"integrity": "sha512-1D+UqZdfnuR+Jy1GgMJwi85bD40H21uNmOPRWQhw4oRSuolZ/B5rixZ45DK2KXOTCvmVCecauWgEhbw8bI7tOw==",
"cpu": [
"arm64"
],
@@ -342,9 +342,9 @@
}
},
"node_modules/@rolldown/binding-win32-x64-msvc": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz",
"integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.1.tgz",
"integrity": "sha512-INAycaWuhlOK3wk4mRHGsdgwYWmd9cChdPdE9bwWmy6rn9VqVNYNFGhOdXrofXUxwHIncSiPNb8tNm8knDVIeQ==",
"cpu": [
"x64"
],
@@ -359,9 +359,9 @@
}
},
"node_modules/@rolldown/pluginutils": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz",
"integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz",
"integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==",
"dev": true,
"license": "MIT"
},
@@ -402,23 +402,23 @@
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
"integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==",
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
"integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==",
"dev": true,
"license": "MIT"
},
"node_modules/@vitest/expect": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.5.tgz",
"integrity": "sha512-PWBaRY5JoKuRnHlUHfpV/KohFylaDZTupcXN1H9vYryNLOnitSw60Mw9IAE2r67NbwwzBw/Cc/8q9BK3kIX8Kw==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.6.tgz",
"integrity": "sha512-7EHDquPthALSV0jhhjgEW8FXaviMx7rSqu8W6oqCoAuOhKov814P99QDV1pxMA3QPv21YudvJngIhjrNI4opLg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@standard-schema/spec": "^1.1.0",
"@types/chai": "^5.2.2",
"@vitest/spy": "4.1.5",
"@vitest/utils": "4.1.5",
"@vitest/spy": "4.1.6",
"@vitest/utils": "4.1.6",
"chai": "^6.2.2",
"tinyrainbow": "^3.1.0"
},
@@ -427,13 +427,13 @@
}
},
"node_modules/@vitest/mocker": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.5.tgz",
"integrity": "sha512-/x2EmFC4mT4NNzqvC3fmesuV97w5FC903KPmey4gsnJiMQ3Be1IlDKVaDaG8iqaLFHqJ2FVEkxZk5VmeLjIItw==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.6.tgz",
"integrity": "sha512-MCFc63czMjEInOlcY2cpQCvCN+KgbAn+60xu9cMgP4sKaLC5JNAKw7JH8QdAnoAC88hW1IiSNZ+GgVXlN1UcMQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "4.1.5",
"@vitest/spy": "4.1.6",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.21"
},
@@ -454,9 +454,9 @@
}
},
"node_modules/@vitest/pretty-format": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.5.tgz",
"integrity": "sha512-7I3q6l5qr03dVfMX2wCo9FxwSJbPdwKjy2uu/YPpU3wfHvIL4QHwVRp57OfGrDFeUJ8/8QdfBKIV12FTtLn00g==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.6.tgz",
"integrity": "sha512-h5SxD/IzNhZYnrSZRsUZQIC+vD0GY8cUvq0iwsmkFKixRCKLLWqCXa/FIQ4S1R+sI+PGoojkHsdNrbZiM9Qpgw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -467,13 +467,13 @@
}
},
"node_modules/@vitest/runner": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.5.tgz",
"integrity": "sha512-2D+o7Pr82IEO46YPpoA/YU0neeyr6FTerQb5Ro7BUnBuv6NQtT/kmVnczngiMEBhzgqz2UZYl5gArejsyERDSQ==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.6.tgz",
"integrity": "sha512-nOPCmn2+yD0ZNmKdsXGv/UxMMWbMuKeD6GyYncNwdkYDxpQvrPSKYj2rWuDjC2Y4b6w6hjip5dBKFzEUuZe3vA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "4.1.5",
"@vitest/utils": "4.1.6",
"pathe": "^2.0.3"
},
"funding": {
@@ -481,14 +481,14 @@
}
},
"node_modules/@vitest/snapshot": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.5.tgz",
"integrity": "sha512-zypXEt4KH/XgKGPUz4eC2AvErYx0My5hfL8oDb1HzGFpEk1P62bxSohdyOmvz+d9UJwanI68MKwr2EquOaOgMQ==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.6.tgz",
"integrity": "sha512-YhsdE6xAVfTDmzjxL2ZDUvjj+ZsgyOKe+TdQzqkD72wIOmHka8NuGQ6NpTNZv9D2Z63fbwWKJPeVpEw4EQgYxw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.5",
"@vitest/utils": "4.1.5",
"@vitest/pretty-format": "4.1.6",
"@vitest/utils": "4.1.6",
"magic-string": "^0.30.21",
"pathe": "^2.0.3"
},
@@ -497,9 +497,9 @@
}
},
"node_modules/@vitest/spy": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.5.tgz",
"integrity": "sha512-2lNOsh6+R2Idnf1TCZqSwYlKN2E/iDlD8sgU59kYVl+OMDmvldO1VDk39smRfpUNwYpNRVn3w4YfuC7KfbBnkQ==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.6.tgz",
"integrity": "sha512-JFKxMx6udhwKh/Ldo270e17QX710vgunMkuPAvXjHSvC6oqLWAHhVhjg/I71q0u0CBSErIODV1Kjv0FQNSWjdg==",
"dev": true,
"license": "MIT",
"funding": {
@@ -507,13 +507,13 @@
}
},
"node_modules/@vitest/utils": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.5.tgz",
"integrity": "sha512-76wdkrmfXfqGjueGgnb45ITPyUi1ycZ4IHgC2bhPDUfWHklY/q3MdLOAB+TF1e6xfl8NxNY0ZYaPCFNWSsw3Ug==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.6.tgz",
"integrity": "sha512-FxIY+U81R3LGKCxaHHFRQ5+g6/iRgGLmeHWdp2Amj4ljQRrEIWHmZyDfDYBRZlpyqA7qKxtS9DD1dhk8RnRIVQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.5",
"@vitest/pretty-format": "4.1.6",
"convert-source-map": "^2.0.0",
"tinyrainbow": "^3.1.0"
},
@@ -959,9 +959,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.13",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.13.tgz",
"integrity": "sha512-qif0+jGGZoLWdHey3UFHHWP0H7Gbmsk8T5VEqyYFbWqPr1XqvLGBbk/sl8V5exGmcYJklJOhOQq1pV9IcsiFag==",
"version": "8.5.14",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
"dev": true,
"funding": [
{
@@ -988,14 +988,14 @@
}
},
"node_modules/rolldown": {
"version": "1.0.0-rc.17",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz",
"integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.1.tgz",
"integrity": "sha512-X0KQHljNnEkWNqqiz9zJrGunh1B0HgOxLXvnFpCOcadzcy5qohZ3tqMEUg00vncoRovXuK3ZqCT9KnnKzoInFQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@oxc-project/types": "=0.127.0",
"@rolldown/pluginutils": "1.0.0-rc.17"
"@oxc-project/types": "=0.130.0",
"@rolldown/pluginutils": "^1.0.0"
},
"bin": {
"rolldown": "bin/cli.mjs"
@@ -1004,21 +1004,21 @@
"node": "^20.19.0 || >=22.12.0"
},
"optionalDependencies": {
"@rolldown/binding-android-arm64": "1.0.0-rc.17",
"@rolldown/binding-darwin-arm64": "1.0.0-rc.17",
"@rolldown/binding-darwin-x64": "1.0.0-rc.17",
"@rolldown/binding-freebsd-x64": "1.0.0-rc.17",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17",
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17",
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17",
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.17",
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.17",
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.17",
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17",
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17"
"@rolldown/binding-android-arm64": "1.0.1",
"@rolldown/binding-darwin-arm64": "1.0.1",
"@rolldown/binding-darwin-x64": "1.0.1",
"@rolldown/binding-freebsd-x64": "1.0.1",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.1",
"@rolldown/binding-linux-arm64-gnu": "1.0.1",
"@rolldown/binding-linux-arm64-musl": "1.0.1",
"@rolldown/binding-linux-ppc64-gnu": "1.0.1",
"@rolldown/binding-linux-s390x-gnu": "1.0.1",
"@rolldown/binding-linux-x64-gnu": "1.0.1",
"@rolldown/binding-linux-x64-musl": "1.0.1",
"@rolldown/binding-openharmony-arm64": "1.0.1",
"@rolldown/binding-wasm32-wasi": "1.0.1",
"@rolldown/binding-win32-arm64-msvc": "1.0.1",
"@rolldown/binding-win32-x64-msvc": "1.0.1"
}
},
"node_modules/siginfo": {
@@ -1119,16 +1119,16 @@
}
},
"node_modules/vite": {
"version": "8.0.10",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz",
"integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==",
"version": "8.0.13",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.13.tgz",
"integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==",
"dev": true,
"license": "MIT",
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
"postcss": "^8.5.10",
"rolldown": "1.0.0-rc.17",
"postcss": "^8.5.14",
"rolldown": "1.0.1",
"tinyglobby": "^0.2.16"
},
"bin": {
@@ -1145,7 +1145,7 @@
},
"peerDependencies": {
"@types/node": "^20.19.0 || >=22.12.0",
"@vitejs/devtools": "^0.1.0",
"@vitejs/devtools": "^0.1.18",
"esbuild": "^0.27.0 || ^0.28.0",
"jiti": ">=1.21.0",
"less": "^4.0.0",
@@ -1197,19 +1197,19 @@
}
},
"node_modules/vitest": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.5.tgz",
"integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.6.tgz",
"integrity": "sha512-6lvjbS3p9b4CrdCmguzbh2/4uoXhGE2q71R4OX5sqF9R1bo9Xd6fGrMAfvp5wnCzlBnFVdCOp6onuTQVbo8iUQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/expect": "4.1.5",
"@vitest/mocker": "4.1.5",
"@vitest/pretty-format": "4.1.5",
"@vitest/runner": "4.1.5",
"@vitest/snapshot": "4.1.5",
"@vitest/spy": "4.1.5",
"@vitest/utils": "4.1.5",
"@vitest/expect": "4.1.6",
"@vitest/mocker": "4.1.6",
"@vitest/pretty-format": "4.1.6",
"@vitest/runner": "4.1.6",
"@vitest/snapshot": "4.1.6",
"@vitest/spy": "4.1.6",
"@vitest/utils": "4.1.6",
"es-module-lexer": "^2.0.0",
"expect-type": "^1.3.0",
"magic-string": "^0.30.21",
@@ -1237,12 +1237,12 @@
"@edge-runtime/vm": "*",
"@opentelemetry/api": "^1.9.0",
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
"@vitest/browser-playwright": "4.1.5",
"@vitest/browser-preview": "4.1.5",
"@vitest/browser-webdriverio": "4.1.5",
"@vitest/coverage-istanbul": "4.1.5",
"@vitest/coverage-v8": "4.1.5",
"@vitest/ui": "4.1.5",
"@vitest/browser-playwright": "4.1.6",
"@vitest/browser-preview": "4.1.6",
"@vitest/browser-webdriverio": "4.1.6",
"@vitest/coverage-istanbul": "4.1.6",
"@vitest/coverage-v8": "4.1.6",
"@vitest/ui": "4.1.6",
"happy-dom": "*",
"jsdom": "*",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
+29
View File
@@ -13,6 +13,20 @@ export interface ConnectedEvent {
export interface HistoryEvent {
type: "history";
/**
* Per-message dicts the frontend consumes directly. Common optional keys:
* - `role`: "user" | "assistant" | "tool"
* - `content`: string or list (image/document parts)
* - `tool_calls`: assistant turns list of `{id, name, arguments, verdict?, output_assessment?}`
* - `tool_call_id`: tool turns id of the originating call
* - `reminders`: metacognitive nudge bubbles (user/tool channels)
* - `advisories`: extracted `UserInterjection` payloads on tool turns
* - `reasoning`: concatenated reasoning text for assistant turns whose
* `provider_data` carried reasoning-bearing blocks (Anthropic
* `thinking`, OpenAI Responses `reasoning`, or synthetic
* `reasoning_text` from path-3 servers). Present only when the
* active model's `surface_persisted_reasoning` flag is true.
*/
messages: Array<Record<string, unknown>>;
}
@@ -38,6 +52,14 @@ export interface StreamEndEvent {
type: "stream_end";
}
/** One-shot replay of the in-progress turn's content + reasoning emitted
* by the events SSE handler when a fresh subscriber connects mid-stream. */
export interface InProgressSnapshotEvent {
type: "in_progress_snapshot";
content: string;
reasoning: string;
}
export interface StateChangeEvent {
type: "state_change";
state: "idle" | "thinking" | "running" | "attention" | "error";
@@ -162,6 +184,7 @@ export type ServerEvent =
| ContentEvent
| ReasoningEvent
| StreamEndEvent
| InProgressSnapshotEvent
| StateChangeEvent
| ToolInfoEvent
| ApproveRequestEvent
@@ -261,6 +284,12 @@ export function isStreamEndEvent(e: ServerEvent): e is StreamEndEvent {
return e.type === "stream_end";
}
export function isInProgressSnapshotEvent(
e: ServerEvent,
): e is InProgressSnapshotEvent {
return e.type === "in_progress_snapshot";
}
export function isStateChangeEvent(e: ServerEvent): e is StateChangeEvent {
return e.type === "state_change";
}
+45
View File
@@ -0,0 +1,45 @@
"""Shared session-test helpers.
Two reasoning-test modules (``test_session_replay_reasoning.py`` and
``test_session_synth_reasoning_block.py``) need the same minimal
``ChatSession`` factory + a ``SessionUIBase`` no-op subclass. Hoisting
keeps a future third caller from drifting on the defaults the third
existing ``_make_session`` (``test_model_registry.py``) deliberately
takes a different signature (registry / model_alias / reasoning_effort
+ ``_FakeUI``) and is NOT a candidate for sharing this helper.
Module is named with a leading underscore so pytest doesn't try to
collect it as a test file it's an importable utility, not a test.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession
from turnstone.core.session_ui_base import SessionUIBase
class NullUI(SessionUIBase):
"""Bare-bones UI satisfying the SessionUIBase contract for tests
that don't care about UI side effects."""
def __init__(self) -> None:
super().__init__()
def make_session(**kwargs: Any) -> ChatSession:
"""Build a ChatSession with minimal defaults; tests override
individual fields via kwargs."""
defaults: dict[str, Any] = {
"client": MagicMock(),
"model": "test-model",
"ui": NullUI(),
"instructions": None,
"temperature": 0.5,
"max_tokens": 4096,
"tool_timeout": 30,
}
defaults.update(kwargs)
return ChatSession(**defaults)
+220
View File
@@ -0,0 +1,220 @@
"""Tests for turnstone-admin DB configuration precedence.
Locks in the alignment with turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded default
The motivation is to keep DB secrets in config.toml (see
feedback_secrets_not_in_env) rather than forcing operators to export
TURNSTONE_DB_URL before every admin invocation.
"""
from __future__ import annotations
import argparse
from typing import TYPE_CHECKING
from unittest.mock import patch
import pytest
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
import turnstone.core.config as config_mod
from turnstone.admin import _get_storage
def _reset_cache() -> None:
config_mod._cache = None
config_mod._config_path = None
def _build_args(config_path: str | None) -> argparse.Namespace:
"""Build an args namespace the way admin.main() does.
Skips ``add_config_arg`` (which reads ``sys.argv``) the test
constructs the args programmatically instead.
"""
config_mod.set_config_path(config_path or "/nonexistent/turnstone-admin-test.toml")
parser = argparse.ArgumentParser()
config_mod.apply_config(parser, ["database"])
sub = parser.add_subparsers(dest="command")
sub.add_parser("list-users")
return parser.parse_args(["list-users"])
@pytest.fixture(autouse=True)
def _clear_db_env(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
"""Clean slate: no TURNSTONE_DB_* env vars unless a test sets them."""
for var in (
"TURNSTONE_DB_BACKEND",
"TURNSTONE_DB_URL",
"TURNSTONE_DB_PATH",
"TURNSTONE_DB_POOL_SIZE",
"TURNSTONE_DB_SSLMODE",
"TURNSTONE_DB_SSLROOTCERT",
"TURNSTONE_DB_SSLCERT",
"TURNSTONE_DB_SSLKEY",
"TURNSTONE_CONFIG",
):
monkeypatch.delenv(var, raising=False)
_reset_cache()
yield
_reset_cache()
def test_defaults_to_sqlite_when_neither_config_nor_env_set() -> None:
args = _build_args(None)
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("sqlite",)
assert init.call_args.kwargs["url"] == ""
assert init.call_args.kwargs["path"] == ""
assert init.call_args.kwargs["pool_size"] == 2
def test_config_toml_database_section_drives_init_storage(tmp_path: Path) -> None:
cfg = tmp_path / "config.toml"
cfg.write_text(
"[database]\n"
'backend = "postgresql"\n'
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
"pool_size = 5\n"
'sslmode = "verify-full"\n'
'sslrootcert = "/etc/ssl/ca.pem"\n'
'sslcert = "/etc/ssl/client.pem"\n'
'sslkey = "/etc/ssl/client.key"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["pool_size"] == 5
assert kw["sslmode"] == "verify-full"
assert kw["sslrootcert"] == "/etc/ssl/ca.pem"
assert kw["sslcert"] == "/etc/ssl/client.pem"
assert kw["sslkey"] == "/etc/ssl/client.key"
def test_env_used_as_fallback_when_config_absent(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "postgresql")
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "7")
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
args = _build_args(None)
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromenv:x@host/db"
assert kw["pool_size"] == 7
assert kw["sslmode"] == "require"
def test_config_toml_wins_over_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""config.toml beats env — operators should put secrets in TOML."""
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "sqlite")
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
cfg = tmp_path / "config.toml"
cfg.write_text(
"[database]\n"
'backend = "postgresql"\n'
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
'sslmode = "verify-full"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["sslmode"] == "verify-full"
def test_partial_config_falls_through_to_env_per_key(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""A key missing from [database] should fall back to its env var."""
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "9")
cfg = tmp_path / "config.toml"
cfg.write_text(
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromconfig:x@host/db"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["sslmode"] == "require"
assert kw["pool_size"] == 9
def test_empty_string_in_config_beats_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""`url = ""` in config.toml beats an env var.
Locks in the `is not None` guard a falsy-but-present TOML value
should NOT silently fall through to the env fallback.
"""
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nbackend = "sqlite"\nurl = ""\n')
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.kwargs["url"] == ""
def test_main_threads_config_toml_through_real_argv(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""End-to-end: ``turnstone-admin --config <toml> list-users`` honors TOML.
Covers the ``add_config_arg`` -> ``apply_config`` -> ``_get_storage``
chain that the programmatic ``_build_args`` helper skips.
"""
cfg = tmp_path / "config.toml"
cfg.write_text(
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromcli:x@host/db"\n'
)
monkeypatch.setattr("sys.argv", ["turnstone-admin", "--config", str(cfg), "list-users"])
fake_storage = patch("turnstone.core.storage.init_storage").start()
fake_storage.return_value.list_users.return_value = []
try:
from turnstone.admin import main
main()
finally:
patch.stopall()
assert fake_storage.call_args.args == ("postgresql",)
assert fake_storage.call_args.kwargs["url"] == "postgresql+psycopg://fromcli:x@host/db"
def test_get_storage_initializes_real_sqlite_backend(tmp_path: Path) -> None:
"""Drives the real ``init_storage`` boundary on a fresh sqlite file.
Mock-only tests would miss a kwarg-name typo (sslmode -> ssl_mode).
This test trips on any such drift because Alembic + the backend
actually run.
"""
from turnstone.core.storage import reset_storage
db_file = tmp_path / "admin.db"
cfg = tmp_path / "config.toml"
cfg.write_text(f'[database]\nbackend = "sqlite"\npath = "{db_file}"\n')
args = _build_args(str(cfg))
reset_storage()
try:
storage = _get_storage(args)
assert storage.list_users() == []
finally:
reset_storage()
+75 -4
View File
@@ -340,7 +340,7 @@ def test_phase8_no_unsafe_dom_write_in_settings_panel() -> None:
def test_phase8_settings_button_in_index_html() -> None:
"""The gear-icon entry-point for the settings panel must remain
"""The gear-icon entry-point for the settings menu must remain
in the appbar's actions span. The console proxy IIFE prepends a
node pill to ``header.firstChild`` (turnstone/console/server.py:
202); our button is appended inside ``<span class='appbar-actions'>``
@@ -351,9 +351,10 @@ def test_phase8_settings_button_in_index_html() -> None:
"index.html must keep the #settings-btn — onclick handlers "
"and the consent badge target it by id."
)
assert 'onclick="openSettingsPanel()"' in body, (
"settings-btn must wire onclick=openSettingsPanel() — losing "
"the binding leaves the panel unreachable."
assert 'onclick="toggleSettingsMenu(this)"' in body, (
"settings-btn must wire onclick=toggleSettingsMenu(this) — "
"the gear opens a dropdown with MCP connections + Logout; "
"losing the binding leaves the menu unreachable."
)
# The button must live inside <span class="appbar-actions"> so the
# console proxy's header.insertBefore(pill, header.firstChild)
@@ -366,6 +367,76 @@ def test_phase8_settings_button_in_index_html() -> None:
)
def test_settings_menu_handlers_defined() -> None:
"""The gear-icon dropdown exposes a toggle/open/close trio that the
inline ``onclick="toggleSettingsMenu(this)"`` in index.html depends
on, plus the menu items themselves must wire to existing entry
points (``openSettingsPanel`` for MCP connections, ``logout`` for
sign-out). Pin all four so a rename or deletion fails loudly here
instead of silently leaving the gear's menu broken or wired to a
stale function."""
body = _APP_JS.read_text(encoding="utf-8")
for name in [
"function toggleSettingsMenu",
"function openSettingsMenu",
"function closeSettingsMenu",
]:
assert name in body, f"Missing required handler: {name}"
# Bound to the settings-menu region so we don't accidentally match
# an unrelated openSettingsPanel/logout call elsewhere in the file.
start = body.index("function openSettingsMenu(")
end = body.index("function closeSettingsMenu(", start)
section = body[start:end]
assert "openSettingsPanel()" in section, (
"Settings menu's MCP-connections item must call openSettingsPanel() "
"— otherwise the existing settings overlay is unreachable from the "
"new dropdown."
)
assert "logout()" in section, (
"Settings menu's Logout item must call logout() — that's the "
"shared auth.js entry point that clears the cookie + session state."
)
def test_dashboard_overlay_is_region_not_dialog() -> None:
"""The dashboard overlay must be role='region' (not role='dialog' +
aria-modal='true'). The role downgrade is what allows ui-header to
stay interactive while the dashboard is open see the comment at
showDashboard() in app.js. A revert to role='dialog' + aria-modal
would re-trap focus and break the gear/theme buttons + the console
proxy's node-picker pill while the dashboard is open."""
body = _INDEX_HTML.read_text(encoding="utf-8")
idx = body.index('id="dashboard"')
# Bound to ~600 chars after the tag so we only check this element's
# attributes — same shape as test_phase8_settings_modal_in_index_html.
chunk = body[idx : idx + 600]
assert 'role="region"' in chunk, (
"dashboard must be role='region' — see showDashboard() comment."
)
assert "aria-modal" not in chunk, (
"dashboard must NOT be aria-modal — re-trapping focus breaks "
"the appbar's interactive controls (theme toggle, settings menu, "
"proxy node-picker pill) while the dashboard is open."
)
def test_close_settings_menu_resets_aria() -> None:
"""closeSettingsMenu must reset aria-expanded='false' AND remove
aria-controls from the gear trigger. Without the reset the gear
keeps reporting 'expanded' to assistive tech after the menu closes;
without the removal aria-controls points at a dead DOM id."""
body = _APP_JS.read_text(encoding="utf-8")
start = body.index("function closeSettingsMenu(")
# Bound to ~600 chars so we don't catch unrelated handlers.
section = body[start : start + 600]
assert 'setAttribute("aria-expanded", "false")' in section, (
"closeSettingsMenu must set aria-expanded='false' on the gear."
)
assert 'removeAttribute("aria-controls")' in section, (
"closeSettingsMenu must remove aria-controls from the gear."
)
def test_phase8_settings_modal_in_index_html() -> None:
"""Both the settings overlay and the revoke-confirmation overlay
must remain in the modal area. The Escape-key deferral list in
+33
View File
@@ -83,6 +83,39 @@ class TestIsPublicPath:
def test_shared_static_public(self):
assert is_public_path("/shared/base.css") is True
# Console proxy: a public proxied path must still be public, otherwise
# the login modal can never re-authenticate from inside a ``/node/{id}/``
# proxied page once the cookie expires.
def test_proxy_v1_login_public(self):
assert is_public_path("/node/node-a/v1/api/auth/login") is True
def test_proxy_no_v1_login_public(self):
assert is_public_path("/node/node-a/api/auth/login") is True
def test_proxy_v1_status_public(self):
assert is_public_path("/node/node-a/v1/api/auth/status") is True
def test_proxy_v1_setup_public(self):
assert is_public_path("/node/node-a/v1/api/auth/setup") is True
def test_proxy_v1_logout_public(self):
assert is_public_path("/node/node-a/v1/api/auth/logout") is True
def test_proxy_v1_oidc_authorize_public(self):
assert is_public_path("/node/node-a/v1/api/auth/oidc/authorize") is True
def test_proxy_v1_oidc_callback_public(self):
assert is_public_path("/node/node-a/v1/api/auth/oidc/callback") is True
def test_proxy_v1_workstreams_still_not_public(self):
"""Proxy prefix must not turn protected paths into public ones."""
assert is_public_path("/node/node-a/v1/api/workstreams") is False
def test_proxy_v1_refresh_still_requires_auth(self):
"""Refresh isn't in PUBLIC_PATHS — the caller must already have
a valid cookie. Proxy-prefix shouldn't change that."""
assert is_public_path("/node/node-a/v1/api/auth/refresh") is False
# ---------------------------------------------------------------------------
# TestRequiredRole
+123
View File
@@ -141,3 +141,126 @@ class TestRemindersWidening:
}
history = _build([msg])
assert history[0]["reminders"] == [{"type": "denial", "text": "ok"}]
class _StubRegistry:
"""Minimal model registry — only ``get_config`` is read by
``_build_history``."""
def __init__(self, surface_persisted_reasoning: bool = True) -> None:
self._cfg = SimpleNamespace(surface_persisted_reasoning=surface_persisted_reasoning)
def get_config(self, alias: str) -> Any:
return self._cfg
def _build_with_registry(
messages: list[dict[str, Any]],
surface_persisted_reasoning: bool = True,
) -> list[dict[str, Any]]:
session = SimpleNamespace(
messages=messages,
_ws_id="ws-test",
_registry=_StubRegistry(surface_persisted_reasoning=surface_persisted_reasoning),
_model_alias="claude-opus-4-7",
)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
return _build_history(session)
class TestReasoningSurfacing:
"""Phase 1 — surface stored Anthropic thinking blocks on the
history payload so refresh-the-page rehydrates the reasoning bubble.
Drives through the real ``AnthropicProvider`` extractor (no mock-of-
extractor) only the model registry is stubbed.
"""
def test_reasoning_surfaces_for_anthropic_thinking_msg(self) -> None:
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "s"},
{"type": "text", "text": "Final answer."},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert len(history) == 1
assert history[0]["reasoning"] == "let me think"
def test_reasoning_empty_when_persist_flag_false(self) -> None:
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "hidden", "signature": "s"},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=False)
assert "reasoning" not in history[0]
def test_provider_content_never_in_wire_entry(self) -> None:
# The build path does not copy ``_provider_content`` into the
# entry dict regardless of flag — wire payload stays tight.
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "x", "signature": "s"},
],
}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert "_provider_content" not in history[0]
def test_no_reasoning_field_when_provider_content_missing(self) -> None:
msg = {"role": "assistant", "content": "plain answer"}
history = _build_with_registry([msg], surface_persisted_reasoning=True)
assert "reasoning" not in history[0]
def test_no_reasoning_field_for_non_assistant_messages(self) -> None:
# Defensive — user/tool messages with a stray _provider_content
# do not get the reasoning field stamped.
msgs: list[dict[str, Any]] = [
{"role": "user", "content": "hi"},
{
"role": "tool",
"tool_call_id": "c1",
"content": "out",
"_provider_content": [{"type": "thinking", "thinking": "leak", "signature": "s"}],
},
]
history = _build_with_registry(msgs, surface_persisted_reasoning=True)
assert "reasoning" not in history[0]
assert "reasoning" not in history[1]
def test_default_true_when_registry_lookup_raises(self) -> None:
# Conservative default — Phase 1 spec mandates rehydration on
# refresh. A registry/alias mismatch must not silently kill the
# bubble.
class BrokenRegistry:
def get_config(self, alias: str) -> Any:
raise KeyError(alias)
session = SimpleNamespace(
messages=[
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "still works", "signature": "s"}
],
}
],
_ws_id="ws-test",
_registry=BrokenRegistry(),
_model_alias="missing-alias",
)
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
history = _build_history(session)
assert history[0]["reasoning"] == "still works"
+112
View File
@@ -19,6 +19,12 @@ class NullUI:
self.infos = []
self.stream_ends = 0
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
@@ -820,3 +826,109 @@ class TestForceCancelThreaded:
assert "idle" in ui.states
assistant_msgs = [m for m in session.messages if m["role"] == "assistant"]
assert any("Fresh response" in m.get("content", "") for m in assistant_msgs)
class TestSynthesizeCancelledResults:
"""Regression coverage for ``_synthesize_cancelled_results`` — must
fire ``on_tool_result`` for each synthesized cancellation so live
SSE listeners (e.g. coord's ``--running`` indicator added by
tool_info) can complete the in-DOM tool batch. Without this, the
coord JS would spin the running indicator forever on cancelled
batches because ``state_change`` doesn't strip ``--running`` from
individual batches."""
def _ui_with_tool_result_tracking(self):
class _TrackingUI(NullUI):
def __init__(self) -> None:
super().__init__()
self.tool_results: list[tuple[str, str, str, bool]] = []
def on_tool_result(self, call_id, name, output, **kwargs):
self.tool_results.append(
(call_id, name, output, bool(kwargs.get("is_error", False))),
)
return _TrackingUI()
def test_synthesizes_tool_result_for_unanswered_calls(self, tmp_db):
ui = self._ui_with_tool_result_tracking()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"content": "calling tools",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
session._synthesize_cancelled_results("Cancelled by user.")
# Both unanswered calls fired ``on_tool_result``.
assert len(ui.tool_results) == 2
ids = {tr[0] for tr in ui.tool_results}
assert ids == {"call_a", "call_b"}
# All emitted as errors so the live UI renders them as
# ``coord-tool-row-result--error``.
assert all(tr[3] is True for tr in ui.tool_results)
# Reason text propagates as the synthetic tool output.
assert all(tr[2] == "Cancelled by user." for tr in ui.tool_results)
# And the message list has the synthesized tool entries
# (preserves the prior contract).
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
assert len(tool_msgs) == 2
def test_skips_calls_already_answered(self, tmp_db):
ui = self._ui_with_tool_result_tracking()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
{"id": "call_b", "function": {"name": "compute", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
# call_a already answered.
session.messages.append(
{"role": "tool", "tool_call_id": "call_a", "content": "result"},
)
session._msg_tokens.append(1)
session._synthesize_cancelled_results("Cancelled by user.")
# Only call_b synthesized.
assert len(ui.tool_results) == 1
assert ui.tool_results[0][0] == "call_b"
def test_ui_emit_failure_does_not_break_synthesis(self, tmp_db):
"""The UI hook is wrapped in try/except — a hook failure
during cancel must NOT compound the problem. Synthesis still
appends to messages + storage."""
class _ExplodingUI(NullUI):
def on_tool_result(self, call_id, name, output, **kwargs):
raise RuntimeError("ui hook blew up")
ui = _ExplodingUI()
session = _make_session(ui=ui)
session.messages.append(
{
"role": "assistant",
"tool_calls": [
{"id": "call_a", "function": {"name": "search", "arguments": "{}"}},
],
},
)
session._msg_tokens.append(1)
# Must not raise.
session._synthesize_cancelled_results("Cancelled by user.")
tool_msgs = [m for m in session.messages if m.get("role") == "tool"]
assert len(tool_msgs) == 1
@@ -0,0 +1,66 @@
"""ChatSession interactivity flag tests (Phase 9).
Validates that ``ChatSession._is_interactive_for_consent`` is computed
correctly from ``client_type`` on construction. This is the front of
the Phase 9 plumb-through: the flag flows from here to
``_dispatch_pool_sync`` to the structured-error pending-consent
write path.
"""
from __future__ import annotations
from tests._session_helpers import make_session
from turnstone.prompts import INTERACTIVE_CONSENT_CLIENT_TYPES, ClientType
def test_web_is_interactive() -> None:
s = make_session(client_type=ClientType.WEB)
assert s._is_interactive_for_consent is True
def test_cli_is_interactive() -> None:
s = make_session(client_type=ClientType.CLI)
assert s._is_interactive_for_consent is True
def test_chat_is_not_interactive() -> None:
# Discord / Slack adapters cannot drive a browser redirect from
# inside the channel — consent prompts must be deferred to the
# dashboard badge.
s = make_session(client_type=ClientType.CHAT)
assert s._is_interactive_for_consent is False
def test_scheduled_is_not_interactive() -> None:
# The scheduler runs autonomously; the user isn't online to
# complete the OAuth redirect.
s = make_session(client_type=ClientType.SCHEDULED)
assert s._is_interactive_for_consent is False
def test_interactive_set_matches_module_constant() -> None:
# Pin the module-level frozenset against the flag computation —
# a future reorganisation that drifts the set vs the per-session
# logic would silently break the gating.
for ct in ClientType:
s = make_session(client_type=ct)
assert s._is_interactive_for_consent == (ct in INTERACTIVE_CONSENT_CLIENT_TYPES), ct
def test_default_client_type_is_cli_interactive() -> None:
# Defaults preserved — make_session uses ChatSession's default
# which is CLI. Sanity check that the default user experience
# stays interactive-for-consent.
s = make_session()
assert s._client_type == ClientType.CLI
assert s._is_interactive_for_consent is True
def test_scheduled_env_file_exists() -> None:
"""The SCHEDULED env module must exist; otherwise
``compose_system_message`` for a scheduled session would 500."""
from turnstone.prompts import _load
text = _load("env/scheduled.md")
assert "Output Environment" in text
assert "consent" in text.lower()
+215
View File
@@ -0,0 +1,215 @@
"""Unit tests for :class:`turnstone.core.child_event_bus.ChildEventBus`.
The bus is the in-process wakeup primitive for ``wait_for_workstream``
(see :mod:`turnstone.console.coordinator_client`). It's a small dict
of ws_id set[threading.Event] under a lock focused tests for
register/notify symmetry, no-subscriber notify, multi-waiter fan-out,
multi-child waiter, and concurrent register/notify (smoke). End-to-end
integration with the dispatch sink lives in
``test_coordinator_adapter.py`` and ``test_coordinator_client.py``.
"""
from __future__ import annotations
import threading
import time
import pytest
from turnstone.core.child_event_bus import ChildEventBus
def test_register_returns_event_that_starts_unset() -> None:
"""A waiter must not see leftover state from before it registered —
a fresh wait should always block until the first notify."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
assert isinstance(event, threading.Event)
assert not event.is_set()
def test_notify_wakes_waiter_on_matching_ws_id() -> None:
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify("ws-1")
assert event.is_set()
def test_notify_does_not_wake_waiter_on_unrelated_ws_id() -> None:
"""Different ws_ids must keep independent waiter sets — a notify on
a stranger ws can't wake the wait or the bus stops being keyed."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify("ws-other")
assert not event.is_set()
def test_notify_with_no_subscribers_is_noop() -> None:
"""The dispatch sink calls notify on every translated event; the
steady state has no wait tool active. Must not raise."""
bus = ChildEventBus()
bus.notify("ws-nobody-cares") # no exception
def test_multi_waiter_each_gets_independent_event() -> None:
"""Two waits on the same ws_id must wake independently — clearing
one Event must not silence the other."""
bus = ChildEventBus()
e1 = bus.register_waiter(["ws-1"])
e2 = bus.register_waiter(["ws-1"])
assert e1 is not e2
bus.notify("ws-1")
assert e1.is_set()
assert e2.is_set()
def test_multi_child_waiter_fires_on_any_listed_ws_id() -> None:
"""A wait on [A, B, C] returns a single Event registered against
all three. Notify on ANY of A/B/C must wake the wait the
caller's snapshot re-read disambiguates which one changed."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-a", "ws-b", "ws-c"])
bus.notify("ws-b")
assert event.is_set()
def test_unregister_removes_event_from_all_listed_ws_ids() -> None:
"""After unregister, notify on any of the previously-watched ws_ids
must NOT wake the Event leaks would mean every future notify on
that ws_id wakes a long-dead wait."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-a", "ws-b"])
bus.unregister_waiter(["ws-a", "ws-b"], event)
bus.notify("ws-a")
bus.notify("ws-b")
assert not event.is_set()
def test_unregister_is_idempotent() -> None:
"""A double-unregister must silently no-op — finally blocks may
run twice in odd shutdown paths, the bus must not raise."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.unregister_waiter(["ws-1"], event)
bus.unregister_waiter(["ws-1"], event) # no exception
def test_unregister_pops_empty_buckets() -> None:
"""Empty per-ws_id buckets must be popped so a long-lived bus
doesn't accumulate dead keys after many waits have churned through.
Reaches into the private state the property is structural, not
behavioral, so the assertion is also."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
assert "ws-1" in bus._waiters
bus.unregister_waiter(["ws-1"], event)
assert "ws-1" not in bus._waiters
def test_unregister_keeps_bucket_with_remaining_waiters() -> None:
"""Removing one waiter from a multi-waiter bucket must not drop
the others popping the bucket would silently disable notifies
for every concurrent wait on the same ws_id."""
bus = ChildEventBus()
e1 = bus.register_waiter(["ws-1"])
e2 = bus.register_waiter(["ws-1"])
bus.unregister_waiter(["ws-1"], e1)
bus.notify("ws-1")
assert not e1.is_set()
assert e2.is_set()
def test_empty_and_falsy_ws_ids_are_skipped_on_register() -> None:
"""Defensive: ``wait_for_workstream`` cleans its inputs but the bus
is reachable from other callers in future use; falsy ids should be
silently dropped, not registered against an empty-string key."""
bus = ChildEventBus()
event = bus.register_waiter(["", "ws-1", ""])
# Only the real ws_id should bucket the waiter.
assert list(bus._waiters.keys()) == ["ws-1"]
bus.notify("") # no crash, no spurious wake
assert not event.is_set()
bus.notify("ws-1")
assert event.is_set()
def test_notify_wakes_waiter_blocking_on_event_wait() -> None:
"""End-to-end wake-up latency: a wait blocked on ``Event.wait``
must return promptly after a notify on a watched ws_id. This is
the property that retires the 0.5s polling cadence."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
woken_at = [0.0]
def _waiter() -> None:
event.wait(timeout=2.0)
woken_at[0] = time.monotonic()
t = threading.Thread(target=_waiter, daemon=True)
t.start()
# Give the waiter a beat to enter Event.wait, then notify.
time.sleep(0.05)
notified_at = time.monotonic()
bus.notify("ws-1")
t.join(timeout=1.0)
assert not t.is_alive(), "waiter did not wake within 1s of notify"
# Latency budget is generous; the contract is "well under the legacy
# 0.5s poll cadence", not microsecond timing.
assert woken_at[0] - notified_at < 0.2
def test_clear_before_check_race_does_not_lose_wake() -> None:
"""The wait-loop pattern is ``clear(); snapshot(); ...; wait()``.
A notify between clear and wait must leave the Event set, so the
next wait returns immediately and the loop re-snapshots. Same
standard subscribe/check race the wait loop guards against."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
# Simulate wait-loop ordering: clear, then notify "between" clear
# and the next wait.
event.clear()
bus.notify("ws-1")
# The next wait must return True immediately (set is sticky until
# the next clear).
assert event.wait(timeout=0.1) is True
def test_concurrent_register_and_notify_is_safe() -> None:
"""Smoke test: many threads registering / notifying / unregistering
in parallel must not raise or deadlock. Doesn't assert specific
interleavings only structural safety of the lock discipline."""
bus = ChildEventBus()
stop = threading.Event()
errors: list[BaseException] = []
def _worker(ws_id: str) -> None:
try:
for _ in range(200):
if stop.is_set():
return
ev = bus.register_waiter([ws_id])
bus.notify(ws_id)
bus.unregister_waiter([ws_id], ev)
except BaseException as e: # noqa: BLE001
errors.append(e)
threads = [threading.Thread(target=_worker, args=(f"ws-{i}",), daemon=True) for i in range(8)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5.0)
stop.set()
assert not errors, f"worker threads raised: {errors!r}"
# All buckets should have been popped (every register paired with
# unregister).
assert bus._waiters == {}
@pytest.mark.parametrize("ws_id", ["", None])
def test_notify_silently_ignores_falsy_ws_id(ws_id: object) -> None:
"""Defensive: the dispatch sink already guards against empty
ws_ids, but a falsy slip-through must not raise."""
bus = ChildEventBus()
event = bus.register_waiter(["ws-1"])
bus.notify(ws_id) # type: ignore[arg-type]
assert not event.is_set()
+35
View File
@@ -50,6 +50,41 @@ def test_load_config_invalid_toml(tmp_path):
assert load_config() == {}
def test_load_config_warns_when_world_readable(tmp_path, caplog):
"""Secrets in config.toml — warn if anyone but the owner can read it."""
import logging
import os
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
os.chmod(cfg, 0o644)
set_config_path(str(cfg))
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
load_config()
messages = [r.getMessage() for r in caplog.records]
assert any("group/world-readable" in m for m in messages)
def test_load_config_quiet_when_mode_0600(tmp_path, caplog):
import logging
import os
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
os.chmod(cfg, 0o600)
set_config_path(str(cfg))
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
load_config()
messages = [r.getMessage() for r in caplog.records]
assert not any("group/world-readable" in m for m in messages)
def test_load_config_caches(tmp_path):
_reset_cache()
cfg = tmp_path / "config.toml"
+210
View File
@@ -3,11 +3,13 @@
import asyncio
import json
import queue
from typing import Any
from unittest.mock import MagicMock
import pytest
from turnstone.console.collector import ClusterCollector, NodeSnapshot
from turnstone.console.server import _PROXY_AUTH_LOCAL_HANDLERS
# Shared test auth — JWT-based
_TEST_JWT_SECRET = "test-jwt-secret-minimum-32-chars!"
@@ -149,6 +151,78 @@ class TestCollectorDiscovery:
assert c._nodes["node-a"].started == 1234567890.0
class TestCollectorNotifyWireIn:
"""NotifyDispatcher-driven discovery — reactive node visibility."""
def test_start_subscribes_to_services_channel(self):
# Stub dispatcher records subscriptions without spawning threads.
class _StubDispatcher:
def __init__(self):
self.subscriptions: list[tuple[str, Any]] = []
def subscribe(self, channel, handler):
self.subscriptions.append((channel, handler))
return lambda: None
stub = _StubDispatcher()
storage = MockStorage()
c = ClusterCollector(
storage=storage,
discovery_interval=999,
notify_dispatcher=stub,
)
try:
c.start()
assert len(stub.subscriptions) == 1
channel, handler = stub.subscriptions[0]
assert channel == "services"
assert handler == c._on_services_notify
finally:
c.stop()
def test_no_dispatcher_means_no_subscribe(self):
# Collector without a dispatcher (single-node / SQLite dev) just
# falls back to the 60 s discovery-loop polling — no error.
c = _make_collector(MockStorage())
try:
c.start()
assert c._notify_unsubscribe is None
finally:
c.stop()
def test_on_notify_runs_discovery(self):
# Construct a synthetic Notify and invoke the handler directly —
# asserts the wire-in delegates back to ``_discover_nodes``.
from turnstone.core.storage._notify import Notify
storage = MockStorage()
c = _make_collector(storage)
c._running = True # bypass start() so we don't spawn threads
q: queue.Queue[dict[str, Any]] = queue.Queue()
c.register_listener(q)
storage.services = [
{"service_id": "node-z", "url": "http://z:8080", "metadata": "{}"},
]
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
event = q.get_nowait()
assert event["type"] == "node_joined"
assert event["node_id"] == "node-z"
def test_on_notify_when_not_running_is_noop(self):
# If a stray notify arrives after stop, the handler doesn't run
# discovery on a half-torn-down collector.
from turnstone.core.storage._notify import Notify
storage = MockStorage()
storage.services = [{"service_id": "node-y", "url": "http://y:8080", "metadata": "{}"}]
c = _make_collector(storage)
# _running stays False (never called start()).
c._on_services_notify(Notify(channel="services", payload="{}", pid=0))
assert c.get_overview()["nodes"] == 0
class TestCollectorSnapshot:
"""Applying node_snapshot SSE events."""
@@ -1489,6 +1563,142 @@ class TestConsoleProxy:
assert sse_mock.await_count == 1
assert sse_mock.await_args.kwargs.get("use_service_auth") is False
# -------------------------------------------------------------------
# Proxied auth endpoints — handled locally by the console, not
# forwarded to the upstream node. Cases derive directly from
# ``_PROXY_AUTH_LOCAL_HANDLERS`` so a new dispatch entry can't be
# added without a matching test (or vice versa). See proxy_api's
# docstring for the JWT-audience reasoning.
# -------------------------------------------------------------------
@pytest.mark.parametrize(
("method", "path", "handler_name"),
[
(method, path, handler_name)
for (method, path), handler_name in sorted(_PROXY_AUTH_LOCAL_HANDLERS.items())
],
)
def test_proxy_auth_endpoint_dispatches_to_local_handler(
self, client, method, path, handler_name
):
"""Every entry in ``_PROXY_AUTH_LOCAL_HANDLERS`` must route to its
local console handler and never reach the upstream proxy. The
lockout class of bug this dispatch was added to fix is exactly
what a regression here would reintroduce silently covering all
eight branches keeps each path tied to its handler."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
with (
patch(
f"turnstone.console.server.{handler_name}",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "ok"}),
) as local_mock,
patch(
"turnstone.console.server._proxy_post",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as post_mock,
patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as get_mock,
):
resp = client.request(method, f"/node/node-a/v1/api/{path}")
assert resp.status_code == 200
assert local_mock.await_count == 1
assert post_mock.await_count == 0
assert get_mock.await_count == 0
def test_proxy_auth_login_works_without_cookie(self, mock_collector):
"""Without this fix the AuthMiddleware 401s before any handler
runs the user is locked out of the proxied UI once the cookie
expires. Test bypasses _TEST_AUTH_HEADERS to reproduce."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
from starlette.testclient import TestClient
from turnstone.console.server import _load_static, create_app
_load_static()
app = create_app(collector=mock_collector, jwt_secret=_TEST_JWT_SECRET)
unauth_client = TestClient(app, raise_server_exceptions=False)
try:
with patch(
"turnstone.console.server.auth_login",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "ok"}),
) as local_mock:
resp = unauth_client.post(
"/node/node-a/v1/api/auth/login",
json={"username": "x", "password": "y"},
)
# AuthMiddleware must classify the proxied login path as
# public (is_public_path change) AND proxy_api must
# dispatch to the local handler (proxy_api change).
assert resp.status_code == 200, (
f"login locked out: got {resp.status_code}, body={resp.text}"
)
assert local_mock.await_count == 1
finally:
unauth_client.close()
def test_proxy_auth_wrong_method_returns_405_not_forwarded(self, client):
"""A non-canonical method on an auth path (e.g. PUT on auth/login)
must short-circuit with 405 instead of falling through to the
upstream proxy falling through would forward the request
authenticated as the console's service token (``_proxy_auth_headers``
fallback)."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
with (
patch(
"turnstone.console.server._proxy_post",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as post_mock,
patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"status": "should-not-be-called"}),
) as get_mock,
):
# PUT on a POST-only auth path → 405
put_resp = client.put("/node/node-a/v1/api/auth/login")
assert put_resp.status_code == 405
# POST on a GET-only auth path → 405
post_resp = client.post("/node/node-a/v1/api/auth/status")
assert post_resp.status_code == 405
assert post_mock.await_count == 0
assert get_mock.await_count == 0
def test_proxy_non_auth_endpoint_still_forwarded(self, client, mock_collector):
"""Sanity: only auth/* paths intercept. Other API paths still
forward to the upstream node."""
from unittest.mock import AsyncMock, patch
from starlette.responses import JSONResponse
mock_collector.get_node_detail.return_value = {
"node_id": "node-a",
"server_url": "http://a:8080",
"reachable": True,
}
with patch(
"turnstone.console.server._proxy_get",
new_callable=AsyncMock,
return_value=JSONResponse({"ok": True}),
) as proxy_mock:
resp = client.get("/node/node-a/v1/api/workstreams")
assert resp.status_code == 200
assert proxy_mock.await_count == 1
# ---------------------------------------------------------------------------
# Proxy URL rewriting unit tests (no HTTP needed)
+345
View File
@@ -0,0 +1,345 @@
"""``GET /v1/api/models`` resolution-chain coverage.
The console handler resolves four defaults from settings + the enabled
model list:
* ``default_alias`` ``model.default_alias``
* ``channel_default_alias`` ``channels.default_model_alias``
* ``coordinator_default_alias`` ``coordinator.model_alias``, falling
back to ``default_alias`` when empty *or* pointing at a disabled /
removed alias (mirrors :mod:`turnstone.console.session_factory`).
* ``judge_default_alias`` ``judge.model``, falling back to the
resolved coordinator alias when empty *or* pointing at a value that
isn't an enabled alias. ``judge.model`` is alias-only — same
contract as the other model roles and
:class:`turnstone.core.judge.IntentJudge` silently inherits the
session model when an unknown value is configured, so the API
surfaces the resolved coordinator alias rather than echoing the
misconfigured string.
These tests pin each branch so the home composer's resolved-alias
placeholder stays correct as the precedence rules evolve.
"""
from __future__ import annotations
from typing import Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route
from starlette.testclient import TestClient
from tests._coord_test_helpers import _AuthMiddleware, _FakeConfigStore
from turnstone.console.server import list_available_models
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "available_models.db"))
def _seed_model(
storage: SQLiteBackend,
*,
definition_id: str,
alias: str,
model: str = "model-x",
enabled: bool = True,
) -> None:
storage.create_model_definition(
definition_id=definition_id,
alias=alias,
model=model,
provider="openai-compatible",
base_url="http://localhost:8000/v1",
api_key="sk-test",
context_window=8192,
capabilities="{}",
enabled=enabled,
created_by="admin",
)
class _StubRegistry:
"""Mimics the surface ``resolve_coordinator_alias`` reads from
``coord_registry``: ``.default`` and ``.has_alias()``.
Production wires this through ``ModelRegistry``, which in turn
pulls aliases from both DB rows and config.toml. The fixture
mirrors the storage's enabled-row set so ``has_alias()`` agrees
with what the placeholder's enabled-row filter would accept —
without that alignment the helper rejects every tier-2 candidate
and the placeholder goes blank in cases that production handles
fine."""
def __init__(self, *, default: str, known: set[str]) -> None:
self.default = default
self._known = known
def has_alias(self, alias: str) -> bool:
return alias in self._known
def _make_client(
storage: SQLiteBackend,
*,
settings: dict[str, str] | None = None,
registry_default: str = "",
config_store: bool = True,
) -> TestClient:
app = Starlette(
routes=[Route("/v1/api/models", list_available_models)],
middleware=[Middleware(_AuthMiddleware)],
)
app.state.auth_storage = storage
if config_store:
app.state.config_store = _FakeConfigStore(dict(settings or {}))
# ``coord_registry`` is always set in production after lifespan
# startup; mirror that here. ``has_alias`` answers from the same
# enabled-rows set the handler filters against.
enabled = {r["alias"] for r in storage.list_model_definitions(enabled_only=True)}
app.state.coord_registry = _StubRegistry(default=registry_default, known=enabled)
client = TestClient(app)
client.headers.update({"X-Test-User": "admin", "X-Test-Perms": ""})
return client
def _get_models(client: TestClient) -> dict[str, Any]:
resp = client.get("/v1/api/models")
assert resp.status_code == 200, resp.text
return resp.json()
# ---------------------------------------------------------------------------
# Coordinator resolution
# ---------------------------------------------------------------------------
def test_no_settings_leaves_all_defaults_blank(storage: SQLiteBackend) -> None:
"""No model.default_alias, no per-role overrides → every default
field is empty and ``models`` is an empty list."""
body = _get_models(_make_client(storage))
assert body == {
"models": [],
"default_alias": "",
"channel_default_alias": "",
"coordinator_default_alias": "",
"judge_default_alias": "",
}
def test_coordinator_inherits_default_alias_when_unset(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, settings={"model.default_alias": "primary"}))
assert body["default_alias"] == "primary"
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_explicit_enabled_alias_passes_through(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "fast",
},
)
)
assert body["coordinator_default_alias"] == "fast"
def test_coordinator_set_to_disabled_alias_falls_back_to_default(
storage: SQLiteBackend,
) -> None:
"""Operator disabled the alias the coordinator was pinned to —
fall back to the registry default rather than advertising a model
that workstream creation would refuse to use."""
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="legacy", enabled=False)
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "legacy",
},
)
)
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_set_to_unknown_alias_falls_back_to_default(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "ghost",
},
)
)
assert body["coordinator_default_alias"] == "primary"
def test_coordinator_falls_back_to_registry_default_when_config_store_empty(
storage: SQLiteBackend,
) -> None:
"""Match ``console/session_factory.py:109-110``: when both
``coordinator.model_alias`` and ``model.default_alias`` are unset, new
coordinator sessions run on ``registry.default`` (loaded from
config.toml ``[model].default``). The placeholder must report the
same alias rather than going blank otherwise the home composer
advertises "Default model" while sessions actually launch on a
concrete alias."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, registry_default="primary"))
assert body["default_alias"] == ""
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
def test_coordinator_skips_registry_default_when_alias_disabled(
storage: SQLiteBackend,
) -> None:
"""Registry default points at an alias that's been disabled in the DB
the placeholder stays blank rather than advertising a model that
workstream creation would refuse to use."""
_seed_model(storage, definition_id="m1", alias="legacy", enabled=False)
body = _get_models(_make_client(storage, registry_default="legacy"))
assert body["coordinator_default_alias"] == ""
def test_coordinator_falls_back_to_registry_default_when_config_store_missing(
storage: SQLiteBackend,
) -> None:
"""Edge case from PR #500 review: lifespan can leave
``app.state.config_store`` as None (e.g. a startup exception) while
``coord_registry`` still binds successfully. The placeholder must
still advertise ``registry.default`` (filtered against enabled rows)
rather than going blank otherwise the home composer is uselessly
empty in a degraded-but-recoverable state."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage, registry_default="primary", config_store=False))
assert body["default_alias"] == ""
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
# ---------------------------------------------------------------------------
# Judge resolution
# ---------------------------------------------------------------------------
def test_judge_empty_inherits_resolved_coordinator_alias(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"coordinator.model_alias": "fast",
},
)
)
assert body["coordinator_default_alias"] == "fast"
assert body["judge_default_alias"] == "fast"
def test_judge_explicit_enabled_alias_passes_through(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="judge-fast")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "judge-fast",
},
)
)
assert body["judge_default_alias"] == "judge-fast"
def test_judge_set_to_unknown_value_inherits_coordinator(
storage: SQLiteBackend,
) -> None:
"""``judge.model`` is alias-only — same contract as the other model
roles. An unknown value silently inherits the session model in
:class:`IntentJudge`, so the API surfaces the resolved coordinator
alias rather than echoing the misconfigured string."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "anthropic/claude-haiku-4-5", # raw, not an alias
},
)
)
assert body["coordinator_default_alias"] == "primary"
assert body["judge_default_alias"] == "primary"
def test_judge_set_to_disabled_alias_inherits_coordinator(
storage: SQLiteBackend,
) -> None:
"""Disabled-alias case is handled identically to the unknown-value
case both trip the alias-not-resolved path."""
_seed_model(storage, definition_id="m1", alias="primary")
_seed_model(storage, definition_id="m2", alias="judge-old", enabled=False)
body = _get_models(
_make_client(
storage,
settings={
"model.default_alias": "primary",
"judge.model": "judge-old",
},
)
)
assert body["judge_default_alias"] == "primary"
# ---------------------------------------------------------------------------
# Pre-existing fields stay correct under the new resolution code
# ---------------------------------------------------------------------------
def test_channel_default_alias_blanked_when_disabled(
storage: SQLiteBackend,
) -> None:
_seed_model(storage, definition_id="m1", alias="primary", enabled=False)
body = _get_models(
_make_client(
storage,
settings={"channels.default_model_alias": "primary"},
)
)
assert body["channel_default_alias"] == ""
def test_models_payload_strips_secret_fields(storage: SQLiteBackend) -> None:
"""Regression guard: only alias/model/provider land in the response,
never api_key / base_url / context_window / capabilities."""
_seed_model(storage, definition_id="m1", alias="primary")
body = _get_models(_make_client(storage))
assert body["models"] == [
{"alias": "primary", "model": "model-x", "provider": "openai-compatible"}
]
+229 -29
View File
@@ -5,11 +5,15 @@ lifting is in ``SessionManager.close_idle`` (covered in
``test_session_manager.py``) and ``bulk_close_stale_orphans`` (covered
in ``test_storage_sqlite.py``). These tests verify the glue:
- the helper runs an initial sweep BEFORE its first sleep (cold-start
- the helper runs an initial sweep BEFORE its first wait (cold-start
cleanup without blocking the lifespan),
- the helper swallows exceptions so a transient DB blip can't kill the
daemon thread,
- the helper exits cleanly when ``stop_event`` is set.
- the helper exits cleanly when ``stop_event`` is set,
- the helper subscribes to ``mgr.subscribe_to_state`` and a state-change
event wakes the next sweep early (event-driven, not polling),
- the helper unsubscribes when the thread exits so the subscriber
doesn't leak past one cleanup-thread lifetime.
The ``stop_event`` parameter is exclusively for tests production
callers pass ``None`` and the daemon runs for process lifetime.
@@ -17,28 +21,36 @@ callers pass ``None`` and the daemon runs for process lifetime.
from __future__ import annotations
import contextlib
import threading
from unittest.mock import patch
import time
from typing import TYPE_CHECKING
from turnstone.console.server import _coord_idle_cleanup_thread
if TYPE_CHECKING:
from collections.abc import Callable
class _StubMgr:
"""Minimal SessionManager substitute exposing only what the cleanup
thread touches: ``close_idle``, ``subscribe_to_state``,
``unsubscribe_from_state``. Records call ordering for assertions
and lets the test fire state-change events manually via
:meth:`fire_state_change`.
"""
def __init__(
self, *, stop_event: threading.Event, expected_calls: int, raise_after: int = -1
) -> None:
self.calls: list[float] = []
self.sleep_calls_at_each_close: list[int] = []
self._stop_event = stop_event
self._expected = expected_calls
self._raise_after = raise_after
self._sleep_count = 0
self._subscribers: list[Callable[[str, object], None]] = []
self._sub_lock = threading.Lock()
def close_idle(self, timeout_sec: float) -> list[str]:
# Snapshot how many sleeps preceded this close — lets the
# "initial sweep" test verify the first close_idle ran with
# zero preceding sleeps.
self.sleep_calls_at_each_close.append(self._sleep_count)
self.calls.append(timeout_sec)
try:
if 0 <= self._raise_after < len(self.calls):
@@ -50,39 +62,78 @@ class _StubMgr:
self._stop_event.set()
return []
def record_sleep(self, _seconds: float) -> None:
self._sleep_count += 1
def subscribe_to_state(self, callback: Callable[[str, object], None]) -> None:
with self._sub_lock:
self._subscribers.append(callback)
def unsubscribe_from_state(self, callback: Callable[[str, object], None]) -> None:
with self._sub_lock, contextlib.suppress(ValueError):
self._subscribers.remove(callback)
@property
def subscribers_count(self) -> int:
with self._sub_lock:
return len(self._subscribers)
def fire_state_change(self, ws_id: str = "ws-x", state: object = "idle") -> None:
with self._sub_lock:
snapshot = list(self._subscribers)
for cb in snapshot:
cb(ws_id, state)
def _run_until_done(mgr: _StubMgr, stop_event: threading.Event, timeout_sec: float) -> None:
with patch("turnstone.console.server.time.sleep", mgr.record_sleep):
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, timeout_sec, stop_event),
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "helper failed to exit on stop_event"
# ``min_sweep_interval=0.0`` disables the production cadence floor
# (default 5 s) so tests can fire many close_idle calls back-to-back
# without waiting real time between them. The floor is exercised
# in its own dedicated test below.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, timeout_sec, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "helper failed to exit on stop_event"
def test_coord_idle_cleanup_runs_initial_sweep_before_sleep() -> None:
"""The first close_idle call must happen BEFORE the first time.sleep
def test_coord_idle_cleanup_runs_initial_sweep_before_wait() -> None:
"""The first close_idle call must happen BEFORE the first wait
otherwise cold-start orphans wait one ``check_every`` interval (~30 min
on default 2h timeout) for the first reap. Crucial because the
lifespan no longer does a synchronous initial sweep."""
lifespan no longer does a synchronous initial sweep.
Verified structurally: a single ``expected_calls=1`` run completes
in well under one ``check_every`` (here 0.04 s timeout 0.01 s
check_every), so the initial sweep must have happened before any
real wait could have blocked it.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
assert mgr.sleep_calls_at_each_close == [0], "first close_idle should run before any sleep"
started = time.monotonic()
_run_until_done(mgr, stop_event, timeout_sec=0.04)
elapsed = time.monotonic() - started
assert len(mgr.calls) == 1
# check_every = min(300.0, 0.04/4) = 0.01 s. An initial sweep
# gated behind one full wait would have taken ~0.01+ s anyway, so
# the upper bound here is "much less than one check_every plus
# process noise" — the explicit 1.0 s gives generous CI headroom
# while still asserting the test is testing the right thing.
assert elapsed < 1.0
def test_coord_idle_cleanup_calls_close_idle_each_tick() -> None:
"""Heartbeat path: with no state-change events, close_idle fires
each ``check_every`` interval. Test uses a tiny timeout so the
test runs fast the contract under test is "the loop iterates",
not the production cadence.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert len(mgr.calls) == 3
assert all(t == 120.0 for t in mgr.calls)
assert all(t == 0.04 for t in mgr.calls)
def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
@@ -91,7 +142,7 @@ def test_coord_idle_cleanup_survives_close_idle_exceptions() -> None:
blip would silently leak orphans forever."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=4, raise_after=1)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
# All four calls must have fired despite calls 2-4 raising.
assert len(mgr.calls) == 4
@@ -102,5 +153,154 @@ def test_coord_idle_cleanup_exits_cleanly_on_stop_event() -> None:
daemon-process termination."""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
_run_until_done(mgr, stop_event, timeout_sec=120.0)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert stop_event.is_set()
def test_state_change_wakes_close_idle_before_heartbeat() -> None:
"""The event-driven path is the whole point of the refactor: a
workstream state-change must wake the cleanup sweep without
waiting one ``check_every`` interval. Tested with a long
timeout_sec so the heartbeat would NOT have fired in the test
window the close_idle call past the initial sweep must come
from a state-change wake.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
# check_every = min(300.0, 120.0/4) = 30 s — well outside the test
# window. Any close_idle call past the initial sweep must come
# from a fire_state_change-driven wake-up.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
# Wait for the initial sweep to complete AND the thread to enter
# its first ``tick_now.wait`` (signalled here by the subscriber
# being registered + calls advancing to 1).
deadline = time.monotonic() + 1.0
while time.monotonic() < deadline:
if mgr.subscribers_count == 1 and len(mgr.calls) >= 1:
break
time.sleep(0.01)
assert mgr.subscribers_count == 1, "thread didn't subscribe to state"
assert len(mgr.calls) == 1, "initial sweep didn't fire"
# One state-change fire wakes the first ``wait`` → close_idle runs
# again → stop_event is set (expected_calls=2) → thread exits.
mgr.fire_state_change()
thread.join(timeout=2.0)
assert not thread.is_alive(), "thread didn't exit after state-change-driven sweep"
# 2 = initial + state-change-driven. If the state change weren't
# being honoured, close_idle would have stalled on the 30 s wait
# and the thread.join would have timed out.
assert len(mgr.calls) == 2
def test_subscriber_unregisters_when_thread_exits() -> None:
"""The cleanup thread's state-change subscriber must be removed
when the thread exits otherwise long-running processes that
restart their cleanup threads (admin model-CRUD path, tests) leak
subscribers and every state change fires N stale callbacks.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=1)
_run_until_done(mgr, stop_event, timeout_sec=0.04)
assert mgr.subscribers_count == 0, "subscriber leaked past thread exit"
def test_state_change_during_close_idle_triggers_followup_sweep() -> None:
"""A state-change fired during the initial sweep (e.g. close_idle's
own ``close()`` calls firing subscribers) must wake the next
``tick_now.wait`` rather than being lost to the clear-before-sweep
ordering. The clear runs INSIDE the loop just before close_idle,
so a fire during the initial sweep which precedes the loop
arrives at an already-set event that the first wait sees set and
returns on immediately.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=2)
real_close_idle = mgr.close_idle
# One-shot fire during the initial sweep, mirroring what
# close_idle's own close() calls do in production (set_state →
# state-change subscribers).
fired = [False]
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
result = real_close_idle(timeout_sec)
if not fired[0]:
fired[0] = True
mgr.fire_state_change()
return result
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.0},
daemon=True,
)
thread.start()
thread.join(timeout=2.0)
assert not thread.is_alive(), "thread blocked on the next wait — mid-sweep wake was lost"
# 2 = initial sweep + state-change-driven follow-up. Without the
# event surviving the clear-before-sweep ordering, the thread
# would have blocked on the 30 s ``wait`` and the test would have
# timed out at thread.join.
assert len(mgr.calls) == 2
def test_min_sweep_interval_floors_close_idle_cadence_under_sustained_wakes() -> None:
"""Cadence floor: even when state-change events keep firing
``tick_now.set()``, ``close_idle`` must not run more often than
``min_sweep_interval`` otherwise the loop tight-spins close_idle
at the rate of its own DB latency, doing 600-1500x more DB work
than the pre-refactor fixed-30 s cadence.
Wires a state-change subscriber that fires another state change
from inside close_idle, so the bus would tick forever if not
floored. Asserts the elapsed-between-sweeps is at least
``min_sweep_interval`` modulo small wall-clock noise.
"""
stop_event = threading.Event()
mgr = _StubMgr(stop_event=stop_event, expected_calls=3)
real_close_idle = mgr.close_idle
sweep_times: list[float] = []
def _instrumented_close_idle(timeout_sec: float) -> list[str]:
sweep_times.append(time.monotonic())
result = real_close_idle(timeout_sec)
# Always fire another state-change to simulate sustained
# activity (each turn fires thinking/running/attention/idle).
# If the floor were absent, the next wake would race the next
# close_idle immediately and ``sweep_times`` deltas would be
# bounded by close_idle latency (microseconds), not the floor.
mgr.fire_state_change()
return result
mgr.close_idle = _instrumented_close_idle # type: ignore[method-assign]
# 0.15 s floor keeps the test fast (~0.3 s total) while still
# representing a meaningful gap relative to close_idle's
# near-zero stub latency.
thread = threading.Thread(
target=_coord_idle_cleanup_thread,
args=(mgr, 120.0, stop_event),
kwargs={"min_sweep_interval": 0.15},
daemon=True,
)
thread.start()
thread.join(timeout=3.0)
assert not thread.is_alive(), "thread didn't exit"
assert len(sweep_times) >= 2, "fewer than two sweeps fired"
# Gap between sweep 1 (post-initial) and sweep 2 must respect
# the floor. Initial sweep at sweep_times[0] is unfloored
# (no prior sweep to compare against), so the meaningful
# assertion is on sweep_times[1] - sweep_times[0].
gap = sweep_times[1] - sweep_times[0]
assert gap >= 0.12, f"floor breached: gap {gap:.3f}s < min_sweep_interval 0.15s"
+219
View File
@@ -0,0 +1,219 @@
"""``console/session_factory.py`` alias-resolution coverage.
The console session factory resolves the coordinator alias through a
three-tier chain that must stay in lockstep with the placeholder logic
in ``console/server.py:list_available_models`` otherwise the home
composer advertises one alias while sessions launch on another.
Tier order (highest priority first):
1. Per-call ``model_alias`` arg, or the ``coordinator.model_alias``
ConfigStore setting (admin-pinned coordinator-specific override).
2. ``model.default_alias`` ConfigStore setting (admin-managed system
default surfaced in the Models tab).
3. ``registry.default`` (config.toml ``[model].default``, the boot-time
fallback).
These tests pin each branch by intercepting ``registry.resolve``
they short-circuit before ChatSession construction so the test never
has to satisfy ChatSession's full kwarg contract.
"""
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
import pytest
from tests._coord_test_helpers import _FakeConfigStore
from turnstone.console.session_factory import build_console_session_factory
class _StopBeforeChatSessionError(Exception):
"""Sentinel raised by the capturing registry to short-circuit
factory execution after alias resolution but before ChatSession is
built. The factory's outer code path is irrelevant to alias
resolution and would force the test to satisfy a long kwarg
contract for no extra coverage."""
class _CapturingRegistry:
"""Records the alias passed to ``resolve()`` and short-circuits.
``has_alias`` answers from the configured known set so the
``model.default_alias`` validation tier behaves realistically.
Mirrors the public surface ``ModelRegistry`` exposes to
session_factory: ``has_alias``, ``resolve``, and ``default``.
"""
def __init__(self, *, default: str, known: set[str]) -> None:
self.default = default
self._known = known
self.captured_alias: str | None = None
def has_alias(self, alias: str) -> bool:
return alias in self._known
def resolve(self, alias: str) -> Any:
self.captured_alias = alias
raise _StopBeforeChatSessionError()
def _build_factory(
*,
registry_default: str = "registry-default",
known_aliases: set[str] | None = None,
settings: dict[str, Any] | None = None,
) -> tuple[Any, _CapturingRegistry]:
"""Construct the factory with stub deps. Returns ``(factory_callable,
registry)`` so tests can read back ``registry.captured_alias``."""
registry = _CapturingRegistry(
default=registry_default,
known=known_aliases if known_aliases is not None else {registry_default},
)
config_store = _FakeConfigStore(dict(settings or {}))
factory = build_console_session_factory(
registry=registry, # type: ignore[arg-type]
config_store=config_store, # type: ignore[arg-type]
node_id="console",
coord_client_factory=lambda ws_id, uid: MagicMock(),
)
return factory, registry
def _invoke(factory: Any, **factory_kwargs: Any) -> None:
"""Call the factory with a stub UI and absorb the sentinel.
Forwards ``factory_kwargs`` to the factory so per-call overrides
(e.g. ``model_alias``) can flow through. Raises if any other
exception comes out the test should fail loudly when alias
resolution itself errors rather than swallowing it.
"""
ui = MagicMock()
ui._user_id = "" # skip storage-backed username lookup branch
with pytest.raises(_StopBeforeChatSessionError):
factory(ui, **factory_kwargs)
# ---------------------------------------------------------------------------
# Tier 1 — explicit pin (per-call arg or coordinator.model_alias)
# ---------------------------------------------------------------------------
def test_per_call_model_alias_arg_wins_over_everything() -> None:
"""The ``model_alias`` kwarg on the factory call (e.g. body field on
POST /workstreams/new) wins over both ConfigStore tiers and the
registry default."""
factory, registry = _build_factory(
known_aliases={"per-call", "coord-pin", "admin-default", "registry-default"},
settings={
"coordinator.model_alias": "coord-pin",
"model.default_alias": "admin-default",
},
)
_invoke(factory, model_alias="per-call")
assert registry.captured_alias == "per-call"
def test_coordinator_model_alias_wins_when_no_per_call_override() -> None:
factory, registry = _build_factory(
known_aliases={"coord-pin", "admin-default", "registry-default"},
settings={
"coordinator.model_alias": "coord-pin",
"model.default_alias": "admin-default",
},
)
_invoke(factory)
assert registry.captured_alias == "coord-pin"
def test_coordinator_model_alias_passed_through_unvalidated() -> None:
"""Tier 1 is an *explicit* operator pin — when it's stale or typoed
we deliberately pass it through to ``registry.resolve`` so the
request layer turns it into a 503 with the alias surfaced in the
error. Falling through silently would mask the misconfiguration."""
factory, registry = _build_factory(
known_aliases={"admin-default", "registry-default"},
settings={
"coordinator.model_alias": "ghost", # unknown
"model.default_alias": "admin-default",
},
)
_invoke(factory)
assert registry.captured_alias == "ghost"
def test_per_call_model_alias_arg_passed_through_unvalidated() -> None:
"""The per-call ``model_alias`` kwarg (POST body field — the more
common production trigger) is the same kind of explicit pin as the
ConfigStore setting, so a stale value passes through to
``registry.resolve`` rather than silently falling through to the
system default."""
factory, registry = _build_factory(
known_aliases={"registry-default"},
settings={"model.default_alias": "registry-default"},
)
_invoke(factory, model_alias="ghost")
assert registry.captured_alias == "ghost"
# ---------------------------------------------------------------------------
# Tier 2 — model.default_alias (admin-managed system default)
# ---------------------------------------------------------------------------
def test_model_default_alias_used_when_coordinator_unset() -> None:
"""Regression for the historical drift: admin sets the system
default in the Models tab, the home composer advertises it, and new
coordinator sessions must launch on the same alias rather than
silently falling through to ``registry.default``."""
factory, registry = _build_factory(
known_aliases={"admin-default", "registry-default"},
settings={"model.default_alias": "admin-default"},
)
_invoke(factory)
assert registry.captured_alias == "admin-default"
def test_unknown_model_default_alias_falls_through_to_registry_default() -> None:
"""Tier 2 is *not* an explicit pin — operators set
``model.default_alias`` once in the UI and forget about it; an alias
that's later disabled or typo'd should not 503 the coordinator,
since tier 3 (``registry.default``) is guaranteed to resolve."""
factory, registry = _build_factory(
known_aliases={"registry-default"}, # admin-default got removed
settings={"model.default_alias": "admin-default"},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
def test_blank_model_default_alias_falls_through_to_registry_default() -> None:
factory, registry = _build_factory(
settings={"model.default_alias": ""},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
# ---------------------------------------------------------------------------
# Tier 3 — registry.default (config.toml [model].default)
# ---------------------------------------------------------------------------
def test_no_settings_uses_registry_default() -> None:
factory, registry = _build_factory()
_invoke(factory)
assert registry.captured_alias == "registry-default"
def test_whitespace_only_coord_alias_falls_through() -> None:
"""``" "`` is not an explicit pin — ``.strip()`` reduces it to
"", which the chain should treat as unset."""
factory, registry = _build_factory(
settings={"coordinator.model_alias": " "},
)
_invoke(factory)
assert registry.captured_alias == "registry-default"
+79
View File
@@ -716,3 +716,82 @@ class TestCoordinatorAdapterDispatchChildEvent:
},
)
assert recorder.enqueued == []
def test_dispatch_notifies_child_event_bus_on_state_event(self) -> None:
"""Every translated state-class event must call
``ChildEventBus.notify(ws_id)`` so a registered
``wait_for_workstream`` waiter wakes promptly. Notify fires
AFTER the UI enqueue so the SSE fan-out keeps priority the
order assertion here is structural (one notify call, matching
ws_id) since the bus side-effect lookup is what guards against
regressions, not the relative event ordering.
"""
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-a1"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "idle",
}
)
assert event.is_set(), "bus notify did not fire on cluster_state dispatch"
def test_dispatch_notifies_for_all_state_class_event_types(self) -> None:
"""The dispatch sink translates six event types into the
``child_ws_*`` SSE shape; all six must also fire the bus so
a wait on any of them wakes. ``ws_created`` is intentionally
NOT in this set waiters register against ws_ids they already
know exist (the wait tool takes a pre-known list)."""
for etype, extra in [
("cluster_state", {"state": "running"}),
("ws_closed", {"reason": "evicted"}),
("ws_rename", {"name": "renamed"}),
("intent_verdict", {"verdict": {"call_id": "c1"}}),
("approval_resolved", {"approved": True}),
("approve_request", {"detail": {}}),
]:
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-a1"])
adapter._dispatch_child_event(
{"type": etype, "ws_id": "child-a1", **extra},
)
assert event.is_set(), f"bus notify did not fire on {etype} dispatch"
def test_dispatch_does_not_notify_for_unrelated_ws_id(self) -> None:
"""Bus is keyed by ws_id — a dispatch for ws X must not wake a
waiter registered against ws Y, or every state change anywhere
in the system would shake every concurrent wait."""
adapter, _, _ = self._setup()
adapter._registry.merge_children("coord-a", ["child-a1"])
bus = adapter.child_event_bus
event = bus.register_waiter(["child-other"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "child-a1",
"state": "idle",
}
)
assert not event.is_set(), "bus notify spuriously fired on unrelated ws_id"
def test_dispatch_does_not_notify_for_unknown_child(self) -> None:
"""Events whose ws_id isn't in any coord's registry are dropped
BEFORE the bus notify (early return at ``coord_id is None``).
Notify only fires for events the dispatch sink fully translated,
keeping the bus side-effect aligned with the UI enqueue."""
adapter, _, _ = self._setup()
bus = adapter.child_event_bus
event = bus.register_waiter(["ws-orphan"])
adapter._dispatch_child_event(
{
"type": "cluster_state",
"ws_id": "ws-orphan",
"state": "idle",
}
)
assert not event.is_set(), "bus notify fired for ws_id the dispatch dropped"
+649 -9
View File
@@ -9,6 +9,7 @@ storage-call path.
from __future__ import annotations
import json
import time
from typing import TYPE_CHECKING, Any
import httpx
@@ -20,6 +21,7 @@ from turnstone.console.coordinator_client import (
CoordinatorTokenManager,
)
from turnstone.core.auth import JWT_AUD_CONSOLE, validate_jwt
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
@@ -145,6 +147,7 @@ def _mock_client(
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
return client, captured
@@ -470,6 +473,7 @@ def _make_read_client(storage: SQLiteBackend) -> CoordinatorClient:
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
@@ -663,6 +667,7 @@ def _make_client_with_cluster_response(
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
@@ -1223,6 +1228,49 @@ def test_list_skills_hides_interactive_only_skills(tmp_path):
assert skill["kind"] in {"coordinator", "any"}
def test_list_skills_omits_allowed_tools_when_empty(tmp_path):
"""``allowed_tools`` is the auto-approve allowlist (tools exempt
from the operator approval gate), NOT the set of tools the skill
can use. An empty list reads as "no tool access" to a model
that doesn't know the semantics — real misdiagnosis source: a
code-review skill with no auto-approve allowlist looked like it
had been spawned with zero tools. Dropping the key when empty
removes the ambiguity at the source; absence of the field carries
the unambiguous meaning "no tool is pre-approved for this skill"
while a tool list reads as "these specific tools bypass the prompt".
"""
st = SQLiteBackend(str(tmp_path / "skills_empty.db"))
st.create_prompt_template(
template_id="s-empty",
name="empty-skill",
category="ops",
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags="[]",
allowed_tools="[]",
)
st.create_prompt_template(
template_id="s-nonempty",
name="nonempty-skill",
category="ops",
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags="[]",
allowed_tools='["read_file"]',
)
client = _make_read_client(st)
result = client.list_skills()
by_name = {s["name"]: s for s in result["skills"]}
assert "allowed_tools" not in by_name["empty-skill"]
assert by_name["nonempty-skill"]["allowed_tools"] == ["read_file"]
def test_list_skills_projects_allowed_tools_capped_with_sentinel(tmp_path):
"""Each row carries the skill's allowed_tools (capped at the projection
cap with a +N more sentinel) so coordinators can pick a skill without
@@ -1443,6 +1491,23 @@ def test_wait_for_workstream_denies_foreign_ws_id(populated_storage):
assert result["elapsed"] < 1.0
def test_wait_for_workstream_denies_cross_tenant_child(populated_storage):
"""Defense-in-depth (Copilot #506): a row whose ``parent_ws_id``
matches the coordinator but whose ``user_id`` belongs to a
different tenant must collapse to ``denied`` otherwise a
forged / migration-era / pre-tenant-gate row would let a
coordinator's LLM observe foreign-tenant state through
``wait_for_workstream``. The ``populated_storage`` fixture's
``cross-tenant-child`` row has exactly this shape
(parent_ws_id="coord-1", user_id="user-2").
"""
client = _make_read_client(populated_storage)
result = client.wait_for_workstream(["cross-tenant-child"], timeout=5, mode="any")
assert result["results"]["cross-tenant-child"]["state"] == "denied"
assert result["complete"] is False
assert result["elapsed"] < 1.0
def test_wait_for_workstream_missing_ws_id_indistinguishable_from_denied(populated_storage):
"""A ws_id that doesn't exist collapses into the same 'denied'
shape as a foreign ws_id so wait can't be used as an existence
@@ -1531,10 +1596,22 @@ def test_wait_for_workstream_dedupes_ws_ids(populated_storage):
assert list(result["results"].keys()) == ["child-a"]
def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monkeypatch):
"""Per-tick polling must issue batched storage calls — at the
documented cap (32 ws_ids over a 600s wait) the naive per-id
shape produced ~38k row reads. Guard against regression."""
def test_wait_for_workstream_never_falls_back_to_per_id_storage_calls(
populated_storage, monkeypatch
):
"""All storage reads issued by ``wait_for_workstream`` must go
through the batched paths. At the documented cap (32 ws_ids over
a 600 s wait) the naive per-id shape produced ~38k row reads, so
a regression to per-id is the meaningful failure mode this test
guards against.
The primary safety net is the ``pytest.fail`` mock on the per-id
``get_workstream`` / ``sum_workstream_tokens`` paths any call
there blows up loudly with the regression message. The
additional ``batch_calls`` / ``sum_calls`` assertions cover the
subtler regression where the call IS batched but only covers a
subset of ws_ids (e.g. one ws_id per call in a loop).
"""
client = _make_read_client(populated_storage)
batch_calls: list[list[str]] = []
sum_calls: list[list[str]] = []
@@ -1565,11 +1642,16 @@ def test_wait_for_workstream_uses_batched_storage_calls(populated_storage, monke
result = client.wait_for_workstream(["child-a", "child-b"], timeout=5, mode="any")
assert result["complete"] is True
# One tick is enough since child-a is already idle (terminal).
assert len(batch_calls) == 1
assert len(sum_calls) == 1
assert set(batch_calls[0]) == {"child-a", "child-b"}
assert set(sum_calls[0]) == {"child-a", "child-b"}
# Every batched call carried the full ws_id set. The exact count
# (currently 2: one pre-loop ownership filter + one snapshot tick)
# is incidental; if either gains another batched read it stays
# batched, which is the property under test.
assert batch_calls, "no batched get_workstreams_batch call observed"
assert sum_calls, "no batched sum_workstream_tokens_batch call observed"
first_batch = set(batch_calls[0])
first_sum = set(sum_calls[0])
assert first_batch == {"child-a", "child-b"}
assert first_sum == {"child-a", "child-b"}
def test_wait_for_workstream_handles_non_string_mode(populated_storage):
@@ -1581,6 +1663,205 @@ def test_wait_for_workstream_handles_non_string_mode(populated_storage):
assert "invalid mode" in result["error"]
# ---------------------------------------------------------------------------
# wait_for_workstream — event-driven (ChildEventBus wired in)
# ---------------------------------------------------------------------------
#
# When the coord adapter wires its ``child_event_bus`` into the client,
# the wait loop blocks on a per-call ``threading.Event`` keyed by ws_id
# and only re-snapshots storage on state-change wakes or the heartbeat
# cap. The legacy ``time.sleep`` poll path remains intact for tests
# that don't wire the bus (above), so this section adds focused
# coverage of the bus-driven behaviour without re-running the full
# matrix of mode / since / cross-tenant cases.
def _make_read_client_with_bus(storage, bus) -> CoordinatorClient:
"""Like ``_make_read_client`` but wires a real ``ChildEventBus``.
Caller owns the bus so the test can call ``bus.notify(ws_id)`` to
simulate the dispatch-sink wake-up.
"""
transport = httpx.MockTransport(lambda r: httpx.Response(200))
http = httpx.Client(transport=transport)
return CoordinatorClient(
console_base_url="http://x",
storage=storage,
token_factory=lambda: "t",
coord_ws_id="coord-1",
user_id="user-1",
http_client=http,
child_event_bus=bus,
)
def test_wait_with_bus_returns_immediately_when_already_terminal(populated_storage):
"""Subscribe-after-terminal race: the wait registers its waiter
BEFORE the first snapshot, then re-snapshots an already-terminal
child must return at once without spinning the heartbeat cap.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
result = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
assert result["complete"] is True
assert result["results"]["child-a"]["state"] == "idle"
assert result["elapsed"] < 1.0
# Waiter must be unregistered on exit so a long-lived bus doesn't
# accumulate dead keys across many waits.
assert "child-a" not in bus._waiters
def test_wait_with_bus_wakes_on_notify(populated_storage):
"""The core property of the refactor: a state-change ``notify``
must wake the wait promptly well under the legacy 0.5 s poll
cadence AND the 2 s heartbeat cap. Test fires a state update
+ notify after a short delay and asserts the wait returns quickly.
"""
import threading as _t
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# child-b starts running; flip to idle + notify after the wait
# blocks. 100 ms is enough that the wait is parked in event.wait()
# but short enough that the test runs fast.
timer = _t.Timer(
0.1,
lambda: (
populated_storage.update_workstream_state("child-b", "idle"),
bus.notify("child-b"),
),
)
timer.start()
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=5.0, mode="any")
elapsed = time.monotonic() - start
assert result["complete"] is True
assert result["results"]["child-b"]["state"] == "idle"
# Bus-driven wake should fire well under 1 s; legacy poll would
# take ~0.5 s but bus-driven should be ~0.1 s (the timer delay)
# plus a few ms. Generous 0.6 s budget for CI noise.
assert elapsed < 0.6, f"wake-up too slow: {elapsed}s"
def test_wait_with_bus_unrelated_notify_does_not_wake(populated_storage):
"""A notify on a ws_id the wait isn't watching must NOT wake it —
otherwise every state change anywhere on the system would shake
every concurrent wait into a redundant storage snapshot.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# child-b is running indefinitely; mode='all' will time out unless
# a relevant notify fires. Fire only unrelated notifies — wait
# should still hit the full timeout.
import threading as _t
def _fire_unrelated() -> None:
for _ in range(5):
bus.notify("ws-unrelated-1")
bus.notify("ws-unrelated-2")
time.sleep(0.05)
t = _t.Thread(target=_fire_unrelated, daemon=True)
t.start()
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=0.5, mode="all")
elapsed = time.monotonic() - start
assert result["complete"] is False, "unrelated notify falsely satisfied wait"
# Wait should burn its full timeout (give or take heartbeat
# granularity). The bus path doesn't have a 0.5 s poll, so the
# bound is "approximately timeout".
assert elapsed >= 0.5
t.join(timeout=1.0)
def test_wait_with_bus_heartbeat_still_progresses_without_notify(populated_storage):
"""Without any notify, the wait must still progress through ticks
via the heartbeat cap so ``progress_callback`` keeps firing for
the sidebar UI. Verified by counting callback firings over an
interval longer than the heartbeat.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# Shrink the heartbeat for test speed via the ClassVar seam —
# instance attribute shadows the class-level default. Production
# stays at 2.0 s; the test exercises the heartbeat-fires-without-
# notify property in well under 1 s.
client._WAIT_HEARTBEAT_INTERVAL = 0.1 # type: ignore[misc]
snapshots: list[dict[str, dict[str, object]]] = []
def _cb(snap: dict[str, dict[str, object]], _elapsed: float) -> None:
snapshots.append(snap)
# child-b is running indefinitely; wait will time out at 0.4 s.
# With heartbeat = 0.1 s, we expect ~3-5 callback firings
# (initial tick + ~3-4 heartbeats). Loose lower bound to avoid
# CI flakiness.
start = time.monotonic()
result = client.wait_for_workstream(["child-b"], timeout=0.4, mode="all", progress_callback=_cb)
elapsed = time.monotonic() - start
assert result["complete"] is False
assert elapsed >= 0.4
# At least 2 callback firings: the initial snapshot plus at least
# one heartbeat-driven re-tick. Tight upper bound would be
# ~ceil(0.4/0.1) + 1 = 5 firings.
assert len(snapshots) >= 2, f"heartbeat didn't fire: {len(snapshots)} snapshots"
def test_wait_with_bus_unregisters_waiter_on_exit(populated_storage):
"""Both the success path and the timeout path must unregister the
waiter otherwise a long-lived bus accumulates dead
``threading.Event`` instances forever.
"""
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
# Success path (already-terminal child).
client.wait_for_workstream(["child-a"], timeout=5, mode="any")
assert bus._waiters == {}, "success path leaked waiter"
# Timeout path (running child, mode='all' that times out).
client.wait_for_workstream(["child-a", "child-b"], timeout=0.3, mode="all")
assert bus._waiters == {}, "timeout path leaked waiter"
def test_wait_with_bus_multi_waiter_independence(populated_storage):
"""Two concurrent waits on the same ws_id must be independent —
one wait completing must not affect the other's wake-up state.
Smoke-tests the multi-Event-per-bucket bus behaviour against the
real wait-loop.
"""
import threading as _t
from turnstone.core.child_event_bus import ChildEventBus
bus = ChildEventBus()
client = _make_read_client_with_bus(populated_storage, bus)
results: dict[str, dict[str, object]] = {}
def _do_wait(label: str) -> None:
results[label] = client.wait_for_workstream(["child-a"], timeout=5, mode="any")
threads = [_t.Thread(target=_do_wait, args=(f"t{i}",), daemon=True) for i in range(3)]
for t in threads:
t.start()
for t in threads:
t.join(timeout=5.0)
for label in ("t0", "t1", "t2"):
assert results[label]["complete"] is True
assert results[label]["results"]["child-a"]["state"] == "idle"
# All waiters must be unregistered after exit.
assert bus._waiters == {}
# ---------------------------------------------------------------------------
# wait_for_workstream — last-message bundling
# ---------------------------------------------------------------------------
@@ -2378,3 +2659,362 @@ def test_cleanup_dead_task_child_refs_storage_batch_failure_swallows(populated_s
populated_storage.get_workstreams_batch = _boom # type: ignore[method-assign]
assert client.cleanup_dead_task_child_refs("coord-1") == 0
# ---------------------------------------------------------------------------
# inspect_workstream — three-tier output compression
# ---------------------------------------------------------------------------
#
# A coord doing a fan-out wave against tool-heavy children would
# otherwise blow the context budget on raw output alone. Mirrors the
# search tool's Tier-1/Tier-2/Tier-3 ladder.
def _make_inspect_result(
*, ws_id: str = "ws-test", state: str = "running", n_messages: int = 5
) -> dict[str, Any]:
"""Build an inspect-result dict shaped like ``coordinator_client.inspect()``.
Production output keys (``ws_id``, ``skill_id``) mirror the storage
row that ``inspect()`` spreads from ``get_workstream``. Tests that
synthesize an inspect result must match these keys otherwise a
formatter that looks at the production keys silently emits null
values against a fixture that uses different ones (real bug-1
regression source: skeleton tier read ``skill`` from a fixture
that wrote ``skill`` while production wrote ``skill_id``).
"""
return {
"ws_id": ws_id,
"state": state,
"title": "test workstream",
"skill_id": "researcher",
"messages": [
{"role": "user" if i % 2 == 0 else "assistant", "content": f"msg {i} content"}
for i in range(n_messages)
],
"verdicts": [],
}
def test_format_inspect_tiered_full_fits_returns_full_tier():
"""Small payloads pass through with `_tier='full'` — no compression."""
from turnstone.console.coordinator_client import _format_inspect_tiered
result = _make_inspect_result(n_messages=3)
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "full"
# Every message verbatim.
assert len(parsed["messages"]) == 3
assert parsed["messages"][0]["content"] == "msg 0 content"
def test_format_inspect_tiered_compact_when_full_exceeds_budget():
"""Large messages trigger the compact tier — head/tail-snipped
content with the rest of the row intact."""
from turnstone.console.coordinator_client import (
_INSPECT_MSG_CONTENT_HEAD,
_INSPECT_MSG_CONTENT_TAIL,
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# Each message ~5KB; with 20 messages, full tier blows the 32KB budget.
fat = "X" * 5000
result = {
"id": "ws-fat",
"state": "running",
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
# Every message preserved (compact keeps the count, just snips content).
assert len(parsed["messages"]) == 20
# Head/tail snip kicked in.
msg_content = parsed["messages"][0]["content"]
assert msg_content.startswith("X" * _INSPECT_MSG_CONTENT_HEAD)
assert msg_content.endswith("X" * _INSPECT_MSG_CONTENT_TAIL)
assert "chars elided" in msg_content
# Budget invariant — the load-bearing contract of the formatter.
# Without this assertion, a future change to ``_tier_note`` or
# ``_compact_message`` could push the output over budget and the
# ``_truncate_output`` head+tail safety net would silently mask
# the regression, re-introducing the middle-message-drop pathology.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_compact_when_content_below_snip_threshold():
"""When per-message content is below the snip threshold but the
message COUNT alone overflows the budget, compact tier must still
stay within budget by trimming the message list (head + tail of
messages) rather than degrading straight to skeleton. Bug-3
regression cover: with 400 × 100-char messages, the original
formatter fell through to skeleton because adding ``_tier_note``
to an un-snipped tier-2 produced output strictly larger than
tier-1 (both over budget). The fix preserves messages from both
ends of the list and inserts an ``_omitted`` sentinel."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# 400 × ~100 chars → Tier-1 ~53 KB (over budget), per-message
# content under the 964-char snip threshold so content-snipping
# saves nothing. Without the list-trim rung the formatter would
# fall to skeleton and drop all 400 messages.
smallish = "S" * 100
result = {
"ws_id": "ws-many-small",
"state": "running",
"messages": [
{"role": "assistant" if i % 2 == 0 else "user", "content": smallish} for i in range(400)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
# Should NOT fall through to skeleton — message-list trim preserves
# head + tail of the conversation.
assert parsed["_tier"] == "compact"
assert "messages" in parsed
# Some messages must survive; the trim shape is head + tail with an
# ``_omitted`` sentinel between them.
assert len(parsed["messages"]) > 0
assert len(parsed["messages"]) < 400
# Budget invariant.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_skeleton_when_compact_also_exceeds_budget():
"""Tier 3 fallback: counts + last assistant preview only. Trigger by
flooding with messages whose content is a multi-block list the
snipper correctly leaves non-string content unchanged (mirrors
Anthropic/OpenAI multi-block content shape), so even after the
(5, 10) message-list trim the surviving 15 messages don't fit in
the 32 KB budget."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# 50 messages × multi-block content (~30 KB each — list-shape
# content bypasses the head/tail string snipper because lists
# aren't strings). Even (5, 10) trim leaves 15 × 30 KB which
# blows the 32 KB budget — forces skeleton.
fat_block = {"type": "text", "text": "Y" * 3000}
result = {
"ws_id": "ws-flood",
"state": "running",
"title": "flood",
"skill_id": "researcher",
"messages": [
{
"role": "assistant" if i % 2 == 0 else "user",
"content": [fat_block] * 10,
}
for i in range(50)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "skeleton"
assert parsed["message_count"] == 50
# Role distribution surfaces — the "what shape of activity" signal.
assert parsed["roles"]["assistant"] == 25
assert parsed["roles"]["user"] == 25
# No `messages` field at skeleton tier — only the aggregate signal.
assert "messages" not in parsed
# Budget invariant.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_skeleton_keeps_terminal_state_fields():
"""``close_reason`` / ``last_error`` survive the skeleton fall — they're
small, load-bearing, and the operator needs them to understand WHY
a terminal child landed in its state."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# Same flood pattern as the bare-skeleton test (multi-block content
# bypasses the string snipper) — paired with terminal-state fields
# that must survive the skeleton fall.
fat_block = {"type": "text", "text": "Z" * 3000}
result = {
"ws_id": "ws-closed",
"state": "closed",
"title": "done",
"skill_id": "researcher",
"messages": [{"role": "user", "content": [fat_block] * 10} for _ in range(50)],
"verdicts": [],
"close_reason": "task complete: report attached",
"live": None, # filtered by truthy check
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "skeleton"
assert parsed["close_reason"] == "task complete: report attached"
# Falsy ``live`` doesn't bleed through.
assert "live" not in parsed
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_error_shapes_bypass_tiering():
"""Cross-tenant / not-found responses keep their original shape — they
carry no messages, are already tiny, and changing them would break
callers that key on the ``error`` field."""
from turnstone.console.coordinator_client import _format_inspect_tiered
result = {"error": "workstream not found", "ws_id": "ws-foreign"}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed == {"error": "workstream not found", "ws_id": "ws-foreign"}
# No `_tier` annotation — error shapes are self-describing.
assert "_tier" not in parsed
def test_format_inspect_tiered_compact_preserves_tool_call_linkage():
"""Compact tier keeps ``tool_name`` / ``tool_call_id`` / ``name`` so a
model reading the snipped trace can still pair a tool call to its
response the linkage is load-bearing for "what happened" signal."""
from turnstone.console.coordinator_client import _format_inspect_tiered
fat = "Q" * 5000
result = {
"ws_id": "ws-tools",
"state": "running",
"messages": [
{
"role": "assistant",
"content": fat,
"tool_name": "bash",
"tool_call_id": "call-1",
}
for _ in range(20)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
first = parsed["messages"][0]
assert first["tool_name"] == "bash"
assert first["tool_call_id"] == "call-1"
def test_format_inspect_tiered_compact_preserves_assistant_tool_calls():
"""Compact tier must preserve the assistant-side ``tool_calls`` list
(OpenAI shape: ``[{id, type, function: {name, arguments}}]``) so a
model reading the snipped trace can see WHICH tool was called and
pair it with the corresponding result row via ``id`` ``tool_call_id``.
Bug-2 regression cover: the pre-fix compactor stripped ``tool_calls``,
leaving the audit reader with a tool-result orphan against an
invisible call.
``function.arguments`` strings are snipped head/tail (analogous to
content) because they can be multi-KB JSON; ``id`` and
``function.name`` are preserved verbatim they're the linkage."""
from turnstone.console.coordinator_client import (
_INSPECT_TOOL_ARG_HEAD,
_INSPECT_TOOL_ARG_TAIL,
_format_inspect_tiered,
)
fat_content = "C" * 5000 # forces compact tier
fat_args = "A" * 5000 # forces argument snipping
tool_calls = [
{
"id": "call-abc-123",
"type": "function",
"function": {"name": "bash", "arguments": fat_args},
},
{
"id": "call-def-456",
"type": "function",
"function": {"name": "read_file", "arguments": fat_args},
},
]
result = {
"ws_id": "ws-tool-calls",
"state": "running",
"messages": [
{"role": "assistant", "content": fat_content, "tool_calls": tool_calls}
for _ in range(20)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
first = parsed["messages"][0]
# tool_calls survives compaction.
assert "tool_calls" in first
assert len(first["tool_calls"]) == 2
# Linkage fields verbatim.
assert first["tool_calls"][0]["id"] == "call-abc-123"
assert first["tool_calls"][0]["function"]["name"] == "bash"
assert first["tool_calls"][1]["id"] == "call-def-456"
assert first["tool_calls"][1]["function"]["name"] == "read_file"
# arguments snipped head/tail — both prefix and suffix preserved.
snipped_args = first["tool_calls"][0]["function"]["arguments"]
assert snipped_args.startswith("A" * _INSPECT_TOOL_ARG_HEAD)
assert snipped_args.endswith("A" * _INSPECT_TOOL_ARG_TAIL)
assert "chars elided" in snipped_args
def test_format_inspect_tiered_compact_passes_small_messages_through_unsnipped():
"""Messages under the snip threshold pass through verbatim at compact
tier snipping a 100-byte message costs more bytes (the elision
marker) than it saves."""
from turnstone.console.coordinator_client import _format_inspect_tiered
# Mix: a few large messages force compact tier; small messages must
# not be snipped.
big = "B" * 5000
small = "S" * 50
result = {
"id": "ws-mixed",
"state": "running",
"messages": [{"role": "assistant", "content": big} for _ in range(15)]
+ [{"role": "user", "content": small}],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
# The trailing small message is exact, not snipped.
assert parsed["messages"][-1]["content"] == small
def test_format_inspect_tiered_emits_tier_note_when_compressed():
"""The ``_tier_note`` advisory tells the LLM how to ask for a tighter
or fuller view next time actionable feedback rather than a bare
"we compressed your output" signal."""
from turnstone.console.coordinator_client import _format_inspect_tiered
fat = "F" * 5000
result = {
"id": "ws-noted",
"state": "running",
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert "_tier_note" in parsed
assert "message_limit" in parsed["_tier_note"]
def test_format_inspect_tiered_full_tier_omits_tier_note():
"""When the full tier fits, no note is emitted — the absence of a
note is the signal that nothing was compressed."""
from turnstone.console.coordinator_client import _format_inspect_tiered
out = _format_inspect_tiered(_make_inspect_result(n_messages=2))
parsed = json.loads(out)
assert parsed["_tier"] == "full"
assert "_tier_note" not in parsed
+3
View File
@@ -40,6 +40,7 @@ from turnstone.console.server import (
_require_coord_mgr,
)
from turnstone.core.auth import AuthResult
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.session_manager import SessionManager
from turnstone.core.session_routes import (
SessionEndpointConfig,
@@ -286,6 +287,7 @@ def test_coordinator_client_spawn_close_delete(tmp_path):
coord_ws_id="coord-42",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
# spawn ---------------------------------------------------------------
@@ -387,6 +389,7 @@ def _read_client(storage: SQLiteBackend) -> CoordinatorClient:
coord_ws_id="coord-root",
user_id="user-1",
http_client=http,
child_event_bus=ChildEventBus(),
)
+33 -4
View File
@@ -215,7 +215,12 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
summary tempted callers to write ``if result["status"] == "idle"``
which silently never matched. The summary now omits the field
entirely; lifecycle state lives on the workstream row and is read
via inspect_workstream."""
via inspect_workstream.
Also asserts the return key is ``child_ws_id`` (not ``ws_id``) so
the coordinator LLM doesn't recency-bias toward feeding the spawn
output back into another ``spawn_workstream(ws_id=...)`` call.
"""
sess, coord, _ui = coord_session
coord.spawn.return_value = {
"ws_id": "child-7",
@@ -227,8 +232,9 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
_call_id, output = sess._exec_spawn_workstream(item)
body = json.loads(output)
assert "status" not in body
assert "ws_id" not in body
# The substantive fields are still here.
assert body["ws_id"] == "child-7"
assert body["child_ws_id"] == "child-7"
assert body["node_id"] == "node-1"
@@ -248,6 +254,10 @@ def test_spawn_batch_exec_does_not_surface_misleading_status_field(coord_session
body = json.loads(output)
assert "0" in body["results"]
assert "status" not in body["results"]["0"]
# Per-result entries surface ``child_ws_id``, not ``ws_id`` — same
# recency-bias rationale as the spawn_workstream test above.
assert body["results"]["0"]["child_ws_id"] == "c-x"
assert "ws_id" not in body["results"]["0"]
def test_spawn_exec_surfaces_client_error(coord_session):
@@ -260,6 +270,21 @@ def test_spawn_exec_surfaces_client_error(coord_session):
assert ui.tool_results[-1][3] is True # is_error
def test_spawn_exec_treats_missing_ws_id_on_success_path_as_error(coord_session):
"""A malformed upstream response (200-success-shape with no
``ws_id``) used to emit ``{"child_ws_id": null}`` to the LLM,
which then chased a null id through follow-up tools. Now matches
the matching guard in ``_exec_spawn_batch``: surface as a tool
error so the model retries instead of acting on garbage."""
sess, coord, ui = coord_session
# No ``error`` field, but ``ws_id`` is missing — the silent-null path.
coord.spawn.return_value = {"name": "c", "node_id": "node-1", "status": 200}
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
_call_id, output = sess._exec_spawn_workstream(item)
assert "no ws_id" in output
assert ui.tool_results[-1][3] is True # is_error
# ---------------------------------------------------------------------------
# inspect_workstream
# ---------------------------------------------------------------------------
@@ -1410,9 +1435,13 @@ def test_spawn_batch_exec_serialises_spawns_and_returns_results(coord_session):
assert body["denied"] == []
# Keyed by input index (stringified).
assert set(body["results"].keys()) == {"0", "1", "2"}
assert body["results"]["0"]["ws_id"] == "child-0"
assert body["results"]["0"]["child_ws_id"] == "child-0"
assert body["results"]["1"]["node_id"] == "n-1"
assert body["results"]["2"]["ws_id"] == "child-2"
assert body["results"]["2"]["child_ws_id"] == "child-2"
# Confirm we don't leak the old ``ws_id`` key alongside the new
# ``child_ws_id`` — see test_spawn_exec_does_not_surface_misleading_status_field
# for the rationale on the rename.
assert "ws_id" not in body["results"]["0"]
def test_spawn_batch_exec_surfaces_per_item_errors_in_denied(coord_session):
+222
View File
@@ -453,3 +453,225 @@ class TestDecorateAdvisoryExtraction:
assert tool_msg["advisories"] == [
{"type": "user_interjection", "text": "check the logs", "priority": "notice"}
]
class TestExtractReasoningForHistory:
"""``extract_reasoning_for_history`` — Phase 1 surfaces stored
Anthropic thinking blocks on assistant messages and strips
``_provider_content`` from the wire payload.
Drives through the real ``AnthropicProvider.extract_reasoning_text``
(no mock-of-extractor) the helper test and the provider unit
test (``tests/test_provider_anthropic_reasoning.py``) together
catch a regression at either layer distinctly.
"""
def _anthropic_thinking_msg(self, text: str = "let me think") -> dict[str, object]:
return {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": text, "signature": "sig"},
{"type": "text", "text": "Final answer."},
],
}
def test_extract_thinking_surfaces_reasoning_field(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("let me think")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "let me think"
def test_strips_provider_content_after_extraction(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("anything")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "_provider_content" not in messages[0]
def test_strips_provider_content_when_flag_false(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [self._anthropic_thinking_msg("anything")]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=False)
# Strip is unconditional; reasoning is the conditional bit.
assert "_provider_content" not in messages[0]
assert "reasoning" not in messages[0]
def test_first_block_thinking_dispatches_to_anthropic(self) -> None:
# Even when text and tool_use blocks follow, the first-block-type
# discriminator routes thinking-prefixed payloads correctly.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "first", "signature": "s"},
{"type": "text", "text": "spoken"},
{"type": "tool_use", "id": "t1", "name": "f", "input": {}},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "first"
def test_first_block_reasoning_dispatches_to_openai_responses(self) -> None:
# Phase 3: dispatcher routes type=="reasoning" to the
# OpenAI Responses extractor, which now returns the
# summary[*].text concatenation. Pre-Phase-3 this asserted
# "" (the stub); the assertion was tightened once the wire
# path landed.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "s"}]}
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "s"
assert "_provider_content" not in messages[0]
def test_unknown_first_block_type_no_op(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [{"type": "text", "text": "no reasoning here"}],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "_provider_content" not in messages[0]
def test_skips_messages_without_provider_content(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [{"role": "assistant", "content": "plain"}]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert messages[0]["content"] == "plain"
def test_user_and_tool_messages_untouched(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages: list[dict[str, object]] = [
{"role": "user", "content": "hi"},
{"role": "tool", "tool_call_id": "c1", "content": "out"},
self._anthropic_thinking_msg("only this one"),
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "reasoning" not in messages[1]
assert messages[2]["reasoning"] == "only this one"
def test_empty_provider_content_no_extraction(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [{"role": "assistant", "content": "x", "_provider_content": []}]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
# Empty-list provider_content is still stripped from the wire.
assert "_provider_content" not in messages[0]
def test_first_block_not_a_dict_skipped(self) -> None:
from turnstone.core.history_decoration import extract_reasoning_for_history
messages: list[dict[str, object]] = [
{
"role": "assistant",
"content": "x",
"_provider_content": ["bogus"],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert "reasoning" not in messages[0]
assert "_provider_content" not in messages[0]
def test_first_block_reasoning_text_dispatches_to_openai_chat(self) -> None:
# Phase 3 path 3: synthetic ``reasoning_text`` blocks (stamped
# by ChatSession._maybe_synth_reasoning_block for vLLM /
# llama.cpp / Gemini-compat conversations) dispatch to
# OpenAIChatCompletionsProvider.extract_reasoning_text.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "synth thought"
assert "_provider_content" not in messages[0]
def test_dispatcher_scans_past_unrecognized_first_blocks(self) -> None:
# Regression for Copilot finding: dispatcher used to inspect
# only provider_content[0]['type']. OpenAI Responses captures
# EVERY output_item.done event into provider_blocks (not just
# reasoning), so a hypothetical [message, reasoning, ...]
# ordering would have silently dropped the reasoning. Now
# walks the list for the first recognised reasoning-bearing
# type and dispatches the whole list to that provider.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
# First block is a non-reasoning OpenAI Responses item.
{"type": "message", "role": "assistant", "content": "answer"},
# Reasoning sits later in the list.
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "deferred"}],
},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "deferred"
assert "_provider_content" not in messages[0]
def test_first_block_redacted_thinking_dispatches_to_anthropic(self) -> None:
# Anthropic's extended-thinking API documents that
# ``redacted_thinking`` blocks (sealed by the safety system)
# can appear before, after, or interleaved with regular
# ``thinking`` blocks. When the redacted block lands first,
# the dispatcher must still route to AnthropicProvider so the
# surrounding real thinking text surfaces — without this the
# reasoning bubble silently disappears on history rehydration.
# Pinned by registering "redacted_thinking" as a second key
# in _BLOCK_TYPE_PROVIDER_FACTORY pointing at the Anthropic
# factory; Anthropic's extractor's type=="thinking" filter
# already correctly skips the redacted block.
from turnstone.core.history_decoration import extract_reasoning_for_history
messages = [
{
"role": "assistant",
"content": "answer",
"_provider_content": [
{"type": "redacted_thinking", "data": "sealed-blob"},
{"type": "thinking", "thinking": "real thought", "signature": "s"},
{"type": "text", "text": "answer"},
],
}
]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == "real thought"
assert "_provider_content" not in messages[0]
@@ -55,6 +55,12 @@ class _FakeUI:
pass
# ChatSession callbacks (no-op for this test)
def on_turn_start(self) -> None:
pass
def on_turn_committed(self) -> None:
pass
def on_thinking_start(self) -> None:
pass
+52
View File
@@ -777,6 +777,58 @@ class TestModelAliasResolution:
assert judge._client_factory_args["api_key"] == "alias-key"
assert judge._client_factory_args["provider_name"] == "openai"
def test_unknown_alias_inherits_session_model(self):
"""``judge.model`` is alias-only. A value that doesn't resolve
through the registry inherits the session model (same path as
an empty config.model) rather than getting pinned onto the
session provider as a raw model id that legacy behavior
silently broke whenever the session provider didn't speak the
configured model id (Anthropic session, ``judge.model =
"gpt-5-mini"`` every verdict came back as ``llm_fallback``)."""
session_provider = _make_mock_provider()
session_provider.provider_name = "anthropic"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
registry = MagicMock()
registry.has_alias.return_value = False # judge.model isn't an alias
config = JudgeConfig(enabled=True, model="gpt-5-mini")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
model_registry=registry,
)
assert judge._provider is session_provider
assert judge._model == "session-default-model"
# Context window mirrors the session, not the (uncalled) caps lookup.
assert judge._judge_context_window == 100_000
def test_empty_model_inherits_session_model(self):
"""Empty ``config.model`` is the documented self-consistency path."""
session_provider = _make_mock_provider()
session_provider.provider_name = "openai"
session_client = MagicMock()
session_client.base_url = "https://session.example/v1"
session_client.api_key = "session-key"
config = JudgeConfig(enabled=True, model="")
judge = IntentJudge(
config=config,
session_provider=session_provider,
session_client=session_client,
session_model="session-default-model",
context_window=100_000,
)
assert judge._provider is session_provider
assert judge._model == "session-default-model"
def test_coordinator_tool_call_returns_llm_verdict_not_fallback(self):
"""Happy-path regression for coordinator tool calls: with a properly
resolved provider, the verdict tier must be ``llm`` the
+122 -1
View File
@@ -51,7 +51,11 @@ class TestIntentVerdictCRUD:
assert v["tier"] == "heuristic"
assert v["judge_model"] == ""
assert v["latency_ms"] == 2
assert v["user_decision"] == ""
# ``user_decision`` defaults to ``"pending"`` (not the empty
# string) so an audit reader can distinguish in-flight rows
# from pre-convention legacy rows that carry the column's
# server_default of ``""``.
assert v["user_decision"] == "pending"
assert "created" in v
def test_get_nonexistent(self, db):
@@ -114,6 +118,123 @@ class TestIntentVerdictCRUD:
assert ok is False
class TestIntentVerdictUpsert:
"""``upsert_intent_verdict`` — the LLM-tier-aware persistence path.
Backs the heuristic llm_fallback "upgrade in place" pattern.
The async judge's fallback verdicts deliberately reuse the
heuristic ``verdict_id``; a plain INSERT would collide on the
PK and the upgrade would be lost to a silently-swallowed
exception (Postgres logged ``intent_verdicts_pkey`` violations
for every fallback delivery on stable/1.5 smoke tests).
"""
def test_upsert_on_fresh_id_inserts(self, db):
"""No conflict — behaves like a regular INSERT."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
v = db.get_intent_verdict("v_001")
assert v is not None
assert v["tier"] == "heuristic"
assert v["user_decision"] == "pending"
def test_upsert_on_conflict_upgrades_tier_reasoning_judge_model(self, db):
"""On PK conflict: tier, reasoning, judge_model update — every
other field is preserved. Mirrors what the judge emits when
promoting heuristic llm_fallback."""
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="heuristic",
reasoning="initial heuristic reasoning",
judge_model="",
)
)
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="llm_fallback",
reasoning="initial heuristic reasoning (LLM judge did not return a verdict)",
judge_model="gpt-5-judge",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
# The three fields that should change.
assert v["tier"] == "llm_fallback"
assert "LLM judge did not return" in v["reasoning"]
assert v["judge_model"] == "gpt-5-judge"
def test_upsert_on_conflict_preserves_user_decision(self, db):
"""LOAD-BEARING: a manually-resolved approval (user_decision=
``"approved"``) or auto-approve-stamped row (user_decision=
``"policy"``/``"blanket"``/etc.) must NOT be clobbered back to
``"pending"`` when the late LLM-fallback verdict lands.
``IntentVerdict.to_dict()`` doesn't project user_decision, so
the upsert's defaulted ``"pending"`` would silently overwrite
the real value if user_decision were in the on-conflict
SET clause."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
ok = db.update_intent_verdict("v_001", user_decision="approved")
assert ok is True
# Simulate the late LLM-fallback delivery — same verdict_id,
# default user_decision (the IntentVerdict.to_dict() shape).
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="llm_fallback",
reasoning="extended (LLM judge did not return a verdict)",
judge_model="gpt-5-judge",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
assert v["user_decision"] == "approved" # NOT clobbered to "pending"
assert v["tier"] == "llm_fallback" # but the upgrade did land
def test_upsert_on_conflict_preserves_identity_and_carried_fields(self, db):
"""Identity columns (ws_id, call_id, func_name, func_args) and
carried-verbatim columns (intent_summary, risk_level,
confidence, recommendation, evidence, latency_ms) are
excluded from the on-conflict SET verify they aren't
changed even when the second upsert passes different values
(defensive against a future judge bug that ships divergent
carried fields)."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
db.upsert_intent_verdict(
**_make_verdict_kwargs(
# Same verdict_id (conflict trigger), divergent everything else.
ws_id="ws-different",
call_id="tc_different",
func_name="bash_v2",
func_args='{"command":"rm -rf /"}',
intent_summary="totally different summary",
risk_level="critical",
confidence=0.0,
recommendation="deny",
evidence='["dangerous"]',
latency_ms=99999,
# The three fields that DO update.
tier="llm_fallback",
reasoning="upgraded reasoning",
judge_model="judge-v2",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
# All preserved from the first upsert (identity + carried).
assert v["ws_id"] == "ws-abc"
assert v["call_id"] == "tc_001"
assert v["func_name"] == "bash"
assert v["func_args"] == '{"command":"echo hello"}'
assert v["intent_summary"] == "Echo a greeting to stdout"
assert v["risk_level"] == "low"
assert v["confidence"] == 0.85
assert v["recommendation"] == "approve"
assert v["evidence"] == '["The command only prints text."]'
assert v["latency_ms"] == 2
# Only the three updated.
assert v["tier"] == "llm_fallback"
assert v["reasoning"] == "upgraded reasoning"
assert v["judge_model"] == "judge-v2"
# ---------------------------------------------------------------------------
# Bulk insert
# ---------------------------------------------------------------------------
+211
View File
@@ -0,0 +1,211 @@
"""Integration tests for the Phase 9 admin bulk-revoke endpoint.
POST /v1/api/admin/mcp-servers/{name}/bulk-revoke clears every user's
OAuth token for a server (admin-side counterpart to the per-user
DELETE /v1/api/mcp/oauth/connections/{server_name} that shipped in
Phase 8).
Coverage:
- requires ``admin.mcp`` permission (401/403 without).
- 404 when the named server is missing.
- 400 when the server's ``auth_type`` is not ``oauth_user``.
- 200 + ``rows_deleted`` + ``consented_users_before`` on success.
- Audit row written with
``upstream_revoke_outcome="bulk_admin_no_upstream"``.
- Token rows are gone from ``mcp_user_tokens`` post-call.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.console.server import admin_mcp_bulk_revoke
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
class _InjectAdminMcp(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="admin-user",
scopes=frozenset({"approve"}),
token_source="config",
permissions=frozenset({"read", "write", "approve", "admin.mcp"}),
)
return await call_next(request)
class _InjectNoAdminMcp(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="regular-user",
scopes=frozenset({"approve"}),
token_source="jwt",
permissions=frozenset({"read", "write", "approve"}),
)
return await call_next(request)
def _build_app(storage: SQLiteBackend, *, with_admin_mcp: bool = True) -> Starlette:
mw = _InjectAdminMcp if with_admin_mcp else _InjectNoAdminMcp
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route(
"/api/admin/mcp-servers/{name}/bulk-revoke",
admin_mcp_bulk_revoke,
methods=["POST"],
),
],
),
],
middleware=[Middleware(mw)],
)
app.state.auth_storage = storage
return app
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
return SQLiteBackend(str(tmp_path / "test.db"))
def _seed_oauth_server(
backend: SQLiteBackend,
*,
name: str = "srv-oauth",
server_id: str = "srv-oauth-id",
) -> None:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="oauth_user",
)
def _seed_static_server(
backend: SQLiteBackend,
*,
name: str = "srv-static",
server_id: str = "srv-static-id",
) -> None:
backend.create_mcp_server(
server_id=server_id,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="static",
)
def _seed_user_tokens(backend: SQLiteBackend, server_name: str, users: int) -> None:
for i in range(users):
backend.create_mcp_user_token(
f"user-{i}",
server_name,
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=None,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
def test_requires_admin_mcp_permission(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage, with_admin_mcp=False))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 403
def test_404_on_missing_server(storage: SQLiteBackend) -> None:
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/never-existed/bulk-revoke")
assert resp.status_code == 404
assert resp.json() == {"error": "No such server"}
def test_400_on_static_server(storage: SQLiteBackend) -> None:
_seed_static_server(storage)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-static/bulk-revoke")
assert resp.status_code == 400
body = resp.json()
assert "oauth_user" in body["error"]
def test_400_on_invalid_server_name(storage: SQLiteBackend) -> None:
# double-underscore is reserved for the prefixed-tool-name encoding.
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/bad__name/bulk-revoke")
assert resp.status_code == 400
def test_200_on_success_with_no_consented_users(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
assert body["rows_deleted"] == 0
assert body["consented_users_before"] == 0
def test_200_clears_all_user_tokens(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_user_tokens(storage, "srv-oauth", users=3)
# Token for another server must survive the bulk-revoke.
_seed_oauth_server(storage, name="srv-other", server_id="srv-other-id")
_seed_user_tokens(storage, "srv-other", users=2)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
body = resp.json()
assert body["status"] == "ok"
assert body["rows_deleted"] == 3
assert body["consented_users_before"] == 3
# Target server's tokens are gone; bystander's tokens survive.
assert storage.count_mcp_consented_users_by_server("srv-oauth") == 0
assert storage.count_mcp_consented_users_by_server("srv-other") == 2
def test_audits_with_bulk_admin_no_upstream(storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_user_tokens(storage, "srv-oauth", users=2)
client = TestClient(_build_app(storage))
resp = client.post("/v1/api/admin/mcp-servers/srv-oauth/bulk-revoke")
assert resp.status_code == 200
# Pull the most-recent audit row for the bulk_revoked action and
# verify it carries the deferral marker.
events = storage.list_audit_events(limit=10)
bulk_rows = [e for e in events if e.get("action") == "mcp_server.oauth.bulk_revoked"]
assert len(bulk_rows) == 1
detail = bulk_rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("upstream_revoke_outcome") == "bulk_admin_no_upstream"
assert detail.get("rows_deleted") == 2
assert detail.get("consented_users_before") == 2
assert detail.get("name") == "srv-oauth"
+146 -1
View File
@@ -788,7 +788,11 @@ class TestSessionIntegration:
assert call_id == "call_789"
assert output == "result text"
mock_mcp.call_tool_sync.assert_called_once_with(
"mcp__test__search", {"query": "hello"}, user_id=None, timeout=30
"mcp__test__search",
{"query": "hello"},
user_id=None,
timeout=30,
is_interactive_for_consent=True,
)
def test_exec_mcp_tool_error(self, tmp_db):
@@ -1056,6 +1060,147 @@ class TestRefreshServer:
asyncio.run(_run())
class TestLastRefreshTracking:
"""Phase 9 admin status pill — ``_last_refresh`` is written on every
refresh path so the admin UI reflects manual-refresh AND auto-
reconnect outcomes uniformly. This test class pins the contract.
"""
@staticmethod
def _seed_minimal(mgr: MCPClientManager, name: str = "srv") -> MagicMock:
mock_session = MagicMock()
mock_session.list_tools = AsyncMock(return_value=MagicMock(tools=[]))
mock_session.list_resources = AsyncMock(return_value=MagicMock(resources=[]))
mock_session.list_resource_templates = AsyncMock(
return_value=MagicMock(resourceTemplates=[])
)
mock_session.list_prompts = AsyncMock(return_value=MagicMock(prompts=[]))
_seed_static_state(
mgr,
name,
session=mock_session,
tools=[],
supports_resources=True,
supports_prompts=True,
)
return mock_session
def test_last_refresh_written_on_success(self) -> None:
async def _run() -> None:
mgr = MCPClientManager({})
self._seed_minimal(mgr)
assert "srv" not in mgr._last_refresh
await mgr._refresh_server("srv")
entry = mgr._last_refresh.get("srv")
assert entry is not None
ts, outcome = entry
assert outcome == "ok"
assert isinstance(ts, float) and ts > 0
asyncio.run(_run())
def test_last_refresh_written_on_tool_refresh_failure(self) -> None:
"""When ``_refresh_server_tools`` raises, the outcome reflects
the exception class and the exception still propagates."""
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = self._seed_minimal(mgr)
mock_session.list_tools = AsyncMock(side_effect=RuntimeError("upstream down"))
with pytest.raises(RuntimeError, match="upstream down"):
await mgr._refresh_server("srv")
entry = mgr._last_refresh.get("srv")
assert entry is not None
_, outcome = entry
assert outcome == "error:RuntimeError"
asyncio.run(_run())
def test_last_refresh_records_first_exception_when_multiple_fail(
self,
) -> None:
"""``return_exceptions=True`` lets sibling tasks complete; the
outcome reflects the FIRST exception encountered."""
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = self._seed_minimal(mgr)
# Tools succeeds; resources raises first (gather preserves
# argument order in its results list, so resources is the
# first failure regardless of which awaitable finished first
# in wall-clock terms).
mock_session.list_resources = AsyncMock(side_effect=ValueError("res boom"))
mock_session.list_prompts = AsyncMock(side_effect=KeyError("prompts boom"))
with pytest.raises((ValueError, KeyError)):
await mgr._refresh_server("srv")
entry = mgr._last_refresh.get("srv")
assert entry is not None
_, outcome = entry
# Either of the two failing tasks could be "first" in
# gather's results list ordering — the order is positional
# so resources (arg #2) comes before prompts (arg #3).
assert outcome == "error:ValueError"
asyncio.run(_run())
def test_refresh_all_overwrites_stale_ok_on_reconnect_failure(
self,
) -> None:
"""The chokepoint bug-1 fix: a prior successful refresh's ``'ok'``
entry MUST be overwritten when a subsequent reconnect fails
otherwise the admin pill shows misleading "ok" while the server
is in fact broken."""
async def _run() -> None:
mgr = MCPClientManager({})
# Server is configured but has no live session — _refresh_all
# routes to the reconnect branch.
mgr._server_configs["srv"] = {"type": "stdio", "command": "x"}
# Pre-seed a stale "ok" from an earlier successful refresh.
mgr._last_refresh["srv"] = (1000.0, "ok")
async def _raise(*_a: object, **_kw: object) -> None:
raise ConnectionError("reconnect failed")
mgr._connect_one = _raise # type: ignore[assignment]
await mgr._refresh_all("srv")
entry = mgr._last_refresh.get("srv")
assert entry is not None
ts, outcome = entry
# Outcome reflects the new failure, not the stale ok.
assert outcome == "error:ConnectionError"
assert ts > 1000.0
asyncio.run(_run())
def test_get_server_status_surfaces_last_refresh_fields(self) -> None:
"""``get_server_status`` surfaces ``last_refresh_at`` and
``last_refresh_outcome`` for the admin pill null when no
refresh has occurred yet, populated after one."""
mgr = MCPClientManager({})
mgr._server_configs["srv"] = {"type": "stdio", "command": "x"}
# No refresh yet — fields must be present and null so the JS
# renderer can branch on absence cleanly.
status = mgr.get_server_status("srv")
assert status["last_refresh_at"] is None
assert status["last_refresh_outcome"] is None
# Populate the tuple directly and re-read.
mgr._last_refresh["srv"] = (12345.5, "ok")
status = mgr.get_server_status("srv")
assert status["last_refresh_at"] == 12345.5
assert status["last_refresh_outcome"] == "ok"
class TestListeners:
def test_add_and_notify(self):
mgr = MCPClientManager({})
+91
View File
@@ -612,6 +612,97 @@ class TestCallback:
assert plain is not None
assert plain["refresh_token"] is None
def test_callback_clears_pending_consent_on_success(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
"""Successful callback must drop any ``mcp_pending_consent`` rows
for the just-consented ``(user, server)`` (Phase 9 lifecycle
contract). Regression guard for the dashboard-stays-stale-after-
consent invariant.
"""
_seed_oauth_user_server(storage)
self._seed_pending(storage)
# Seed a deferred-consent record that a prior non-interactive run
# would have left behind. Plus a cross-tenant record that must
# NOT be touched.
storage.upsert_mcp_pending_consent(
user_id="user-1",
server_name="srv-oauth",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id="ws-1",
last_tool_call_id="tool-1",
now_iso="2026-05-11T12:00:00",
)
storage.upsert_mcp_pending_consent(
user_id="other-user",
server_name="srv-oauth",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T12:00:00",
)
token_store = _make_token_store(storage)
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
http_client_mock.post.return_value = _mk_response(
200,
{"access_token": "opaque-aaa", "expires_in": 3600},
)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
with _public_addr_patch():
resp = client.get(
"/v1/api/mcp/oauth/callback?code=c&state=valid-state",
follow_redirects=False,
)
assert resp.status_code == 302
# Callback completed → user-1's deferred-consent row was cleared.
assert storage.list_mcp_pending_consent_by_user("user-1") == []
# Cross-tenant row survives — clear is per-(user, server).
other = storage.list_mcp_pending_consent_by_user("other-user")
assert len(other) == 1
assert other[0]["server_name"] == "srv-oauth"
def test_callback_storage_failure_does_not_block_redirect(
self, storage: SQLiteBackend, http_client_mock: MagicMock
) -> None:
"""If the post-persist ``delete_mcp_pending_consent`` raises, the
callback's redirect still completes (best-effort contract). The
stale badge is preferred over a broken consent flow.
"""
_seed_oauth_user_server(storage)
self._seed_pending(storage)
token_store = _make_token_store(storage)
http_client_mock.get.return_value = _mk_response(200, _good_as_metadata_doc())
http_client_mock.post.return_value = _mk_response(
200,
{"access_token": "opaque-aaa", "expires_in": 3600},
)
app = _build_app(storage=storage, http_client=http_client_mock, token_store=token_store)
client = TestClient(app, raise_server_exceptions=False)
original_delete = storage.delete_mcp_pending_consent
def _raise(*_a: Any, **_kw: Any) -> bool:
raise RuntimeError("storage offline")
storage.delete_mcp_pending_consent = _raise # type: ignore[method-assign]
try:
with _public_addr_patch():
resp = client.get(
"/v1/api/mcp/oauth/callback?code=c&state=valid-state",
follow_redirects=False,
)
finally:
storage.delete_mcp_pending_consent = original_delete # type: ignore[method-assign]
assert resp.status_code == 302
# Token persistence still succeeded — the user-visible contract.
plain = token_store.get_user_token("user-1", "srv-oauth")
assert plain is not None
# ---------------------------------------------------------------------------
# 503 paths when mcp_token_store is None
+304
View File
@@ -0,0 +1,304 @@
"""Boundary tests for the Phase 9 pending-consent write path.
Drives ``MCPClientManager._dispatch_pool_sync`` (and the helper it
calls, ``_record_pending_consent_best_effort``) and asserts that
deferred-consent records reach storage only on non-interactive callers.
Per ``feedback_tests_through_boundaries.md``, at least one test must
drive the real sync dispatcher real ``_is_structured_error``
real ``_record_pending_consent_best_effort`` plumb-through; the
``_helpers`` unit tests below cover the classifier in isolation, but
the end-to-end test is the structural gate that catches
plumb-through regressions.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import threading
from typing import Any
from unittest.mock import patch
import pytest
from tests.conftest import make_mcp_token_cipher
from turnstone.core.mcp_client import (
_PENDING_CONSENT_PERSIST_CODES,
MCPClientManager,
_parse_pending_consent_envelope,
)
from turnstone.core.mcp_crypto import MCPTokenStore
from turnstone.core.mcp_oauth import TokenLookupResult
# ---------------------------------------------------------------------------
# Helper-level unit tests (cheap, no event loop)
# ---------------------------------------------------------------------------
class TestParseEnvelope:
def test_consent_required_no_scopes(self) -> None:
env = json.dumps({"error": {"code": "mcp_consent_required", "server": "x", "detail": "d"}})
assert _parse_pending_consent_envelope(env) == ("mcp_consent_required", None)
def test_insufficient_scope_with_scopes(self) -> None:
env = json.dumps(
{
"error": {
"code": "mcp_insufficient_scope",
"server": "x",
"detail": "d",
"scopes_required": ["read", "write"],
}
}
)
assert _parse_pending_consent_envelope(env) == (
"mcp_insufficient_scope",
["read", "write"],
)
def test_operator_codes_filtered(self) -> None:
# Key-unknown / url-insecure / *_forbidden are operator-actionable,
# NOT user-consent-shaped. They must not produce pending-consent
# rows, regardless of whether the caller is interactive.
for code in (
"mcp_token_undecryptable_key_unknown",
"mcp_oauth_url_insecure",
"mcp_tool_call_forbidden",
"mcp_resource_read_forbidden",
"mcp_prompt_get_forbidden",
):
env = json.dumps({"error": {"code": code, "server": "x", "detail": "d"}})
assert _parse_pending_consent_envelope(env) is None, code
def test_malformed_json_returns_none(self) -> None:
assert _parse_pending_consent_envelope("not json") is None
assert _parse_pending_consent_envelope("") is None
def test_persist_codes_set_is_expected(self) -> None:
# Pin the contract — adding a new persistable code here is a
# deliberate design decision and should require a test update.
assert {
"mcp_consent_required",
"mcp_insufficient_scope",
} == _PENDING_CONSENT_PERSIST_CODES
# ---------------------------------------------------------------------------
# End-to-end plumb-through (drives _dispatch_pool_sync)
# ---------------------------------------------------------------------------
def _seed_oauth_server(backend: Any, *, name: str = "pool-srv") -> None:
backend.create_mcp_server(
server_id="srv-" + name,
name=name,
transport="streamable-http",
command="",
args="[]",
url="https://example.com/mcp",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-" + name, auth_type="oauth_user")
@pytest.fixture
def running_loop_mgr():
cfg: dict[str, Any] = {}
mgr = MCPClientManager(cfg)
loop = asyncio.new_event_loop()
thread = threading.Thread(target=loop.run_forever, daemon=True, name="phase9-test-loop")
thread.start()
mgr._loop = loop
try:
yield mgr, loop, thread
finally:
async def _drain(m: MCPClientManager) -> None:
task = m._user_pool_eviction_task
if task is not None:
task.cancel()
with contextlib.suppress(BaseException):
await task
m._user_pool_eviction_task = None
with contextlib.suppress(Exception):
asyncio.run_coroutine_threadsafe(_drain(mgr), loop).result(timeout=2)
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=2)
def _wire_mgr(mgr: MCPClientManager, backend: Any) -> None:
cipher = make_mcp_token_cipher()
from types import SimpleNamespace
from unittest.mock import MagicMock
app_state = SimpleNamespace(
auth_storage=backend,
mcp_token_store=MCPTokenStore(backend, cipher, node_id="test"),
mcp_oauth_http_client=MagicMock(),
mcp_oauth_refresh_locks={},
mcp_oauth_metadata_cache={},
)
mgr.set_storage(backend)
mgr.set_app_state(app_state)
def test_dispatch_persists_pending_for_non_interactive_caller(
running_loop_mgr: Any, backend: Any
) -> None:
"""Non-interactive caller hits ``mcp_consent_required`` → a
``mcp_pending_consent`` row appears for ``(user_id, server_name)``."""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
# Structured error envelope surfaces as RuntimeError to the caller.
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_consent_required"
# Persistent row written for the dashboard badge.
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
assert r["user_id"] == "user-a"
assert r["server_name"] == "pool-srv"
assert r["error_code"] == "mcp_consent_required"
assert r["occurrence_count"] == 1
def test_dispatch_does_not_persist_for_interactive_caller(
running_loop_mgr: Any, backend: Any
) -> None:
"""Interactive caller hits the same error path → NO row written.
Interactive (WEB / CLI) sessions surface the consent prompt in-flight
via the Phase 8 SSE renderer; persisting would just produce
immediately-stale dashboard badges.
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError),
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=True,
)
assert backend.list_mcp_pending_consent_by_user("user-a") == []
def test_dispatch_returns_envelope_unchanged_on_storage_failure(
running_loop_mgr: Any, backend: Any
) -> None:
"""When ``upsert_mcp_pending_consent`` raises, the agent-observable
contract is unchanged: the structured-error ``RuntimeError`` still
surfaces with the original ``mcp_consent_required`` code. The doc-
string promises best-effort persistence; this test pins that
promise so a regression that propagates the storage exception would
fail visibly.
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _missing_token(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="missing")
original_upsert = backend.upsert_mcp_pending_consent
def _raise(*_a: Any, **_kw: Any) -> None:
raise RuntimeError("storage offline")
backend.upsert_mcp_pending_consent = _raise # type: ignore[method-assign]
try:
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_missing_token,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
finally:
backend.upsert_mcp_pending_consent = original_upsert # type: ignore[method-assign]
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_consent_required"
def test_dispatch_does_not_persist_for_operator_actionable_code(
running_loop_mgr: Any, backend: Any
) -> None:
"""Decrypt-failure → operator-actionable; even non-interactive callers
must NOT produce a user-facing pending-consent record (the user can't
resolve this by re-consenting).
"""
mgr, _loop, _ = running_loop_mgr
_seed_oauth_server(backend)
_wire_mgr(mgr, backend)
async def _decrypt_failure(**kwargs: Any) -> TokenLookupResult:
return TokenLookupResult(kind="decrypt_failure")
with (
patch(
"turnstone.core.mcp_client.get_user_access_token_classified",
side_effect=_decrypt_failure,
),
pytest.raises(RuntimeError) as exc_info,
):
mgr.call_tool_sync(
"mcp__pool-srv__echo",
{"payload": "hi"},
user_id="user-a",
timeout=10,
is_interactive_for_consent=False,
)
payload = json.loads(str(exc_info.value)).get("error", {})
assert payload.get("code") == "mcp_token_undecryptable_key_unknown"
# The operator-actionable code does NOT produce a pending-consent row.
assert backend.list_mcp_pending_consent_by_user("user-a") == []
+259
View File
@@ -0,0 +1,259 @@
"""HTTP tests for the Phase 9 pending-consent endpoints.
Covers:
- ``GET /v1/api/mcp/oauth/pending`` (install gate + read path)
- ``DELETE /v1/api/mcp/oauth/pending/{server_name}`` (single clear)
- ``DELETE /v1/api/mcp/oauth/pending`` (bulk clear)
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.core.auth import AuthResult
from turnstone.core.mcp_oauth import (
handle_mcp_oauth_clear_all_pending,
handle_mcp_oauth_clear_pending,
handle_mcp_oauth_list_pending,
)
from turnstone.core.storage._sqlite import SQLiteBackend
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
class _InjectAuthMiddleware(BaseHTTPMiddleware):
"""Stamp a fixed authenticated user on every request."""
def __init__(self, app: Any, user_id: str = "user-1") -> None:
super().__init__(app)
self._user_id = user_id
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id=self._user_id,
scopes=frozenset({"write"}),
token_source="config",
permissions=frozenset({"read", "write"}),
)
return await call_next(request)
def _build_app(storage: SQLiteBackend, *, user_id: str = "user-1") -> Starlette:
class _Mw(_InjectAuthMiddleware):
def __init__(self, app: Any) -> None:
super().__init__(app, user_id=user_id)
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/mcp/oauth/pending", handle_mcp_oauth_list_pending),
Route(
"/api/mcp/oauth/pending",
handle_mcp_oauth_clear_all_pending,
methods=["DELETE"],
),
Route(
"/api/mcp/oauth/pending/{server_name}",
handle_mcp_oauth_clear_pending,
methods=["DELETE"],
),
],
),
],
middleware=[Middleware(_Mw)],
)
app.state.auth_storage = storage
return app
@pytest.fixture
def storage(tmp_path: Any) -> SQLiteBackend:
backend = SQLiteBackend(str(tmp_path / "test.db"))
backend.create_user("user-1", "user1", "User One", "hash")
backend.create_user("user-2", "user2", "User Two", "hash")
return backend
def _seed_oauth_server(backend: SQLiteBackend, *, name: str = "srv-x") -> None:
backend.create_mcp_server(
server_id="srv-id-" + name,
name=name,
transport="streamable-http",
url="https://example.com/mcp",
auth_type="oauth_user",
)
def _seed_pending(
backend: SQLiteBackend,
*,
user_id: str = "user-1",
server_name: str = "srv-x",
error_code: str = "mcp_consent_required",
now_iso: str = "2026-05-11T12:00:00",
) -> None:
backend.upsert_mcp_pending_consent(
user_id=user_id,
server_name=server_name,
error_code=error_code,
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=now_iso,
)
class TestListPending:
def test_install_gate_short_circuits_on_no_oauth_servers(self, storage: SQLiteBackend) -> None:
# Seed a pending row but NO oauth_user MCP server — the gate
# must short-circuit to {pending: 0} regardless.
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"pending": 0, "servers": []}
def test_lists_pending_records_for_authenticated_user(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
body = resp.json()
assert body["pending"] == 1
assert len(body["servers"]) == 1
assert body["servers"][0]["server_name"] == "srv-x"
assert body["servers"][0]["error_code"] == "mcp_consent_required"
def test_does_not_leak_cross_user_records(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage, user_id="user-2")
client = TestClient(_build_app(storage, user_id="user-1"))
resp = client.get("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"pending": 0, "servers": []}
class TestClearPending:
def test_delete_single(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
assert storage.list_mcp_pending_consent_by_user("user-1") == []
def test_delete_missing_still_returns_204(self, storage: SQLiteBackend) -> None:
# Idempotent — must not leak cross-user existence info via 404.
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
assert resp.status_code == 204
def test_delete_does_not_touch_cross_user_rows(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage, user_id="user-1")
_seed_pending(storage, user_id="user-2")
client = TestClient(_build_app(storage, user_id="user-1"))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
# User-2's row survives.
assert len(storage.list_mcp_pending_consent_by_user("user-2")) == 1
class TestAuditTrail:
def test_single_dismiss_audits(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_pending(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/srv-x")
assert resp.status_code == 204
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "single"
assert detail.get("cleared") == 1
def test_single_dismiss_audits_even_when_no_row_existed(self, storage: SQLiteBackend) -> None:
# Cross-tenant non-observability requires a 204 in the never-existed
# case — the audit row distinguishes a real dismiss from a stuffed
# attempt by recording ``cleared=0``.
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending/never-existed")
assert resp.status_code == 204
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "single"
assert detail.get("cleared") == 0
def test_bulk_dismiss_audits(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_oauth_server(storage, name="srv-y")
_seed_pending(storage, server_name="srv-x")
_seed_pending(storage, server_name="srv-y")
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 2}
events = storage.list_audit_events(limit=10)
rows = [
e for e in events if e.get("action") == "mcp_server.oauth.pending_consent_dismissed"
]
assert len(rows) == 1
detail = rows[0].get("detail")
if isinstance(detail, str):
import json as _json
detail = _json.loads(detail)
assert detail.get("mode") == "bulk"
assert detail.get("cleared") == 2
class TestClearAllPending:
def test_bulk_clear(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
_seed_oauth_server(storage, name="srv-y")
_seed_pending(storage, server_name="srv-x")
_seed_pending(storage, server_name="srv-y")
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 2}
assert storage.list_mcp_pending_consent_by_user("user-1") == []
def test_bulk_clear_zero_when_empty(self, storage: SQLiteBackend) -> None:
_seed_oauth_server(storage)
client = TestClient(_build_app(storage))
resp = client.delete("/v1/api/mcp/oauth/pending")
assert resp.status_code == 200
assert resp.json() == {"cleared": 0}
+263
View File
@@ -0,0 +1,263 @@
"""Storage CRUD tests for the Phase 9 ``mcp_pending_consent`` table.
Validates protocol additions backing the dashboard pending-consent badge:
- ``upsert_mcp_pending_consent`` insert + on-conflict refresh
- ``list_mcp_pending_consent_by_user`` read path
- ``delete_mcp_pending_consent`` single-row clear
- ``delete_all_mcp_pending_consent_by_user`` bulk clear
- ``count_mcp_consented_users_by_server`` admin status pill
- ``any_oauth_user_mcp_servers`` install-level gate
"""
from __future__ import annotations
def _iso(ts: str = "2026-05-11T12:00:00") -> str:
return ts
class TestUpsertAndList:
def test_insert_round_trip(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required="read write",
last_ws_id="ws-1",
last_tool_call_id="tool-1",
now_iso=_iso(),
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
assert r["user_id"] == "user-a"
assert r["server_name"] == "srv-x"
assert r["error_code"] == "mcp_consent_required"
assert r["scopes_required"] == "read write"
assert r["last_ws_id"] == "ws-1"
assert r["last_tool_call_id"] == "tool-1"
assert r["occurrence_count"] == 1
assert r["first_seen_at"] == r["last_seen_at"]
def test_upsert_bumps_count_and_refreshes_recency(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T12:00:00",
)
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_insufficient_scope",
scopes_required="read",
last_ws_id="ws-2",
last_tool_call_id="tool-2",
now_iso="2026-05-11T13:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert len(rows) == 1
r = rows[0]
# Recency fields refreshed to the second call's values; count bumped.
assert r["occurrence_count"] == 2
assert r["error_code"] == "mcp_insufficient_scope"
assert r["scopes_required"] == "read"
assert r["last_ws_id"] == "ws-2"
assert r["last_tool_call_id"] == "tool-2"
assert r["last_seen_at"] == "2026-05-11T13:00:00"
# first_seen_at preserved — that's the load-bearing audit value.
assert r["first_seen_at"] == "2026-05-11T12:00:00"
def test_list_orders_by_last_seen_desc(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-old",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T10:00:00",
)
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-new",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso="2026-05-11T11:00:00",
)
rows = backend.list_mcp_pending_consent_by_user("user-a")
assert [r["server_name"] for r in rows] == ["srv-new", "srv-old"]
def test_per_user_isolation(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.list_mcp_pending_consent_by_user("user-b") == []
class TestDelete:
def test_delete_single(self, backend) -> None:
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name="srv-x",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is True
assert backend.list_mcp_pending_consent_by_user("user-a") == []
# Second delete returns False (no row).
assert backend.delete_mcp_pending_consent("user-a", "srv-x") is False
def test_delete_missing_returns_false(self, backend) -> None:
assert backend.delete_mcp_pending_consent("never", "missing") is False
def test_delete_all_by_user(self, backend) -> None:
for name in ("srv-a", "srv-b", "srv-c"):
backend.upsert_mcp_pending_consent(
user_id="user-a",
server_name=name,
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
# Cross-user row that must NOT be touched.
backend.upsert_mcp_pending_consent(
user_id="user-b",
server_name="srv-z",
error_code="mcp_consent_required",
scopes_required=None,
last_ws_id=None,
last_tool_call_id=None,
now_iso=_iso(),
)
assert backend.delete_all_mcp_pending_consent_by_user("user-a") == 3
assert backend.list_mcp_pending_consent_by_user("user-a") == []
assert len(backend.list_mcp_pending_consent_by_user("user-b")) == 1
class TestCountConsentedUsersByServer:
def _seed_server(self, backend, name: str = "srv-x") -> None:
backend.create_mcp_server(
server_id="srv-id-" + name,
name=name,
transport="streamable-http",
command="",
args="[]",
url="https://example.com/mcp",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-id-" + name, auth_type="oauth_user")
def test_counts_distinct_non_expired_users(self, backend) -> None:
self._seed_server(backend)
future = "2099-01-01T00:00:00"
backend.create_mcp_user_token(
"alice",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=future,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
backend.create_mcp_user_token(
"bob",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=None, # null treated as non-expired
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
# Different server — must not count.
self._seed_server(backend, name="srv-y")
backend.create_mcp_user_token(
"carol",
"srv-y",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at=future,
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
assert backend.count_mcp_consented_users_by_server("srv-x") == 2
assert backend.count_mcp_consented_users_by_server("srv-y") == 1
def test_excludes_expired(self, backend) -> None:
self._seed_server(backend)
backend.create_mcp_user_token(
"alice",
"srv-x",
access_token_ct=b"ct",
refresh_token_ct=None,
expires_at="2020-01-01T00:00:00", # well in the past
scopes=None,
as_issuer="https://as.example.com",
audience="https://example.com/mcp",
)
assert backend.count_mcp_consented_users_by_server("srv-x") == 0
def test_zero_when_no_rows(self, backend) -> None:
assert backend.count_mcp_consented_users_by_server("missing") == 0
class TestInstallGate:
def test_any_oauth_user_returns_false_on_empty(self, backend) -> None:
assert backend.any_oauth_user_mcp_servers() is False
def test_any_oauth_user_ignores_static_rows(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-1",
name="static-only",
transport="streamable-http",
command="",
args="[]",
url="https://example.com",
headers='{"Authorization": "Bearer x"}',
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
assert backend.any_oauth_user_mcp_servers() is False
def test_any_oauth_user_returns_true_when_one_exists(self, backend) -> None:
backend.create_mcp_server(
server_id="srv-2",
name="oauth-srv",
transport="streamable-http",
command="",
args="[]",
url="https://example.com",
headers="{}",
env="{}",
auto_approve=False,
enabled=True,
created_by="admin",
)
backend.update_mcp_server("srv-2", auth_type="oauth_user")
assert backend.any_oauth_user_mcp_servers() is True
+70
View File
@@ -212,3 +212,73 @@ class TestModelDefinitionStorage:
m = db.get_model_definition(did)
assert m is not None
assert m["temperature"] is None
def test_reasoning_flags_default(self, db: SQLiteBackend) -> None:
"""surface_persisted_reasoning defaults True; replay_reasoning_to_model defaults False."""
did = _make_id()
db.create_model_definition(definition_id=did, alias="reason-default", model="gpt-5")
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is True
assert m["replay_reasoning_to_model"] is False
def test_create_with_explicit_reasoning_flags(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(
definition_id=did,
alias="reason-explicit",
model="claude-opus-4-7",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is False
assert m["replay_reasoning_to_model"] is True
# Same values must round-trip via the alias lookup too.
m_alias = db.get_model_definition_by_alias("reason-explicit")
assert m_alias is not None
assert m_alias["surface_persisted_reasoning"] is False
assert m_alias["replay_reasoning_to_model"] is True
def test_update_surface_persisted_reasoning(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(definition_id=did, alias="upd-persist", model="gpt-5")
ok = db.update_model_definition(did, surface_persisted_reasoning=False)
assert ok is True
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is False
assert m["replay_reasoning_to_model"] is False # untouched
def test_update_replay_reasoning_to_model(self, db: SQLiteBackend) -> None:
did = _make_id()
db.create_model_definition(definition_id=did, alias="upd-replay", model="gpt-5")
ok = db.update_model_definition(did, replay_reasoning_to_model=True)
assert ok is True
m = db.get_model_definition(did)
assert m is not None
assert m["surface_persisted_reasoning"] is True # untouched
assert m["replay_reasoning_to_model"] is True
def test_list_returns_reasoning_flags(self, db: SQLiteBackend) -> None:
db.create_model_definition(
definition_id=_make_id(),
alias="list-a",
model="gpt-5",
surface_persisted_reasoning=True,
replay_reasoning_to_model=False,
)
db.create_model_definition(
definition_id=_make_id(),
alias="list-b",
model="claude-opus-4-7",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
models = db.list_model_definitions()
by_alias = {m["alias"]: m for m in models}
assert by_alias["list-a"]["surface_persisted_reasoning"] is True
assert by_alias["list-a"]["replay_reasoning_to_model"] is False
assert by_alias["list-b"]["surface_persisted_reasoning"] is False
assert by_alias["list-b"]["replay_reasoning_to_model"] is True
+133 -4
View File
@@ -76,6 +76,23 @@ class TestModelConfig:
assert cfg.temperature == 0.0
assert cfg.temperature is not None
def test_reasoning_flags_default(self) -> None:
cfg = ModelConfig(alias="x", base_url="x", api_key="x", model="x")
assert cfg.surface_persisted_reasoning is True
assert cfg.replay_reasoning_to_model is False
def test_reasoning_flags_set(self) -> None:
cfg = ModelConfig(
alias="x",
base_url="x",
api_key="x",
model="x",
surface_persisted_reasoning=False,
replay_reasoning_to_model=True,
)
assert cfg.surface_persisted_reasoning is False
assert cfg.replay_reasoning_to_model is True
# ---------------------------------------------------------------------------
# ModelRegistry
@@ -314,7 +331,11 @@ class TestLoadModelRegistry:
api_key="dummy",
model="local-model",
)
assert reg.count == 2 # "openai" + "default"
# The CLI ``"default"`` shim is suppressed once ``[models.*]``
# populates configs — only the explicit alias survives.
assert reg.count == 1
assert reg.has_alias("openai")
assert not reg.has_alias("default")
assert reg.default == "openai"
_, model, _ = reg.resolve()
assert model == "gpt-4o"
@@ -545,7 +566,12 @@ class TestLoadModelRegistryWithDB:
assert cfg.source == "config"
def test_db_only_models_coexist(self) -> None:
"""DB models coexist alongside config.toml models."""
"""DB models coexist alongside config.toml models.
The CLI ``"default"`` shim is suppressed when DB / config models
already populate the registry see
``test_cli_default_shim_skipped_when_db_models_present``.
"""
storage = _MockStorage(
[
{
@@ -569,7 +595,7 @@ class TestLoadModelRegistryWithDB:
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
assert reg.has_alias("db-only")
assert reg.has_alias("config-only")
assert reg.has_alias("default")
assert not reg.has_alias("default")
assert reg.get_config("db-only").source == "db"
assert reg.get_config("config-only").source == "config"
@@ -589,10 +615,12 @@ class TestLoadModelRegistryWithDB:
}
]
)
# The CLI default shim is suppressed when the DB row populates
# configs, so only the DB-sourced alias exists here.
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
assert reg.get_config("from-db").source == "db"
assert reg.get_config("default").source == ""
assert not reg.has_alias("default")
def test_disabled_db_models_excluded(self) -> None:
"""Disabled DB models are not loaded."""
@@ -686,6 +714,53 @@ class TestLoadModelRegistryWithDB:
assert cfg.max_tokens is None
assert cfg.reasoning_effort is None
def test_db_reasoning_flags_loaded(self) -> None:
"""Per-model reasoning flags from DB are carried in ModelConfig."""
storage = _MockStorage(
[
{
"alias": "anth-thinking",
"model": "claude-opus-4-7",
"provider": "anthropic",
"base_url": "",
"api_key": "sk-anth",
"context_window": 200000,
"capabilities": "{}",
"enabled": True,
"surface_persisted_reasoning": False,
"replay_reasoning_to_model": True,
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
cfg = reg.get_config("anth-thinking")
assert cfg.surface_persisted_reasoning is False
assert cfg.replay_reasoning_to_model is True
def test_db_reasoning_flags_default_when_absent(self) -> None:
"""Pre-052 rows without the columns degrade to dataclass defaults."""
storage = _MockStorage(
[
{
"alias": "legacy-row",
"model": "gpt-5",
"provider": "openai",
"base_url": "",
"api_key": "",
"context_window": 32768,
"capabilities": "{}",
"enabled": True,
# surface_persisted_reasoning + replay_reasoning_to_model intentionally absent
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "x", storage=storage)
cfg = reg.get_config("legacy-row")
assert cfg.surface_persisted_reasoning is True
assert cfg.replay_reasoning_to_model is False
def test_db_default_alias_not_clobbered(self) -> None:
"""DB model with alias='default' is not overwritten by CLI args."""
storage = _MockStorage(
@@ -869,6 +944,8 @@ class _FakeUI:
self.infos: list[str] = []
self.errors: list[str] = []
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -1609,6 +1686,58 @@ class TestLoadModelRegistryDBOnly:
reg = load_model_registry(model="", storage=storage)
assert not reg.has_alias("default")
def test_cli_default_shim_skipped_when_db_models_present(self) -> None:
"""An auto-detected ``--model`` does NOT synthesise a ``default``
alias when the DB already contributes models.
Regression for the silent bypass of ``model.task_alias`` /
``model.plan_alias``: a synthesised ``default`` aliased to whatever
``--base-url`` was at boot leaks into the LLM-visible alias list,
and the LLM picks it for ``task_agent(model="default")`` which
then routes around the operator-configured per-role default.
"""
storage = _MockStorage(
[
{
"alias": "gh200",
"model": "deepseek-ai/DeepSeek-V4-Flash",
"provider": "openai",
"base_url": "http://gh200:8000/v1",
"api_key": "sk-gh200",
"context_window": 1048576,
"capabilities": "{}",
"enabled": True,
}
]
)
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry(
base_url="http://flatspark:8000/v1",
api_key="sk-flatspark",
model="qwen3.6-35B-A3B", # populated by ``detect_model``
storage=storage,
)
assert reg.has_alias("gh200")
assert not reg.has_alias("default")
def test_cli_default_shim_skipped_when_config_models_present(self) -> None:
"""Same shim suppression when only ``[models.*]`` populates configs."""
fake_cfg: dict[str, Any] = {
"models": {"local": {"model": "qwen3-32b"}},
}
with patch("turnstone.core.model_registry.load_config", return_value=fake_cfg):
reg = load_model_registry("http://x/v1", "x", "fallback-model")
assert reg.has_alias("local")
assert not reg.has_alias("default")
def test_cli_default_shim_still_fires_when_registry_empty(self) -> None:
"""Single-model CLI mode (no DB, no config.toml [models.*]) keeps
the back-compat ``default`` alias."""
with patch("turnstone.core.model_registry.load_config", return_value={}):
reg = load_model_registry("http://x/v1", "x", "lone-model")
assert reg.has_alias("default")
assert reg.get_config("default").model == "lone-model"
# ---------------------------------------------------------------------------
# server._effective_routing / _apply_routing_overrides
+1
View File
@@ -197,6 +197,7 @@ _EXPECTED_AFFECTING_KEYS = frozenset(
"coordinator.model_alias",
"coordinator.reasoning_effort",
"judge.model",
"channels.default_model_alias",
}
)
+400
View File
@@ -0,0 +1,400 @@
"""Tests for the console-side ``NotifyDispatcher``.
Exercises the dispatcher against the SQLite synthetic-sweep path so the
suite runs without a Postgres dependency. The PG path is shaped the
same way (same handler invocation semantics) the only difference is
the underlying stream's wake-up source, which is covered separately in
``test_storage_notify.py::TestPostgresNotify``.
"""
from __future__ import annotations
import threading
import time
import pytest
@pytest.fixture
def dispatcher_factory(storage):
"""Yield a factory that constructs + tracks dispatchers for teardown."""
from turnstone.console.notify_dispatcher import NotifyDispatcher
created: list[NotifyDispatcher] = []
def _make(*, channels: list[str]) -> NotifyDispatcher:
d = NotifyDispatcher(storage, channels=channels)
created.append(d)
return d
yield _make
for d in created:
d.stop(timeout=2.0)
def _wait_for(predicate, deadline_sec: float = 3.0) -> bool:
"""Poll ``predicate`` until True or timeout. Returns bool."""
deadline = time.monotonic() + deadline_sec
while time.monotonic() < deadline:
if predicate():
return True
time.sleep(0.02)
return False
def _start_ready(d, *, timeout: float = 5.0) -> None:
"""``d.start()`` + assert the listener is actually listening.
Closes the start-vs-notify race for backends where ``storage.listen``
blocks on the network (Postgres ``LISTEN`` over a fresh psycopg
connection): without the sync, a same-thread ``storage.notify`` can
fire before the LISTEN registers and the notification is lost.
"""
d.start()
if not d.wait_until_ready(timeout=timeout):
msg = f"dispatcher listener did not open within {timeout}s"
raise AssertionError(msg)
class TestSubscribe:
def test_subscribe_registers_handler(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen: list = []
d.subscribe("alpha", lambda n: seen.append(n))
_start_ready(d)
# Fire a notify via the storage layer — dispatcher delivers to handler.
storage.notify("alpha", "hello")
assert _wait_for(lambda: any(n.payload == "hello" for n in seen))
def test_subscribe_undeclared_channel_raises(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
with pytest.raises(ValueError, match="not declared"):
d.subscribe("beta", lambda n: None)
def test_subscribe_returns_unsubscribe_callable(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen: list = []
unsub = d.subscribe("alpha", lambda n: seen.append(n))
_start_ready(d)
storage.notify("alpha", "first")
assert _wait_for(lambda: any(n.payload == "first" for n in seen))
unsub()
# After unsubscribe, the handler no longer fires. Drain old hits
# so the next notify-vs-handler-count check is unambiguous.
seen.clear()
storage.notify("alpha", "second")
# Give the dispatcher a beat to deliver if it were going to.
time.sleep(0.2)
assert not any(n.payload == "second" for n in seen)
def test_construction_requires_at_least_one_channel(self, storage):
from turnstone.console.notify_dispatcher import NotifyDispatcher
with pytest.raises(ValueError, match="at least one"):
NotifyDispatcher(storage, channels=[])
def test_duplicate_channels_deduplicated(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha", "alpha", "beta"])
assert d.channels == ["alpha", "beta"]
class TestDispatch:
def test_multiple_handlers_each_invoked(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
seen_a: list = []
seen_b: list = []
d.subscribe("alpha", lambda n: seen_a.append(n))
d.subscribe("alpha", lambda n: seen_b.append(n))
_start_ready(d)
storage.notify("alpha", "shared")
assert _wait_for(lambda: seen_a and seen_b)
assert seen_a[0].payload == "shared"
assert seen_b[0].payload == "shared"
def test_handler_exception_does_not_break_dispatch(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
survived: list = []
def _broken(_n):
msg = "boom"
raise RuntimeError(msg)
d.subscribe("alpha", _broken)
d.subscribe("alpha", lambda n: survived.append(n))
_start_ready(d)
storage.notify("alpha", "after_broken")
# The second handler runs even though the first raised.
assert _wait_for(lambda: any(n.payload == "after_broken" for n in survived))
def test_dispatch_filters_by_channel(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha", "beta"])
seen_a: list = []
seen_b: list = []
d.subscribe("alpha", lambda n: seen_a.append(n))
d.subscribe("beta", lambda n: seen_b.append(n))
_start_ready(d)
storage.notify("alpha", "for_a")
storage.notify("beta", "for_b")
assert _wait_for(lambda: seen_a and seen_b)
assert all(n.payload == "for_a" for n in seen_a)
assert all(n.payload == "for_b" for n in seen_b)
class TestReconnect:
"""Reconnect + synthetic ``reconcile`` notify on stream-open success.
Uses a stub storage that owns its own listen stream so the test can
drive a controlled stream-error sequence the SQLite path can't
raise :class:`NotifyConnectionError`, and the PG path requires a
real database outage to exercise this code, neither of which fits a
unit test. The dispatcher's threading and reconcile-pending logic
are storage-agnostic the dispatcher sees the same
:class:`NotifyStream` Protocol regardless of backend.
"""
def test_reconcile_fires_after_reopen_not_before(self):
from turnstone.console.notify_dispatcher import NotifyDispatcher
from turnstone.core.storage._notify import Notify, NotifyConnectionError
# State machine: open -> first poll raises NotifyConnectionError
# -> dispatcher waits backoff then reopens -> second open's first
# poll blocks forever (test stops the dispatcher before then).
# The fix: synthetic reconcile fires AFTER the second open
# succeeds, not after the first open fails.
sequence: list[str] = []
reopen_event = threading.Event()
class _StubStream:
def __init__(self, fail_first_poll: bool):
self._fail = fail_first_poll
self._closed = False
def poll(self, _timeout):
if self._closed:
return []
if self._fail:
self._fail = False
sequence.append("poll_raises")
msg = "fake-disconnect"
raise NotifyConnectionError(msg)
sequence.append("poll_returns")
# Block until close to simulate a quiet steady-state.
time.sleep(0.5)
return []
def close(self):
self._closed = True
class _StubStorage:
def __init__(self):
self._open_count = 0
def listen(self, _channels):
import contextlib as _contextlib
@_contextlib.contextmanager
def _cm():
self._open_count += 1
sequence.append(f"open_{self._open_count}")
if self._open_count == 2:
reopen_event.set()
stream = _StubStream(fail_first_poll=(self._open_count == 1))
try:
yield stream
finally:
stream.close()
return _cm()
# Speed up backoff so the reopen happens promptly in the test.
import turnstone.console.notify_dispatcher as nd_mod
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
try:
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
got: list[Notify] = []
d.subscribe("alpha", lambda n: got.append(n))
d.start()
try:
# Wait for the second open (post-reconnect).
assert reopen_event.wait(3.0), "dispatcher did not reopen after disconnect"
# Reconcile should be delivered shortly after the reopen.
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if any(n.payload == "reconcile" for n in got):
break
time.sleep(0.02)
assert any(n.payload == "reconcile" for n in got), (
f"no reconcile delivered; sequence={sequence}, got={got}"
)
# The reconcile must NOT fire before the second open —
# if it did, the index of 'open_2' in sequence would
# come after any reconcile-emitting work. Check ordering:
# 'open_1' < 'poll_raises' < 'open_2' (synthesize happens
# inside the with-block of the SECOND open).
ix_open_1 = sequence.index("open_1")
ix_raises = sequence.index("poll_raises")
ix_open_2 = sequence.index("open_2")
assert ix_open_1 < ix_raises < ix_open_2
finally:
d.stop(timeout=2.0)
finally:
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
def test_generic_exception_path_also_synthesizes_reconcile(self):
"""Exceptions thrown during ``listen()`` (not via stream.poll) still trigger reconcile.
Models the ``psycopg.connect()`` / initial ``LISTEN`` failure
shape, which doesn't go through the stream's exception
translator and would hit the generic ``except Exception``
branch. Pre-fix, that branch emitted no reconcile.
"""
from turnstone.console.notify_dispatcher import NotifyDispatcher
reopen_event = threading.Event()
class _StubStream:
def __init__(self):
self._closed = False
def poll(self, _timeout):
if self._closed:
return []
time.sleep(0.5)
return []
def close(self):
self._closed = True
class _StubStorage:
def __init__(self):
self._open_count = 0
def listen(self, _channels):
import contextlib as _contextlib
self._open_count += 1
if self._open_count == 1:
# First open raises a generic exception (e.g.
# ``psycopg.OperationalError`` from a failed connect)
# — landing in the dispatcher's generic except branch.
msg = "fake-connect-failure"
raise RuntimeError(msg)
@_contextlib.contextmanager
def _cm():
reopen_event.set()
stream = _StubStream()
try:
yield stream
finally:
stream.close()
return _cm()
import turnstone.console.notify_dispatcher as nd_mod
original_backoff = nd_mod._RECONNECT_BACKOFF_INITIAL
nd_mod._RECONNECT_BACKOFF_INITIAL = 0.05
try:
d = NotifyDispatcher(_StubStorage(), channels=["alpha"])
got: list = []
d.subscribe("alpha", lambda n: got.append(n))
d.start()
try:
assert reopen_event.wait(3.0), "dispatcher did not reopen after generic exception"
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if any(n.payload == "reconcile" for n in got):
break
time.sleep(0.02)
assert any(n.payload == "reconcile" for n in got), (
"no reconcile delivered after generic-exception recovery"
)
finally:
d.stop(timeout=2.0)
finally:
nd_mod._RECONNECT_BACKOFF_INITIAL = original_backoff
class TestCoalescing:
"""Same-channel burst collapses to one handler invocation per batch."""
def test_burst_coalesces_to_one_handler_call_per_channel(self, dispatcher_factory, storage):
d = dispatcher_factory(channels=["alpha"])
invocations: list = []
# Slow handler to ensure all bursts queue up before the first
# call returns — gives the dispatch loop time to coalesce.
coalesce_gate = threading.Event()
def _slow_handler(n):
invocations.append(n)
coalesce_gate.wait(0.05)
d.subscribe("alpha", _slow_handler)
_start_ready(d)
# Burst of 10 notifies on the same channel — should coalesce
# down to many fewer handler invocations.
for i in range(10):
storage.notify("alpha", str(i))
# Wait until the dispatch settles (handler is called at least once
# and the queue empties).
deadline = time.monotonic() + 2.0
while time.monotonic() < deadline:
if invocations and d._dispatch_queue.empty():
time.sleep(0.1) # allow any final coalesced call to land
break
time.sleep(0.02)
coalesce_gate.set()
# At least one handler call; well fewer than 10 (coalescing
# collapsed the burst). Exact count depends on timing — typical
# is 1-2 invocations per burst on a fast machine.
assert invocations, "handler never fired"
assert len(invocations) < 10, (
f"expected coalescing to collapse burst of 10; got {len(invocations)} invocations"
)
class TestLifecycle:
def test_start_is_idempotent(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
d.start() # No-op, no thread doubling
# Single listener + single dispatch thread are spawned regardless.
# Inspect by name so we don't depend on the exact thread count of
# the test runner.
listener_threads = [
t for t in threading.enumerate() if t.name == "notify-dispatcher-listener"
]
dispatch_threads = [
t for t in threading.enumerate() if t.name == "notify-dispatcher-dispatch"
]
assert len(listener_threads) == 1
assert len(dispatch_threads) == 1
def test_stop_is_idempotent(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
d.stop(timeout=2.0)
d.stop(timeout=2.0) # No-op, no error
def test_stop_without_start_is_noop(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.stop(timeout=1.0) # No-op, no thread to join
def test_stop_joins_threads(self, dispatcher_factory):
d = dispatcher_factory(channels=["alpha"])
d.start()
# Capture thread references then stop and assert they exited.
threads_before = [
t
for t in threading.enumerate()
if t.name in {"notify-dispatcher-listener", "notify-dispatcher-dispatch"}
]
assert threads_before
d.stop(timeout=3.0)
time.sleep(0.05)
for t in threads_before:
assert not t.is_alive(), f"{t.name} still alive after stop"
+35 -6
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import logging
import threading
import pytest
@@ -401,8 +402,10 @@ class TestValidation:
class TestValidUntil:
"""``valid_until`` predicate: drain re-checks freshness; falsy /
raising predicates drop the entry without delivery.
"""``valid_until`` predicate: drain re-checks freshness. Falsy
predicates drop the entry without delivery and log at ``info``
(normal lifecycle outcome); raising predicates drop the entry and
log at ``warning`` with ``exc_info`` (misbehaving predicate).
"""
def test_valid_until_true_delivers(self):
@@ -411,26 +414,52 @@ class TestValidUntil:
out = q.drain({"any"})
assert out == [("a", "1", None)]
def test_valid_until_false_drops_silently(self):
def test_valid_until_false_drops_with_info_log(self, caplog: pytest.LogCaptureFixture):
q = NudgeQueue()
q.enqueue("a", "1", "any", valid_until=lambda: False)
out = q.drain({"any"})
with caplog.at_level(logging.INFO, logger="turnstone.core.nudge_queue"):
out = q.drain({"any"})
assert out == []
# Already removed from queue (drain partition removes BEFORE
# predicate check — falsy doesn't return to queue).
assert len(q) == 0
# The drop emits a structured info record so a wiring
# regression (a predicate that always returns False) is still
# observable, without spamming ``warning`` for the routine
# lifecycle case where ``valid_until`` is doing its job.
# structlog renders the event name + extras into ``msg`` as a
# single rendered string, so substring-match like the
# ``watch_dispatch.queue_full`` assertion in
# tests/test_watch_dispatch.py.
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
assert len(drops) == 1
assert drops[0].levelno == logging.INFO
assert "predicate_false" in drops[0].getMessage()
assert "'nudge_type': 'a'" in drops[0].getMessage()
assert "'channel': 'any'" in drops[0].getMessage()
assert "'text_len': 1" in drops[0].getMessage()
def test_valid_until_exception_drops_silently(self):
def test_valid_until_exception_drops_with_warning(self, caplog: pytest.LogCaptureFixture):
q = NudgeQueue()
def boom() -> bool:
raise RuntimeError("predicate crash")
q.enqueue("a", "1", "any", valid_until=boom)
out = q.drain({"any"})
with caplog.at_level(logging.WARNING, logger="turnstone.core.nudge_queue"):
out = q.drain({"any"})
assert out == []
# Crash-on-predicate is treated as "no longer valid" — drop, not propagate.
assert len(q) == 0
# Stays at ``warning`` (with ``exc_info``) because a raising
# predicate is a bug, not a normal lifecycle outcome.
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
assert len(drops) == 1
assert drops[0].levelno == logging.WARNING
rendered = drops[0].getMessage()
assert "predicate_raised" in rendered
assert "RuntimeError" in rendered
assert "predicate crash" in rendered
def test_valid_until_evaluated_outside_lock(self):
"""The predicate may do non-trivial work (e.g. storage I/O)
+6
View File
@@ -11,6 +11,12 @@ from turnstone.core.session import ChatSession, _render_template
class NullUI:
"""UI adapter that discards all output."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
+125
View File
@@ -0,0 +1,125 @@
"""Tests for ``AnthropicProvider.extract_reasoning_text``.
Phase 1 of the optional-reasoning-persistence feature: provider-side
extractor that walks stored ``provider_blocks`` and returns the
concatenated thinking text, capped at the operator-friendly UI display
size.
These tests drive through the real ``AnthropicProvider`` instance no
mocks of the extractor itself using fixture-shaped blocks that match
what ``_iter_anthropic_stream`` actually accumulates at
``_anthropic.py:713-724`` (``thinking_delta`` + ``signature_delta``
combined into ``{"type": "thinking", "thinking": <text>, "signature":
<sig>}``).
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
from turnstone.core.providers._protocol import (
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
)
@pytest.fixture
def anthropic() -> AnthropicProvider:
return AnthropicProvider()
class TestExtractReasoningText:
def test_none_input_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
assert anthropic.extract_reasoning_text(None) == ""
def test_empty_list_returns_empty_string(self, anthropic: AnthropicProvider) -> None:
assert anthropic.extract_reasoning_text([]) == ""
def test_no_thinking_blocks_returns_empty(self, anthropic: AnthropicProvider) -> None:
blocks: list[dict[str, object]] = [
{"type": "text", "text": "hello"},
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_single_thinking_block_returns_text(self, anthropic: AnthropicProvider) -> None:
blocks = [{"type": "thinking", "thinking": "Let me think about this.", "signature": "abc"}]
assert anthropic.extract_reasoning_text(blocks) == "Let me think about this."
def test_multiple_thinking_blocks_joined_with_newline(
self, anthropic: AnthropicProvider
) -> None:
blocks = [
{"type": "thinking", "thinking": "first thought", "signature": "s1"},
{"type": "thinking", "thinking": "second thought", "signature": "s2"},
]
assert anthropic.extract_reasoning_text(blocks) == "first thought\nsecond thought"
def test_mixed_blocks_extracts_only_thinking(self, anthropic: AnthropicProvider) -> None:
blocks = [
{"type": "thinking", "thinking": "reason A", "signature": "s"},
{"type": "text", "text": "visible answer"},
{"type": "tool_use", "id": "t1", "name": "x", "input": {}},
{"type": "thinking", "thinking": "reason B", "signature": "s"},
]
assert anthropic.extract_reasoning_text(blocks) == "reason A\nreason B"
def test_thinking_block_without_thinking_field_skipped(
self, anthropic: AnthropicProvider
) -> None:
blocks = [{"type": "thinking", "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_thinking_block_with_empty_text_skipped(self, anthropic: AnthropicProvider) -> None:
blocks = [{"type": "thinking", "thinking": "", "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == ""
def test_truncation_at_64kib_cap(self, anthropic: AnthropicProvider) -> None:
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
blocks = [{"type": "thinking", "thinking": long_text, "signature": "s"}]
result = anthropic.extract_reasoning_text(blocks)
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
def test_just_under_cap_not_truncated(self, anthropic: AnthropicProvider) -> None:
text = "y" * (_MAX_REASONING_DISPLAY_CHARS - 1)
blocks = [{"type": "thinking", "thinking": text, "signature": "s"}]
assert anthropic.extract_reasoning_text(blocks) == text
def test_malformed_block_entry_skipped(self, anthropic: AnthropicProvider) -> None:
# A defensive sanity check — we should not crash if some
# entry isn't a dict (e.g. a corrupted JSON payload).
blocks = [
"not a dict", # type: ignore[list-item]
{"type": "thinking", "thinking": "good one", "signature": "s"},
]
assert anthropic.extract_reasoning_text(blocks) == "good one" # type: ignore[arg-type]
def test_non_list_input_returns_empty(self, anthropic: AnthropicProvider) -> None:
# Defensive against a corrupted provider_data payload.
assert anthropic.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
assert anthropic.extract_reasoning_text({"type": "thinking"}) == "" # type: ignore[arg-type]
class TestOtherProvidersDefault:
"""Non-Anthropic providers return "" for the same fixture shapes."""
def test_openai_chat_returns_empty(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
assert provider.extract_reasoning_text(blocks) == ""
def test_openai_responses_returns_empty(self) -> None:
provider = OpenAIResponsesProvider()
blocks = [{"type": "thinking", "thinking": "would-be-text", "signature": "s"}]
assert provider.extract_reasoning_text(blocks) == ""
def test_openai_responses_extracts_reasoning_summary(self) -> None:
# Phase 3: extractor now walks reasoning items captured via
# include=["reasoning.encrypted_content"] and returns the
# summary[*].text concatenation. Pre-Phase-3 this returned
# "" — the stub was replaced once the wire path landed.
provider = OpenAIResponsesProvider()
blocks = [{"type": "reasoning", "summary": [{"type": "summary_text", "text": "x"}]}]
assert provider.extract_reasoning_text(blocks) == "x"
+532
View File
@@ -0,0 +1,532 @@
"""Tests for Phase 2 wire-build replay flag + shape filter on AnthropicProvider.
Phase 2 of optional reasoning persistence wraps the verbatim
``_provider_content`` replay path at ``_anthropic.py:_convert_messages``
with two gates:
1. ``ANTHROPIC_VALID_BLOCK_TYPES`` per-block shape filter foreign-
shaped blocks (OpenAI Responses ``type="reasoning"``, Gemini thought
parts, the synthetic ``reasoning_text`` from path-3 capture) are
dropped individually; valid Anthropic blocks in the same message
still ride the verbatim path. When NO valid blocks survive, the
converter falls through to the text+tool_calls rebuild path.
2. ``replay_reasoning_to_model`` operator flag when False (the
``model_definitions`` server_default), thinking blocks are
stripped before the wire payload is built. Tool_use /
server_tool_use / web_search_tool_result blocks (which carry
web-search ``encrypted_content``) intentionally survive the
strip predicate is narrow by design.
Drives through the real ``AnthropicProvider._convert_messages`` with
fixture-shaped messages, no mocks of the converter. Edge cases come
from the briefing's "Edges & validation memo" sections.
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._anthropic import (
ANTHROPIC_REASONING_BLOCK_TYPES,
ANTHROPIC_VALID_BLOCK_TYPES,
AnthropicProvider,
)
@pytest.fixture
def provider() -> AnthropicProvider:
return AnthropicProvider()
def _assistant_with_thinking(content: str = "Final answer.") -> dict[str, object]:
"""Build an assistant message with a thinking + text + tool_use shape
matching what the streaming layer captures at ``_anthropic.py:713-724``."""
return {
"role": "assistant",
"content": content,
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "sig"},
{"type": "text", "text": content},
{
"type": "tool_use",
"id": "call_abc",
"name": "search",
"input": {"q": "x"},
},
],
"tool_calls": [
{
"id": "call_abc",
"type": "function",
"function": {"name": "search", "arguments": '{"q": "x"}'},
}
],
}
class TestReplayFlagStripsThinking:
"""``replay_reasoning_to_model=False`` strips thinking; ``True`` preserves."""
def test_replay_true_preserves_thinking_block(self, provider: AnthropicProvider) -> None:
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present
assert "text" in types_present
assert "tool_use" in types_present
def test_replay_false_strips_thinking_block(self, provider: AnthropicProvider) -> None:
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present
assert "text" in types_present # final answer survives
assert "tool_use" in types_present # tool dispatch survives
def test_replay_false_strips_redacted_thinking_too(self, provider: AnthropicProvider) -> None:
# Anthropic emits redacted_thinking blocks when the safety system
# rewrites a thinking block. Phase 2 strip predicate must include
# both shapes.
msg = {
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{"type": "redacted_thinking", "data": "redacted-blob"},
{"type": "text", "text": "Answer."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "redacted_thinking" not in types_present
assert "text" in types_present
def test_default_kwarg_preserves_existing_behaviour(self, provider: AnthropicProvider) -> None:
"""Pre-Phase-2 callers that don't pass the kwarg get the verbatim
replay (default True), matching the behaviour all production
Anthropic-with-thinking turns shipped with for months."""
msg = _assistant_with_thinking()
_, converted = provider._convert_messages([msg]) # no kwarg
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present
class TestWebSearchBlocksSurviveStrip:
"""Edge 14: Anthropic web-search ``encrypted_content`` rides on
``server_tool_use`` / ``web_search_tool_result`` blocks (NOT
thinking blocks). Strip predicate is intentionally narrow."""
def test_server_tool_use_survives(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "From search: ...",
"_provider_content": [
{"type": "thinking", "thinking": "I should search", "signature": "s"},
{
"type": "server_tool_use",
"id": "stu_1",
"name": "web_search",
"input": {"query": "turnstone bird"},
},
{
"type": "web_search_tool_result",
"tool_use_id": "stu_1",
"content": [{"type": "web_search_result", "url": "https://e.com"}],
"encrypted_content": "abc123encrypted",
"encrypted_index": "idx456encrypted",
},
{"type": "text", "text": "From search: ..."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present # stripped
assert "server_tool_use" in types_present # survives
assert "web_search_tool_result" in types_present # survives
assert "text" in types_present # survives
# encrypted_content rides through intact — required for round-trip continuity
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
assert wsr["encrypted_content"] == "abc123encrypted"
def test_tool_use_block_survives(self, provider: AnthropicProvider) -> None:
# Plain tool_use (not server-side) — used by client-side function
# tools. Strip predicate must not touch these.
msg = {
"role": "assistant",
"content": "Calling tool",
"_provider_content": [
{"type": "thinking", "thinking": "I should call tool", "signature": "s"},
{"type": "tool_use", "id": "tu_1", "name": "f", "input": {"a": 1}},
],
"tool_calls": [
{
"id": "tu_1",
"type": "function",
"function": {"name": "f", "arguments": '{"a": 1}'},
}
],
}
# Provide the tool result so orphan-tool detection doesn't synthesize
msgs = [
msg,
{"role": "tool", "tool_call_id": "tu_1", "content": "ok"},
]
_, converted = provider._convert_messages(msgs, replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" not in types_present
assert "tool_use" in types_present
def test_orphan_tool_use_synthesized_after_strip(self, provider: AnthropicProvider) -> None:
"""Pin the post-strip orphan-tool branch at _anthropic.py:397-433.
The implementation comment specifically calls out reading
``provider_content`` (not ``wire_blocks``) for the orphan-tool
ID walk after the strip keeping the read on the source-of-
truth list so a future refactor that swapped them would still
get the same set of tool_use IDs. This test exercises that
branch end-to-end: replay=False strips the thinking block,
AND the message has a tool_use whose result is missing. The
converter must synthesize a 'cancelled' tool_result for the
orphaned tool_use ID (matching the existing pre-Phase-2
behaviour for the verbatim path).
"""
msg = {
"role": "assistant",
"content": "Calling tool",
"_provider_content": [
{"type": "thinking", "thinking": "let me think", "signature": "s"},
{"type": "tool_use", "id": "orphan_tu", "name": "f", "input": {"a": 1}},
],
"tool_calls": [
{
"id": "orphan_tu",
"type": "function",
"function": {"name": "f", "arguments": '{"a": 1}'},
}
],
}
# NO tool result follows — orphan branch must synthesize one.
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
# Synthetic tool_result lands as a user-role message immediately
# after the assistant turn (per existing behaviour at
# _anthropic.py:421-430).
assistant = next(m for m in converted if m["role"] == "assistant")
# Stripped: thinking gone, tool_use survives.
a_types = [b["type"] for b in assistant["content"]]
assert "thinking" not in a_types
assert "tool_use" in a_types
# Synthesized: cancelled tool_result for orphan_tu attached to a
# following user-role message.
user_msgs_after = [m for m in converted if m["role"] == "user"]
assert user_msgs_after, (
"Expected a synthetic user message carrying the cancelled "
"tool_result for the orphaned tool_use"
)
flat_results = [
block
for um in user_msgs_after
if isinstance(um["content"], list)
for block in um["content"]
if isinstance(block, dict) and block.get("type") == "tool_result"
]
synth = next(
(b for b in flat_results if b.get("tool_use_id") == "orphan_tu"),
None,
)
assert synth is not None, f"Expected synthetic tool_result for orphan_tu in {flat_results}"
assert synth.get("is_error") is True
assert "cancelled" in synth.get("content", "").lower()
class TestShapeFilterFallthrough:
"""Foreign / empty / mixed-shape ``_provider_content`` falls through
to the text+tool_calls rebuild path rather than reaching the wire
as a malformed block."""
def test_foreign_shape_openai_reasoning_falls_through(
self, provider: AnthropicProvider
) -> None:
# OpenAI Responses style block (Phase 3 will land this shape into
# _provider_content via include=["reasoning.encrypted_content"]).
# Mid-workstream model switch from OpenAI -> Anthropic must NOT
# reach the API with an OpenAI-shaped block (which would 400).
msg = {
"role": "assistant",
"content": "Final answer from openai turn.",
"_provider_content": [
{
"type": "reasoning",
"summary": [{"type": "summary_text", "text": "I reasoned..."}],
"encrypted_content": "openai-encrypted",
}
],
"tool_calls": [],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
# Rebuilt from text — no foreign block reached the wire.
for b in assistant["content"]:
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES, (
f"Foreign block type leaked through: {b}"
)
# And the foreign block specifically is NOT present.
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present
def test_mixed_shape_drops_foreign_keeps_valid(self, provider: AnthropicProvider) -> None:
# Per-block filter: a single foreign block in a mostly-Anthropic
# payload no longer forces fall-through. Valid Anthropic blocks
# ride the verbatim path; the foreign block is dropped.
msg = {
"role": "assistant",
"content": "Mixed.",
"_provider_content": [
{"type": "thinking", "thinking": "anth shape", "signature": "s"},
{"type": "text", "text": "Mixed."},
{"type": "reasoning", "summary": []}, # foreign
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present # foreign dropped
assert "thinking" in types_present # valid + replay=True kept
assert "text" in types_present
for b in assistant["content"]:
assert b.get("type") in ANTHROPIC_VALID_BLOCK_TYPES
def test_mixed_shape_preserves_web_search_encrypted_content(
self, provider: AnthropicProvider
) -> None:
# The motivating case for per-block (vs all-or-nothing) filter:
# cross-model resumption stamps a foreign ``reasoning`` block
# alongside Anthropic web-search blocks carrying encrypted
# citations. An all-or-nothing filter would discard the whole
# message and rebuild from text+tool_calls — silently losing
# the encrypted_content the API needs for round-trip continuity.
msg = {
"role": "assistant",
"content": "From search: ...",
"_provider_content": [
{"type": "reasoning", "summary": []}, # foreign (e.g. OpenAI)
{
"type": "server_tool_use",
"id": "stu_1",
"name": "web_search",
"input": {"query": "x"},
},
{
"type": "web_search_tool_result",
"tool_use_id": "stu_1",
"content": [{"type": "web_search_result", "url": "https://e.com"}],
"encrypted_content": "encrypted-blob-must-survive",
"encrypted_index": "encrypted-idx-must-survive",
},
{"type": "text", "text": "From search: ..."},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "reasoning" not in types_present # foreign dropped
assert "server_tool_use" in types_present
assert "web_search_tool_result" in types_present
assert "text" in types_present
wsr = next(b for b in assistant["content"] if b["type"] == "web_search_tool_result")
assert wsr["encrypted_content"] == "encrypted-blob-must-survive"
assert wsr["encrypted_index"] == "encrypted-idx-must-survive"
def test_all_foreign_blocks_fall_through_to_rebuild(self, provider: AnthropicProvider) -> None:
# When every block is foreign-shaped (no Anthropic-valid block
# survives the per-block filter), the converter still falls
# through to text+tool_calls rebuild rather than emitting an
# empty assistant turn.
msg = {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "reasoning", "summary": []},
{"type": "reasoning_text", "text": "synthetic"}, # path-3 shape
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
# Rebuild path: msg.content lifted into a single text block.
assert assistant["content"] == [{"type": "text", "text": "Final answer."}]
def test_empty_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "Plain text answer.",
"_provider_content": [],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
# Falls through to text rebuild
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
def test_none_provider_content_falls_through(self, provider: AnthropicProvider) -> None:
msg = {
"role": "assistant",
"content": "Plain text answer.",
"_provider_content": None,
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Plain text answer."}]
def test_provider_content_not_a_list_falls_through(self, provider: AnthropicProvider) -> None:
# Defensive against a corrupted provider_data deserialization.
msg = {
"role": "assistant",
"content": "Plain.",
"_provider_content": "not a list",
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Plain."}]
def test_non_dict_and_missing_type_blocks_are_dropped(
self, provider: AnthropicProvider
) -> None:
# Defensive branches in the per-block walk: a stray non-dict
# element (corrupted JSON) or a dict with no/None ``type`` key
# (provider drift) must be silently dropped without raising.
# Valid blocks in the same list still ride the verbatim path.
msg = {
"role": "assistant",
"content": "ok",
"_provider_content": [
{"type": "text", "text": "ok"},
"stray-string", # non-dict
{"type": None, "text": "huh"}, # None type
{"no_type_key": 1}, # missing type
{"type": "thinking", "thinking": "t", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=True)
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b.get("type") for b in assistant["content"]]
assert types_present == ["text", "thinking"]
class TestLegacyAnthropicRowsNoRegression:
"""Critical property: rows persisted before Phase 2 carry valid
Anthropic-shape _provider_content (only Anthropic captured this lane
historically). They must stay in the verbatim path and keep their
thinking context across the migration boundary when replay=True
(the legacy default).
"""
def test_legacy_thinking_row_preserved_with_default_kwarg(
self, provider: AnthropicProvider
) -> None:
"""No kwarg passed (matches the pre-Phase-2 production call site)."""
msg = {
"role": "assistant",
"content": "Old answer from months ago.",
"_provider_content": [
{"type": "thinking", "thinking": "old reasoning", "signature": "s"},
{"type": "text", "text": "Old answer from months ago."},
],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
types_present = [b["type"] for b in assistant["content"]]
assert "thinking" in types_present # preserved -> no regression
# The thinking block IS the same dict as the source (verbatim path).
assert assistant["content"][0]["thinking"] == "old reasoning"
def test_replay_false_only_strips_when_explicitly_requested(
self, provider: AnthropicProvider
) -> None:
# Operator flips persist+replay flags off. Strip fires.
# Pinning that the strip is gated on the explicit flag value,
# not silently triggered by some other condition.
msg = {
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{"type": "thinking", "thinking": "stripped", "signature": "s"},
{"type": "text", "text": "Answer."},
],
}
_, converted_default = provider._convert_messages([msg])
_, converted_strip = provider._convert_messages([msg], replay_reasoning_to_model=False)
default_types = [b["type"] for b in converted_default[0]["content"]]
strip_types = [b["type"] for b in converted_strip[0]["content"]]
assert "thinking" in default_types
assert "thinking" not in strip_types
class TestStripAllBlocksFallthrough:
"""When the message is 100% thinking (no text, no tool_use) and
replay=False strips everything, the message falls through to the
text+tool_calls rebuild path. If both are also empty, the assistant
turn is silently skipped correct: stripped reasoning has nothing
to replay."""
def test_only_thinking_strip_falls_to_rebuild_with_text(
self, provider: AnthropicProvider
) -> None:
# Provider_content = only thinking; msg.content has the spoken text.
# Strip drops thinking; rebuild path picks up the content as a
# text block. No information lost.
msg = {
"role": "assistant",
"content": "Spoken answer.",
"_provider_content": [
{"type": "thinking", "thinking": "internal", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
assistant = next(m for m in converted if m["role"] == "assistant")
assert assistant["content"] == [{"type": "text", "text": "Spoken answer."}]
def test_only_thinking_strip_with_no_content_skips_message(
self, provider: AnthropicProvider
) -> None:
# Edge: provider_content was 100% thinking AND msg.content is
# empty AND no tool_calls. The rebuild path sees nothing to
# emit — assistant turn silently skipped. Anthropic's API
# would reject an empty assistant content array anyway.
msg = {
"role": "assistant",
"content": "",
"_provider_content": [
{"type": "thinking", "thinking": "only", "signature": "s"},
],
}
_, converted = provider._convert_messages([msg], replay_reasoning_to_model=False)
# Assistant turn skipped — no entry for it in `converted`.
assert all(m["role"] != "assistant" for m in converted)
class TestConstants:
"""Pin the constant contents so a future edit doesn't accidentally
widen the strip set or narrow the valid set."""
def test_reasoning_block_types_is_narrow(self) -> None:
# Strip predicate MUST cover only reasoning shapes. Adding
# tool_use here would break web-search round-trip.
assert frozenset({"thinking", "redacted_thinking"}) == ANTHROPIC_REASONING_BLOCK_TYPES
def test_valid_block_types_includes_web_search(self) -> None:
# Without server_tool_use / web_search_tool_result, Anthropic
# web-search results would fall through to the rebuild path
# and lose their encrypted_content.
assert "server_tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
assert "web_search_tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
assert "tool_use" in ANTHROPIC_VALID_BLOCK_TYPES
assert "tool_result" in ANTHROPIC_VALID_BLOCK_TYPES
def test_reasoning_subset_of_valid(self) -> None:
# The strip set must be a subset of the valid set — otherwise
# the strip predicate would never match anything (we only
# strip after shape validity passes).
assert ANTHROPIC_REASONING_BLOCK_TYPES.issubset(ANTHROPIC_VALID_BLOCK_TYPES)
@@ -0,0 +1,320 @@
"""Tests for OpenAI Responses reasoning capture + replay (Phase 3 path 2).
Phase 3 wires:
1. ``include=["reasoning.encrypted_content"]`` on the request when
the operator flag AND the model capability both allow.
2. ``_convert_messages`` round-tripping stored reasoning items as
``ResponseReasoningItemParam`` input items on subsequent turns.
3. ``OpenAIResponsesProvider.extract_reasoning_text`` walking
reasoning items and returning concatenated summary + content text.
All tests drive through the real ``OpenAIResponsesProvider`` no
mocks of the converter/build_kwargs themselves; only the SDK boundary
is mocked where relevant.
"""
from __future__ import annotations
import pytest
from turnstone.core.providers._openai_responses import (
OpenAIResponsesProvider,
_reasoning_item_for_input,
)
from turnstone.core.providers._protocol import (
MAX_REASONING_DISPLAY_CHARS as _MAX_REASONING_DISPLAY_CHARS,
)
from turnstone.core.providers._protocol import ModelCapabilities
@pytest.fixture
def provider() -> OpenAIResponsesProvider:
return OpenAIResponsesProvider()
def _capable_caps() -> ModelCapabilities:
"""Capability fixture for a reasoning-replay-capable model."""
return ModelCapabilities(
context_window=400000,
max_output_tokens=128000,
supports_temperature=False,
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
supports_reasoning_replay=True,
)
class TestExtractReasoningText:
def test_none_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
assert provider.extract_reasoning_text(None) == ""
def test_empty_list_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
assert provider.extract_reasoning_text([]) == ""
def test_no_reasoning_items_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
blocks = [
{"type": "message", "role": "assistant", "content": "hi"},
{"type": "function_call", "call_id": "c1", "name": "x", "arguments": "{}"},
]
assert provider.extract_reasoning_text(blocks) == ""
def test_summary_text_extracted(self, provider: OpenAIResponsesProvider) -> None:
# Per ResponseReasoningItem (response_reasoning_item.py:31-62):
# summary is always present; content is optional.
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [
{"type": "summary_text", "text": "I considered X"},
{"type": "summary_text", "text": "then Y"},
],
}
]
assert provider.extract_reasoning_text(blocks) == "I considered X\nthen Y"
def test_content_text_extracted_alongside_summary(
self, provider: OpenAIResponsesProvider
) -> None:
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "summary line"}],
"content": [{"type": "reasoning_text", "text": "raw reasoning"}],
}
]
# Order: summary first, then content (matches the order the SDK
# surfaces them via streaming events).
result = provider.extract_reasoning_text(blocks)
assert "summary line" in result
assert "raw reasoning" in result
def test_truncation_at_64kib_cap(self, provider: OpenAIResponsesProvider) -> None:
long_text = "x" * (_MAX_REASONING_DISPLAY_CHARS + 1024)
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": long_text}],
}
]
result = provider.extract_reasoning_text(blocks)
assert len(result) == _MAX_REASONING_DISPLAY_CHARS
def test_malformed_summary_entry_skipped(self, provider: OpenAIResponsesProvider) -> None:
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [
"not a dict",
{"type": "summary_text"}, # missing text
{"type": "summary_text", "text": ""}, # empty text
{"type": "summary_text", "text": "good"},
],
}
]
assert provider.extract_reasoning_text(blocks) == "good"
def test_non_list_input_returns_empty(self, provider: OpenAIResponsesProvider) -> None:
assert provider.extract_reasoning_text("not a list") == "" # type: ignore[arg-type]
def test_other_block_types_skipped_in_walk(self, provider: OpenAIResponsesProvider) -> None:
# Mixed payload: only the reasoning block contributes.
blocks = [
{"type": "message", "role": "assistant", "content": "hi"},
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "thought"}],
},
{"type": "function_call", "call_id": "c1", "name": "x", "arguments": "{}"},
]
assert provider.extract_reasoning_text(blocks) == "thought"
class TestReasoningItemForInput:
"""``_reasoning_item_for_input`` projects a stored ``ResponseReasoningItem``
dict into ``ResponseReasoningItemParam`` shape (drops server-only
``status``)."""
def test_minimal_item_round_trip(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
"status": "completed",
}
result = _reasoning_item_for_input(stored)
assert result["type"] == "reasoning"
assert result["id"] == "r_1"
assert result["summary"] == [{"type": "summary_text", "text": "x"}]
# status NOT round-tripped (server-only field per
# ResponseReasoningItemParam at response_reasoning_item_param.py).
assert "status" not in result
def test_encrypted_content_round_trips_when_present(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
"encrypted_content": "opaque-blob",
}
result = _reasoning_item_for_input(stored)
assert result["encrypted_content"] == "opaque-blob"
def test_encrypted_content_omitted_when_absent(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
}
result = _reasoning_item_for_input(stored)
assert "encrypted_content" not in result
def test_content_round_trips_when_present(self) -> None:
stored = {
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "s"}],
"content": [{"type": "reasoning_text", "text": "raw"}],
}
result = _reasoning_item_for_input(stored)
assert result["content"] == [{"type": "reasoning_text", "text": "raw"}]
class TestBuildKwargsInclude:
"""``_build_kwargs`` adds ``include=["reasoning.encrypted_content"]``
when the resolved operator flag is True. The capability AND-gate
lives upstream in ``ChatSession._resolve_replay_reasoning_to_model``
(single source of truth across providers); the provider trusts the
bool it receives. See
``test_session_replay_reasoning.py::TestSessionToOpenAIResponsesBoundaryIntegration``
for the end-to-end gate test."""
def test_include_added_when_flag_true(self, provider: OpenAIResponsesProvider) -> None:
kwargs = provider._build_kwargs(
model="gpt-5",
messages=[{"role": "user", "content": "hi"}],
tools=None,
max_tokens=1024,
temperature=0.5,
reasoning_effort="medium",
deferred_names=None,
capabilities=_capable_caps(),
replay_reasoning_to_model=True,
)
assert kwargs.get("include") == ["reasoning.encrypted_content"]
def test_include_omitted_when_flag_false(self, provider: OpenAIResponsesProvider) -> None:
kwargs = provider._build_kwargs(
model="gpt-5",
messages=[{"role": "user", "content": "hi"}],
tools=None,
max_tokens=1024,
temperature=0.5,
reasoning_effort="medium",
deferred_names=None,
capabilities=_capable_caps(),
replay_reasoning_to_model=False,
)
assert "include" not in kwargs
class TestConvertMessagesReasoningReplay:
"""``_convert_messages`` round-trips stored reasoning items as input."""
def test_reasoning_item_emitted_before_assistant_when_replay_true(
self, provider: OpenAIResponsesProvider
) -> None:
messages = [
{"role": "user", "content": "explain"},
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "I thought"}],
"encrypted_content": "abc",
}
],
},
{"role": "user", "content": "follow up"},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
# Find the reasoning input item.
types = [it.get("type") for it in items]
# Expected: user, reasoning, message (assistant), user.
assert types == ["message", "reasoning", "message", "message"]
reasoning_idx = types.index("reasoning")
r_item = items[reasoning_idx]
assert r_item["id"] == "r_1"
assert r_item["encrypted_content"] == "abc"
# And the reasoning item appears immediately BEFORE the
# assistant message it belongs to.
assert items[reasoning_idx + 1]["role"] == "assistant"
def test_reasoning_item_dropped_when_replay_false(
self, provider: OpenAIResponsesProvider
) -> None:
messages = [
{
"role": "assistant",
"content": "Answer.",
"_provider_content": [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "thought"}],
}
],
},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=False)
types = [it.get("type") for it in items]
assert "reasoning" not in types
def test_no_reasoning_items_when_provider_content_lacks_reasoning(
self, provider: OpenAIResponsesProvider
) -> None:
# Anthropic-shaped _provider_content reaching OpenAI Responses
# (cross-provider — operator switch from Anthropic to GPT-5):
# no type=="reasoning" items, so nothing emitted.
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{"type": "thinking", "thinking": "anth", "signature": "s"},
],
},
]
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
types = [it.get("type") for it in items]
assert "reasoning" not in types
def test_default_replay_reasoning_false_omits_reasoning(
self, provider: OpenAIResponsesProvider
) -> None:
# Pre-Phase-3 callers (no kwarg) get the back-compat behaviour:
# reasoning items are silently dropped (sanitize_messages was
# already stripping _provider_content anyway).
messages = [
{
"role": "assistant",
"content": "x",
"_provider_content": [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": "x"}],
}
],
},
]
_, items = provider._convert_messages(messages) # no kwarg
types = [it.get("type") for it in items]
assert "reasoning" not in types
@@ -0,0 +1,333 @@
"""Audit-log discipline test for reasoning text.
Phase 1 of optional reasoning persistence surfaces stored thinking
blocks on the ``/history`` payload (UI rehydration). The bytes ride
through the helper (``extract_reasoning_for_history``), through the
provider extractor (``AnthropicProvider.extract_reasoning_text``), and
through the server build path (``_build_history``).
This test pins the security-sensitive contract:
Reasoning text MAY land on ``msg["reasoning"]`` (UI-bound),
but MUST NOT appear in any ``Logger.info`` / ``warning`` /
``error`` payload at any layer in the pipeline.
The test mocks the standard-library ``logging.Logger`` info/warning/
error methods, runs a thinking-bearing turn through the relevant
extractors and history build, then asserts no captured log call's
positional args or kwargs contain the unique marker string. Replaces
the v4 grep-the-output approach (fragile when log strings are
formatted) with a structural mock-and-assert (tests the actual
contract rather than the rendered text).
"""
from __future__ import annotations
import logging
from types import SimpleNamespace
from typing import Any
from unittest.mock import patch
from tests._session_helpers import make_session
from turnstone.core.history_decoration import (
extract_reasoning_for_history,
extract_reasoning_text_from_provider_content,
)
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
from turnstone.core.providers._protocol import StreamChunk, UsageInfo
from turnstone.server import _build_history
_MARKER = "SECRET_REASONING_MARKER_xyz123_unlikely_collision"
def _payload_contains_marker(args: tuple[Any, ...], kwargs: dict[str, Any]) -> bool:
"""Walk a captured log call's args + kwargs for the marker string.
Logger.info-style calls accept a format string + positional substitution
args; the marker could appear in either the format string itself or
the substitution values. Format-time strings (``%`` substitution) are
NOT inspected because they're a stdlib formatting concern, not a
callable our pipeline reaches into. The structural check is "no
user-controlled marker appears in any arg slot we passed".
"""
for a in args:
if isinstance(a, str) and _MARKER in a:
return True
# Defensive — a list/dict/exception arg might carry the marker too.
try:
if _MARKER in repr(a):
return True
except Exception:
continue
for v in kwargs.values():
if isinstance(v, str) and _MARKER in v:
return True
try:
if _MARKER in repr(v):
return True
except Exception:
continue
return False
def _capture_log_calls():
"""Capture every Logger.info / warning / error call into a single list."""
captured: list[tuple[str, tuple[Any, ...], dict[str, Any]]] = []
def make_recorder(level: str):
def _rec(*args: Any, **kwargs: Any) -> None:
captured.append((level, args, kwargs))
return _rec
return captured, [
patch.object(logging.Logger, "info", side_effect=make_recorder("info"), autospec=True),
patch.object(
logging.Logger, "warning", side_effect=make_recorder("warning"), autospec=True
),
patch.object(logging.Logger, "error", side_effect=make_recorder("error"), autospec=True),
]
class TestReasoningAuditLogDiscipline:
"""Reasoning text never lands at INFO+ severity on any logger."""
def _thinking_msg(self, text: str = _MARKER) -> dict[str, Any]:
return {
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": text, "signature": "sig"},
{"type": "text", "text": "Final answer."},
],
}
def test_anthropic_extractor_does_not_log_reasoning(self) -> None:
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
provider = AnthropicProvider()
text = provider.extract_reasoning_text(
[{"type": "thinking", "thinking": _MARKER, "signature": "s"}]
)
assert text == _MARKER # extractor IS allowed to return it
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"AnthropicProvider.extract_reasoning_text leaked reasoning text "
f"into INFO+ logs: {offending}"
)
def test_dispatch_helper_does_not_log_reasoning(self) -> None:
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
text = extract_reasoning_text_from_provider_content(
[{"type": "thinking", "thinking": _MARKER, "signature": "s"}]
)
assert text == _MARKER
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"extract_reasoning_text_from_provider_content leaked reasoning "
f"text into INFO+ logs: {offending}"
)
def test_list_helper_does_not_log_reasoning(self) -> None:
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
messages = [self._thinking_msg(_MARKER)]
extract_reasoning_for_history(messages, surface_persisted_reasoning_flag=True)
assert messages[0]["reasoning"] == _MARKER # UI-bound is allowed
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"extract_reasoning_for_history leaked reasoning text into INFO+ logs: {offending}"
)
def test_build_history_does_not_log_reasoning(self) -> None:
registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(surface_persisted_reasoning=True)
)
session = SimpleNamespace(
messages=[self._thinking_msg(_MARKER)],
_ws_id="ws-audit",
_registry=registry,
_model_alias="claude-opus-4-7",
)
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
with patch(
"turnstone.server._load_verdict_indexes",
return_value=({}, {}),
):
history = _build_history(session)
assert history[0]["reasoning"] == _MARKER # UI-bound is allowed
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], f"_build_history leaked reasoning text into INFO+ logs: {offending}"
# ------------------------------------------------------------------
# Phase 2 + Phase 3 surfaces — added in response to a code-review
# finding that the original 4-test coverage missed every code path
# introduced after Phase 1. Each new test mirrors the structure
# above: capture every Logger.info / warning / error call across
# the operation, assert the marker doesn't appear in any captured
# payload (UI-bound returns IS allowed; logging at INFO+ is NOT).
# ------------------------------------------------------------------
def test_openai_responses_extractor_does_not_log_reasoning(self) -> None:
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
provider = OpenAIResponsesProvider()
blocks = [
{
"type": "reasoning",
"id": "r_1",
"summary": [{"type": "summary_text", "text": _MARKER}],
}
]
text = provider.extract_reasoning_text(blocks)
assert _MARKER in text # UI-bound return is allowed
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"OpenAIResponsesProvider.extract_reasoning_text leaked reasoning "
f"text into INFO+ logs: {offending}"
)
def test_openai_chat_extractor_does_not_log_reasoning(self) -> None:
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
provider = OpenAIChatCompletionsProvider()
blocks = [{"type": "reasoning_text", "text": _MARKER, "source": "vllm"}]
text = provider.extract_reasoning_text(blocks)
assert text == _MARKER
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"OpenAIChatCompletionsProvider.extract_reasoning_text leaked "
f"reasoning text into INFO+ logs: {offending}"
)
def test_synth_reasoning_block_via_stream_response_does_not_log_reasoning(
self,
) -> None:
"""Drives ChatSession._stream_response (which calls
_maybe_synth_reasoning_block at end-of-stream) with a fake
``reasoning_delta=_MARKER`` chunk; asserts no log call carried
the marker text."""
session = make_session()
chunks = [
StreamChunk(reasoning_delta=_MARKER, is_first=True),
StreamChunk(content_delta="answer"),
StreamChunk(
finish_reason="stop",
usage=UsageInfo(prompt_tokens=10, completion_tokens=20, total_tokens=30),
),
]
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
msg = session._stream_response(iter(chunks))
# Synth block stamped onto _provider_content with the marker.
assert msg["_provider_content"][0]["text"] == _MARKER
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"_stream_response + _maybe_synth_reasoning_block leaked reasoning "
f"text into INFO+ logs: {offending}"
)
def test_anthropic_convert_messages_strip_does_not_log_reasoning(self) -> None:
"""Drives the Phase 2 strip predicate
(``replay_reasoning_to_model=False``) which walks thinking
blocks to filter them out before the wire payload is built;
asserts no log call carried the marker text."""
captured, patchers = _capture_log_calls()
for p in patchers:
p.start()
try:
provider = AnthropicProvider()
messages = [
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": _MARKER, "signature": "s"},
{"type": "text", "text": "Final answer."},
],
},
]
_, converted = provider._convert_messages(messages, replay_reasoning_to_model=False)
# Strip fired — thinking block dropped from wire.
assistant = next(m for m in converted if m["role"] == "assistant")
block_types = [b.get("type") for b in assistant["content"]]
assert "thinking" not in block_types
finally:
for p in patchers:
p.stop()
offending = [
(lvl, args, kwargs)
for lvl, args, kwargs in captured
if _payload_contains_marker(args, kwargs)
]
assert offending == [], (
f"AnthropicProvider._convert_messages strip predicate leaked "
f"reasoning text into INFO+ logs: {offending}"
)
+668 -19
View File
@@ -255,12 +255,17 @@ function makeEl(tag) {
setAttribute(k, v) { this._attrs[k] = v; },
getAttribute(k) { return this._attrs[k] !== undefined ? this._attrs[k] : null; },
get classList() {
// Real DOMTokenList is array-like (length + indexed access) AND
// exposes add/remove/contains. The hljs language-extraction
// loop reads .length + [j], so we return a fresh Array snapshot
// each get + bolt the mutator methods on. add/remove operate on
// the live _classes set so subsequent reads see updates.
const self = this;
return {
add(...c) { c.forEach(x => self._classes.add(x)); },
remove(...c) { c.forEach(x => self._classes.delete(x)); },
contains(c) { return self._classes.has(c); },
};
const arr = Array.from(self._classes);
arr.add = (...c) => c.forEach((x) => self._classes.add(x));
arr.remove = (...c) => c.forEach((x) => self._classes.delete(x));
arr.contains = (c) => self._classes.has(c);
return arr;
},
get className() { return Array.from(this._classes).join(' '); },
set className(v) {
@@ -269,9 +274,33 @@ function makeEl(tag) {
get textContent() {
return this._textContent || this.children.map(c => c.textContent || '').join('');
},
set textContent(v) { this._textContent = v; this.children = []; },
set textContent(v) {
// Real DOM: assigning textContent ALSO replaces innerHTML with
// an entity-escaped representation of the same text. escapeHtml
// (utils.js) round-trips via this side effect without it,
// every escapeHtml() call returns '' and renderMarkdown emits
// empty <p> tags.
this._textContent = v;
this.children = [];
this._innerHTML = String(v)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
},
get innerHTML() { return this._innerHTML; },
set innerHTML(v) { this._innerHTML = v; this.children = []; },
set innerHTML(v) {
// Real DOM invalidates the previous textContent when innerHTML
// is replaced leaving _textContent intact would return stale
// data from subsequent textContent reads and mask bugs that
// depend on innerHTML/textContent consistency. We don't HTML-
// parse here, so the cheap correct behavior is to clear
// _textContent and let the children-derived fallback in the
// textContent getter (which is empty after this children = [])
// take over.
this._innerHTML = v;
this.children = [];
this._textContent = '';
},
get isConnected() {
// In real DOM this checks attachment to the document; for the
// test harness we approximate via the parent chain. After
@@ -304,17 +333,28 @@ function makeEl(tag) {
this.parent = null;
},
querySelectorAll(selector) {
// Only supports the literal "pre code.language-mermaid"
// selector that postRenderMermaid uses.
// Supports the two selectors the post-render passes use:
// "pre code.language-mermaid" (postRenderMermaid)
// "pre code[class*='language-']" (postRenderHljs)
const out = [];
const wantsMermaid = selector === "pre code.language-mermaid";
function matchesLangAttr(el) {
for (const cls of el._classes) {
if (cls.startsWith('language-')) return true;
}
return false;
}
function walk(node) {
for (const c of (node.children || [])) {
if (
const isCodeInPre =
c.tagName === 'CODE' &&
c.parent && c.parent.tagName === 'PRE' &&
c._classes.has('language-mermaid')
) {
out.push(c);
c.parent && c.parent.tagName === 'PRE';
if (isCodeInPre) {
if (wantsMermaid) {
if (c._classes.has('language-mermaid')) out.push(c);
} else if (matchesLangAttr(c)) {
out.push(c);
}
}
walk(c);
}
@@ -352,6 +392,20 @@ global.mermaid = {
},
};
// hljs stub. highlightElement mutates the element in place: replaces
// innerHTML with a deterministic synthetic span keyed by the source,
// and adds the hljs class same surface postRenderHljs depends on.
// hljsHighlightCallCount lets tests assert "ran N times" semantics.
let hljsHighlightCallCount = 0;
global.hljs = {
configure: () => {},
highlightElement: (el) => {
hljsHighlightCallCount++;
el._classes.add('hljs');
el._innerHTML = '<span class="hljs-tok">' + el._textContent + '</span>';
},
};
vm.runInThisContext(fs.readFileSync(%(utils)s, 'utf8'));
vm.runInThisContext(fs.readFileSync(%(renderer)s, 'utf8'));
@@ -514,7 +568,7 @@ def test_mermaid_cache_evicts_oldest_at_cap() -> None:
scenario = """
const cap = _MERMAID_CACHE_MAX;
for (let i = 0; i < cap + 5; i++) {
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
}
process.stdout.write(JSON.stringify({
size: _mermaidSvgCache.size,
@@ -536,10 +590,10 @@ def test_mermaid_overwrite_does_not_evict() -> None:
const cap = _MERMAID_CACHE_MAX;
// Fill exactly to cap.
for (let i = 0; i < cap; i++) {
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
}
// Overwrite an existing entry must not evict src-0.
_cacheMermaidEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null});
_cacheFifoEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null}, cap);
process.stdout.write(JSON.stringify({
size: _mermaidSvgCache.size,
hasOldest: _mermaidSvgCache.has('src-0'),
@@ -558,8 +612,8 @@ def test_mermaid_cache_cleared_on_init() -> None:
the rendered output depends on themeVariables which change
on init."""
scenario = """
_cacheMermaidEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null});
_cacheMermaidEntry(_mermaidErrorCache, 'src-bad', 'old error');
_cacheFifoEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null}, _MERMAID_CACHE_MAX);
_cacheFifoEntry(_mermaidErrorCache, 'src-bad', 'old error', _MERMAID_CACHE_MAX);
_initMermaid();
process.stdout.write(JSON.stringify({
svgSize: _mermaidSvgCache.size,
@@ -624,3 +678,598 @@ def test_streaming_render_invokes_mermaid_post_render() -> None:
"_streamingRenderApply must call postRenderMermaid for "
"progressive diagram rendering during streaming"
)
# ---------------------------------------------------------------------------
# _normalizeMermaidSource — autoquote labels with bare shape-delimiter
# chars. Mermaid rejects unquoted ( ) [ ] { } inside other labels with
# a "got 'PS'" parse error (paren-start in shape context). The two
# diagrams in the screenshot regression case are encoded here verbatim.
# ---------------------------------------------------------------------------
def _run_normalize(source: str) -> str:
"""Drive _normalizeMermaidSource against the JS harness and return
its output. The function is pure, so no container / mermaid stub
setup is required."""
scenario = f"""
const input = {json.dumps(source)};
const output = _normalizeMermaidSource(input);
process.stdout.write(JSON.stringify({{ output: output }}));
"""
out = _run_mermaid_scenario(scenario)
return str(out["output"])
# Diagram 1 from the screenshot regression — unquoted edge labels with
# parens and <br/> markers. Mermaid rejects both edge labels with
# "got 'PS'"; quoting them resolves it.
_SCREENSHOT_DIAGRAM_1_IN = (
"flowchart LR\n"
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
" -->|22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)|"
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
" B -->|13 jasl patches<br/>(Python only:<br/>tunings, kernels,"
"<br/>warmup, etc.)|"
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
)
_SCREENSHOT_DIAGRAM_1_OUT = (
"flowchart LR\n"
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
' -->|"22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)"|'
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
' B -->|"13 jasl patches<br/>(Python only:<br/>tunings, kernels,'
'<br/>warmup, etc.)"|'
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
)
# Diagram 2 from the screenshot regression — unquoted RECTANGLE node
# label `D[untouched<br/>(.so, _version.py,<br/>install-vendored)]`.
# Same parser failure mode; quoting the bracket label fixes it.
_SCREENSHOT_DIAGRAM_2_IN = (
"flowchart LR\n"
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
" B -->|not in archive| D[untouched<br/>(.so, _version.py,"
"<br/>install-vendored)]\n"
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
)
_SCREENSHOT_DIAGRAM_2_OUT = (
"flowchart LR\n"
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
' B -->|not in archive| D["untouched<br/>(.so, _version.py,'
'<br/>install-vendored)"]\n'
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
)
@pytest.mark.parametrize(
("source", "expected"),
[
(_SCREENSHOT_DIAGRAM_1_IN, _SCREENSHOT_DIAGRAM_1_OUT),
(_SCREENSHOT_DIAGRAM_2_IN, _SCREENSHOT_DIAGRAM_2_OUT),
],
)
def test_mermaid_autoquote_fixes_screenshot_diagrams(source: str, expected: str) -> None:
"""The two exact diagrams from the screenshot regression. If
these stop being rewritten with quoted labels, mermaid will
again reject them with `Expecting ... got 'PS'` during live
streaming."""
assert _run_normalize(source) == expected
@pytest.mark.parametrize(
"source",
[
# Clean diagram — no shape delimiters in any label.
"graph TD\n A[foo] --> B[bar]",
# Edge label with no special chars.
"A --> B\nA -->|plain text| B",
# Already-correctly-quoted node label.
'A["already (quoted)"] --> B',
# Already-correctly-quoted edge label.
'A -->|"already (quoted)"| B',
# Cylinder shape — inner () is part of the shape syntax.
"A[(database)] --> B",
# Subroutine shape — inner [] is part of the shape syntax.
"A[[subroutine]] --> B",
# Trapezoid shape — inner / is part of the shape syntax.
"A[/trapezoid/] --> B",
# Reverse trapezoid.
"A[\\trap\\] --> B",
# Mermaid directive — braces here are config, not a label.
'%%{init: {"theme": "dark"}}%%\ngraph TD\n A --> B',
# <br/> tags on their own don't trip quoting.
"A[line1<br/>line2] --> B",
# Sequence diagram — different grammar; we only target labels
# in shape/edge syntax that match the regex anchors.
"sequenceDiagram\n A->>B: hello",
],
)
def test_mermaid_autoquote_leaves_valid_source_alone(source: str) -> None:
"""The autoquoter must not rewrite syntactically valid Mermaid —
a false positive here would break a working diagram. Each case
covers a syntax form whose delimiters are intentional and must
not be wrapped."""
assert _run_normalize(source) == source
def test_mermaid_autoquote_edge_label_with_parens() -> None:
"""Bare-parens edge label gets wrapped. The bare `(` would
otherwise re-enter Mermaid's shape parser."""
src = "A -->|note (with parens)| B"
assert _run_normalize(src) == 'A -->|"note (with parens)"| B'
def test_mermaid_autoquote_node_label_with_parens() -> None:
"""Bare-parens node label gets wrapped."""
src = "D[label (foo, bar)]"
assert _run_normalize(src) == 'D["label (foo, bar)"]'
def test_mermaid_autoquote_node_label_with_braces() -> None:
"""Bare-braces in a rectangle label get wrapped. (Diamond {}
shapes are left alone only single-bracket [] labels are
rewritten.)"""
src = "A[config {key: value}]"
assert _run_normalize(src) == 'A["config {key: value}"]'
def test_mermaid_autoquote_preserves_br_tag_with_parens() -> None:
"""`<br/>` inside a label that also has parens stays — only the
quoting needs to be added around the whole label."""
src = "A[line1<br/>(line2)] --> B"
assert _run_normalize(src) == 'A["line1<br/>(line2)"] --> B'
def test_mermaid_autoquote_skips_label_with_internal_quote() -> None:
"""If a label contains a literal `"`, wrapping would produce
nested unescaped quotes. The autoquoter must punt leaving the
parse error to surface, rather than silently producing a worse
one."""
src = 'A[he said "hi" (lol)]'
assert _run_normalize(src) == src
def test_mermaid_autoquote_multiple_edges_on_one_line() -> None:
"""Both edge labels on a single line get rewritten independently."""
src = "A -->|first (paren)| B -->|second (paren)| C"
expected = 'A -->|"first (paren)"| B -->|"second (paren)"| C'
assert _run_normalize(src) == expected
def test_mermaid_autoquote_normalized_source_hits_cache() -> None:
"""The SVG cache keys on the normalized source — same malformed
input that the LLM streamed earlier still hits the cache on
re-render rather than re-invoking mermaid.render every tick."""
bad = "A[label (with parens)] --> B"
scenario = (
_build_mermaid_container_js([bad])
+ _MERMAID_DRAIN_JS
+ """
postRenderMermaid(container);
setTimeout(() => setTimeout(() => {
const container2 = buildContainer(sources);
postRenderMermaid(container2);
setTimeout(() => {
process.stdout.write(JSON.stringify({
renderCalls: renderCallCount,
normalized: container.children[0]._attrs['data-mermaid-source'],
}));
}, 0);
}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["renderCalls"] == 1, "second render bypassed the cache"
assert out["normalized"] == 'A["label (with parens)"] --> B'
def test_mermaid_normalize_memo_populates_on_first_call() -> None:
"""First postRenderMermaid call populates _mermaidNormalizeCache
with a rawnormalized entry. A second call on identical raw
textContent then hits the memo (size stays at 1, no second
normalize call), which is the perf-1 fix avoids re-running
split + per-line regex per rAF tick when the diagram hasn't
changed."""
bad = "A[label (with parens)] --> B"
scenario = (
_build_mermaid_container_js([bad])
+ _MERMAID_DRAIN_JS
+ """
postRenderMermaid(container);
const sizeAfterFirst = _mermaidNormalizeCache.size;
const cachedNorm = _mermaidNormalizeCache.get(sources[0]);
// Re-render on a fresh container with the same source.
const container2 = buildContainer(sources);
postRenderMermaid(container2);
setTimeout(() => setTimeout(() => {
process.stdout.write(JSON.stringify({
sizeAfterFirst: sizeAfterFirst,
cachedNorm: cachedNorm,
sizeAfterSecond: _mermaidNormalizeCache.size,
}));
}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["sizeAfterFirst"] == 1, "first call didn't populate normalize memo"
assert out["cachedNorm"] == 'A["label (with parens)"] --> B'
assert out["sizeAfterSecond"] == 1, (
"second call added a new entry — memo missed on identical source"
)
def test_mermaid_normalize_memo_is_consulted_before_normalize() -> None:
"""Pre-seed _mermaidNormalizeCache with a sentinel value for a
raw source. postRenderMermaid must use the sentinel rather than
re-running _normalizeMermaidSource. Catches a regression where
the memo gets populated but the lookup path is skipped."""
bad = "A[label (with parens)] --> B"
sentinel = "SENTINEL_FROM_MEMO --> X"
raw_js = json.dumps(bad)
sentinel_js = json.dumps(sentinel)
scenario = (
_build_mermaid_container_js([bad])
+ _MERMAID_DRAIN_JS
+ f"""
_mermaidNormalizeCache.set({raw_js}, {sentinel_js});
postRenderMermaid(container);
setTimeout(() => setTimeout(() => {{
process.stdout.write(JSON.stringify({{
sourceAttr: container.children[0]._attrs['data-mermaid-source'],
}}));
}}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["sourceAttr"] == sentinel, (
"postRenderMermaid bypassed the normalize memo and re-ran normalize"
)
def test_mermaid_normalize_memo_distinct_sources_cache_separately() -> None:
"""Two distinct raw sources produce two memo entries. Confirms
the memo keys on raw textContent, not on something coarser like
container identity."""
bad1 = "A[label (with parens)] --> B"
bad2 = "C[other (label)] --> D"
scenario = (
_build_mermaid_container_js([bad1, bad2])
+ _MERMAID_DRAIN_JS
+ """
postRenderMermaid(container);
setTimeout(() => setTimeout(() => {
process.stdout.write(JSON.stringify({
size: _mermaidNormalizeCache.size,
hasBad1: _mermaidNormalizeCache.has(sources[0]),
hasBad2: _mermaidNormalizeCache.has(sources[1]),
}));
}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["size"] == 2
assert out["hasBad1"] is True
assert out["hasBad2"] is True
# ---------------------------------------------------------------------------
# Code-fence pairing — close requires \n / EOS, content can't cross
# another close-pattern. Repros the streaming bug where ```mermaid +
# later ```python were paired by the regex, handing mermaid a
# truncated source.
# ---------------------------------------------------------------------------
def _render_md(source: str) -> str:
"""Drive renderMarkdown against the JS harness and return the
rendered HTML. The function is a pure string transform; no DOM
container scaffolding is required."""
scenario = f"""
const input = {json.dumps(source)};
const output = renderMarkdown(input);
process.stdout.write(JSON.stringify({{ output: output }}));
"""
out = _run_mermaid_scenario(scenario)
return str(out["output"])
_FENCE = "```"
def test_fence_partial_open_emits_no_code_block() -> None:
"""While a fence is still open and there's no other ``` later in
the buffer, no <code> block is emitted the open fence stays as
plain markdown text until the real close arrives."""
src = "Intro\n" + _FENCE + 'mermaid\nA["x"] -->|note (with parens)| B["y"]\nstill streaming'
html = _render_md(src)
assert "<code" not in html, f"open fence should not emit <code> mid-stream: {html!r}"
def test_fence_partial_with_later_open_does_not_pair_wrongly() -> None:
"""Before the fence-pair fix: an unclosed ```mermaid followed by
a ```python (also unclosed) would have paired up as
<code class=mermaid>...</code>python..., handing mermaid a
truncated source. With the new regex, neither fence emits a
block until its OWN closing line arrives."""
src = "Intro\n" + _FENCE + "mermaid\nA --> B\n" + _FENCE + 'python\nprint("hi")'
html = _render_md(src)
assert 'class="language-mermaid"' not in html, (
f"mermaid fence should not emit while open: {html!r}"
)
assert 'class="language-python"' not in html, (
f"python fence should not emit while open: {html!r}"
)
def test_fence_close_paired_with_next_open_is_rejected() -> None:
"""Repro of the live-streaming failure: mermaid fence open, then
```python opens and ``` closes the python block. Without the
fix, the regex paired mermaid's open with python's *open* (or
backtracked all the way to python's close), producing
<code class=mermaid>truncated</code>. With the fix mermaid stays
open (content can't cross another \\1 run; close must be at line
boundary) and only python's pair matches."""
src = (
"Intro\n"
+ _FENCE
+ 'mermaid\nA["x"] -->|note (with parens)| B["y"]\n'
+ _FENCE
+ 'python\nprint("hi")\n'
+ _FENCE
)
html = _render_md(src)
assert 'class="language-mermaid"' not in html, f"mermaid fence misparing reintroduced: {html!r}"
assert 'class="language-python"' in html, f"python fence on its own should match: {html!r}"
def test_fence_closed_emits_code_block() -> None:
"""Baseline: a properly closed fence with its close on its own
line emits the <code> block as expected the anchor doesn't
break the normal case."""
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + "\nAfter"
html = _render_md(src)
assert 'class="language-python"' in html
assert "import os" in html
def test_fence_close_at_end_of_buffer_emits() -> None:
"""A fence that closes at the very end of the buffer (no trailing
newline) still emits the anchor accepts end-of-string as a
valid line boundary, so the rehydration / static-render path
where the buffer ends cleanly at ``` still works."""
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE
html = _render_md(src)
assert 'class="language-python"' in html
assert "import os" in html
def test_fence_close_with_trailing_whitespace_emits() -> None:
"""A close followed only by spaces / tabs before \\n still counts
CommonMark allows trailing whitespace on the close line."""
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + " \nAfter"
html = _render_md(src)
assert 'class="language-python"' in html
# ---------------------------------------------------------------------------
# postRenderHljs — progressive syntax highlighting + source-keyed cache
# ---------------------------------------------------------------------------
def _build_hljs_container_js(blocks: list[tuple[str, str]]) -> str:
"""Build a container with <pre><code class="language-LANG"> blocks.
``blocks`` is a list of ``(language, source)`` tuples the language
becomes the ``language-X`` class, the source becomes textContent."""
arr = "[" + ", ".join(f"[{json.dumps(lang)}, {json.dumps(src)}]" for lang, src in blocks) + "]"
return f"""
function buildHljsContainer(blocks) {{
const container = document.createElement('div');
for (const [lang, src] of blocks) {{
const pre = document.createElement('pre');
const code = document.createElement('code');
code.classList.add('language-' + lang);
code.textContent = src;
pre.appendChild(code);
container.appendChild(pre);
}}
return container;
}}
const blocks = {arr};
const container = buildHljsContainer(blocks);
"""
def test_hljs_cache_hit_skips_highlight_call() -> None:
"""Two postRenderHljs calls on identical source must invoke
hljs.highlightElement exactly once the second call hits the
cache and applies the stored markup synchronously. Mirrors the
mermaid SVG-cache invariant that lets streamingRender fire on
every rAF tick without re-tokenizing every code block."""
scenario = (
_build_hljs_container_js([("python", "import os")])
+ """
postRenderHljs(container);
const container2 = buildHljsContainer(blocks);
postRenderHljs(container2);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
firstHtml: container.children[0].children[0]._innerHTML,
secondHtml: container2.children[0].children[0]._innerHTML,
secondHasHljsClass: container2.children[0].children[0]._classes.has('hljs'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 1, (
"second postRenderHljs call invoked highlightElement — cache miss"
)
assert out["cacheSize"] == 1
assert out["firstHtml"] == out["secondHtml"]
assert out["secondHasHljsClass"] is True
def test_hljs_distinct_sources_highlight_independently() -> None:
"""Distinct sources each trigger one highlight and cache one entry.
Cache key includes the source string, not e.g. just the language."""
scenario = (
_build_hljs_container_js([("python", "import os"), ("python", "print('hi')")])
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 2
assert out["cacheSize"] == 2
def test_hljs_cache_separates_by_language() -> None:
"""Same source text under different language fences must NOT
collide in the cache language is part of the key. Otherwise a
`python` block of `foo` and a `ruby` block of `foo` would share
a single (wrongly-highlighted) cache entry."""
scenario = (
_build_hljs_container_js([("python", "foo"), ("ruby", "foo")])
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 2
assert out["cacheSize"] == 2
def test_hljs_skips_no_highlight_langs() -> None:
"""language-mermaid / language-text / language-plaintext etc. must
get the `nohighlight` class without invoking hljs.highlightElement.
Highlighting plaintext or mermaid source would be both wasteful
and ugly."""
scenario = (
_build_hljs_container_js(
[("mermaid", "graph TD\\nA-->B"), ("text", "plain"), ("plaintext", "p")]
)
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
mermaidNoHighlight: container.children[0].children[0]._classes.has('nohighlight'),
textNoHighlight: container.children[1].children[0]._classes.has('nohighlight'),
plaintextNoHighlight: container.children[2].children[0]._classes.has('nohighlight'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 0
assert out["cacheSize"] == 0
assert out["mermaidNoHighlight"] is True
assert out["textNoHighlight"] is True
assert out["plaintextNoHighlight"] is True
def test_hljs_terminal_lang_marks_pre_for_terminal_styling() -> None:
"""Shell-family languages (bash / sh / zsh / console / terminal)
must add the `code-terminal` class to the parent <pre>, so the
stylesheet can give them the terminal look-and-feel."""
scenario = (
_build_hljs_container_js([("bash", "echo hi")])
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
preHasTerminalClass: container.children[0]._classes.has('code-terminal'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 1
assert out["preHasTerminalClass"] is True
def test_hljs_cache_evicts_oldest_at_cap() -> None:
"""FIFO eviction at _HLJS_CACHE_MAX. Mirrors the mermaid cache —
prevents unbounded growth on long sessions with many distinct
code blocks."""
scenario = """
const cap = _HLJS_CACHE_MAX;
for (let i = 0; i < cap + 5; i++) {
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
}
process.stdout.write(JSON.stringify({
size: _hljsCache.size,
hasOldest: _hljsCache.has('key-0'),
hasNewest: _hljsCache.has('key-' + (cap + 4)),
}));
"""
out = _run_mermaid_scenario(scenario)
assert out["size"] == 64
assert out["hasOldest"] is False
assert out["hasNewest"] is True
def test_hljs_overwrite_does_not_evict() -> None:
"""Overwriting an existing key is an in-place update, not a new
insertion must not evict the oldest unrelated entry. Same
invariant as the mermaid cache."""
scenario = """
const cap = _HLJS_CACHE_MAX;
for (let i = 0; i < cap; i++) {
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
}
_cacheFifoEntry(_hljsCache, 'key-5', 'val-updated', cap);
process.stdout.write(JSON.stringify({
size: _hljsCache.size,
hasOldest: _hljsCache.has('key-0'),
updated: _hljsCache.get('key-5'),
}));
"""
out = _run_mermaid_scenario(scenario)
assert out["size"] == 64
assert out["hasOldest"] is True, "overwrite evicted oldest unnecessarily"
assert out["updated"] == "val-updated"
def test_post_render_markdown_invokes_hljs() -> None:
"""postRenderMarkdown is the public end-of-stream entry point and
must still run syntax highlighting after the postRenderHljs
refactor regression guard for the public API surface that
app.js / coordinator code already call."""
scenario = (
_build_hljs_container_js([("python", "import os")])
+ """
postRenderMarkdown(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
hasHljsClass: container.children[0].children[0]._classes.has('hljs'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 1
assert out["hasHljsClass"] is True
def test_streaming_render_invokes_hljs() -> None:
"""_streamingRenderApply must call postRenderHljs so closed code
fences appear progressively (syntax-highlighted) during streaming,
not only at stream_end via streamingRenderFinalize. The cache
keeps the per-tick cost down to a synchronous lookup."""
body = _RENDERER_JS.read_text(encoding="utf-8")
start = body.index("function _streamingRenderApply")
hljs_call = body.find("postRenderHljs(el)", start, start + 4000)
assert hljs_call != -1, (
"_streamingRenderApply must call postRenderHljs for progressive "
"syntax highlighting during streaming"
)
+6
View File
@@ -14,6 +14,12 @@ from turnstone.core.session import ChatSession
class NullUI:
"""UI adapter that discards all output."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
+43
View File
@@ -297,6 +297,49 @@ def test_extra_fields_ignored():
assert e.text == "hi"
def test_in_progress_snapshot_event_round_trip():
from turnstone.sdk.events import InProgressSnapshotEvent
payload = {
"type": "in_progress_snapshot",
"ws_id": "ws1",
"content": "Partial content...",
"reasoning": "Partial reasoning...",
}
e = ServerEvent.from_dict(payload)
assert isinstance(e, InProgressSnapshotEvent)
assert e.ws_id == "ws1"
assert e.content == "Partial content..."
assert e.reasoning == "Partial reasoning..."
def test_in_progress_snapshot_event_strips_internal_seq():
"""``_seq`` is server-internal plumbing — even if a stray copy
leaks through, ``from_dict`` must drop it (not a declared field)."""
from turnstone.sdk.events import InProgressSnapshotEvent
e = ServerEvent.from_dict(
{
"type": "in_progress_snapshot",
"ws_id": "ws1",
"content": "x",
"reasoning": "",
"_seq": 42,
}
)
assert isinstance(e, InProgressSnapshotEvent)
assert not hasattr(e, "_seq")
def test_state_change_event_round_trip():
from turnstone.sdk.events import StateChangeEvent
e = ServerEvent.from_dict({"type": "state_change", "ws_id": "ws1", "state": "thinking"})
assert isinstance(e, StateChangeEvent)
assert e.state == "thinking"
assert e.ws_id == "ws1"
def test_missing_type_defaults_to_base():
e = ServerEvent.from_dict({"ws_id": "ws1"})
assert type(e) is ServerEvent
+6
View File
@@ -70,6 +70,12 @@ class RecordingUI:
self.errors: list[str] = []
self.infos: list[str] = []
def on_turn_start(self):
self.events.append(("turn_start",))
def on_turn_committed(self):
self.events.append(("turn_committed",))
def on_thinking_start(self):
self.events.append(("thinking_start",))
+542 -3
View File
@@ -14,6 +14,12 @@ from turnstone.core.session import _IMAGE_EXTENSIONS, _IMAGE_SIZE_CAP, ChatSessi
class NullUI:
"""UI adapter that discards all output. Used for testing."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
@@ -452,6 +458,213 @@ class TestPlanExec:
assert messages[0]["content"] == ChatSession._PLAN_IDENTITY
# ---------------------------------------------------------------------------
# Tests — _exec_task (optional skill substitutes the hardcoded identity)
# ---------------------------------------------------------------------------
class TestTaskExec:
"""Tests for _exec_task: optional skill= replaces the default persona,
but operating guidance (one-shot, tool-use over narration, no follow-ups)
is always preserved."""
@staticmethod
def _capture_exec_messages(session, item):
"""Run _exec_task with _run_agent patched; return system message text."""
captured: dict = {}
def fake_run_agent(messages, **kwargs):
captured["messages"] = list(messages)
return "done"
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_task(item)
return captured["messages"][0]["content"]
def test_known_skill_renders_into_system_message(self, tmp_db) -> None:
"""Validated skill content (with template vars resolved) replaces
the default '# Task Agent' persona, but the operating guidance
(the numbered list) is preserved those are sub-agent semantics
that a persona should layer on top of, not replace.
Covers the full prepareexec round-trip so a future regression
in either half (skill not stored on the item, or exec ignoring it)
is caught."""
session = _make_session()
skill = {
"name": "research",
"content": "# Research Agent\nws={{ws_id}} model={{model}} node={{node_id}}",
}
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
# Item carries the minimized projection — name/content/risk_level
# only — not the raw prompt_templates row.
assert item["skill"] == {
"name": "research",
"content": skill["content"],
"risk_level": "",
}
assert item.get("needs_approval") is True
assert "skill: research" in item["header"]
sys_msg = self._capture_exec_messages(session, item)
# Skill persona rendered with template vars resolved
assert "# Research Agent" in sys_msg
assert f"ws={session._ws_id}" in sys_msg
assert f"model={session.model}" in sys_msg
# Default persona is gone — skill substitutes for it.
assert "# Task Agent" not in sys_msg
assert "autonomous task agent with full tool access" not in sys_msg
# Operating guidance survives regardless of skill.
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
def test_omitted_skill_uses_hardcoded_identity(self, tmp_db) -> None:
"""Regression guard: without skill=, the default '# Task Agent'
persona AND the operating guidance both appear verbatim.
Pins the no-skill path so the substitution branch can't
accidentally swallow the default case."""
session = _make_session()
item = session._prepare_task("c1", {"prompt": "do x"})
assert item["skill"] is None
assert "skill:" not in item["header"]
sys_msg = self._capture_exec_messages(session, item)
assert ChatSession._TASK_DEFAULT_IDENTITY in sys_msg
assert ChatSession._TASK_OPERATING_GUIDANCE in sys_msg
# Default-persona literals also present (sanity check on the constant).
assert "# Task Agent" in sys_msg
assert "autonomous task agent with full tool access" in sys_msg
@pytest.mark.parametrize("skill_value", ["", " ", "\t\n"])
def test_prepare_task_empty_or_whitespace_skill_treated_as_omitted(
self, tmp_db, skill_value
) -> None:
"""Documented contract: ``skill=""`` (and whitespace-only) behaves
identically to omitting the skill arg. LLMs sometimes echo empty
strings rather than omit the field; this pins the documented
behavior so a future refactor of the ``(args.get("skill") or "").strip()``
chokepoint can't quietly diverge."""
session = _make_session()
item = session._prepare_task("c1", {"prompt": "do x", "skill": skill_value})
assert item.get("needs_approval") is True
assert item["skill"] is None
assert "skill:" not in item["header"]
def test_prepare_task_unknown_skill_returns_error(self, tmp_db) -> None:
"""Unknown skill name → clean error item, no approval needed.
Skill validation lives in _prepare_task so an LLM passing a
bogus name fails fast at approval time rather than at exec."""
session = _make_session()
with patch("turnstone.core.session.get_skill_by_name", return_value=None):
item = session._prepare_task("c1", {"prompt": "do x", "skill": "ghost"})
assert item.get("needs_approval") is False
assert "unknown skill 'ghost'" in item["error"]
assert "skill(action='search')" in item["error"]
def test_prepare_task_disabled_skill_returns_error(self, tmp_db) -> None:
"""Disabled skill → distinct error, mirrors the enabled gate that
``_exec_skill(action='load')`` (session.py:8404) and skill-search
already apply. Distinct from the unknown-skill phrasing so the
LLM's recovery path can tell 'not found' from 'quarantined'."""
session = _make_session()
disabled_skill = {
"name": "retired",
"content": "# Retired",
"enabled": False,
}
with patch("turnstone.core.session.get_skill_by_name", return_value=disabled_skill):
item = session._prepare_task("c1", {"prompt": "do x", "skill": "retired"})
assert item.get("needs_approval") is False
assert "is disabled" in item["error"]
# Distinct wording from the unknown-skill error, so the LLM can
# tell them apart at recovery time.
assert "unknown skill" not in item["error"]
def test_prepare_task_high_risk_skill_surfaces_in_header(self, tmp_db, caplog) -> None:
"""High/critical risk skills surface the tier in the approval header
and emit a structured warning, mirroring the signal ``_load_skills``
emits for session-level skills (session.py:1336)."""
import logging
session = _make_session()
risky_skill = {
"name": "danger",
"content": "# Danger",
"enabled": True,
"risk_level": "critical",
}
with (
caplog.at_level(logging.WARNING, logger="turnstone.core.session"),
patch("turnstone.core.session.get_skill_by_name", return_value=risky_skill),
):
item = session._prepare_task("c1", {"prompt": "do x", "skill": "danger"})
assert item.get("needs_approval") is True
assert "skill: danger" in item["header"]
assert "risk: critical" in item["header"]
warning_seen = any("high_risk_skill" in r.getMessage() for r in caplog.records)
assert warning_seen, "expected task_agent.high_risk_skill warning"
def test_prepare_task_normal_risk_skill_omits_tier_from_header(self, tmp_db) -> None:
"""Header only surfaces high/critical — low/medium/safe skills don't
pollute the approval line."""
session = _make_session()
ok_skill = {
"name": "research",
"content": "# Research",
"enabled": True,
"risk_level": "low",
}
with patch("turnstone.core.session.get_skill_by_name", return_value=ok_skill):
item = session._prepare_task("c1", {"prompt": "do x", "skill": "research"})
assert "skill: research" in item["header"]
assert "risk:" not in item["header"]
def test_evaluate_intent_projects_skill_for_task_agent(self, tmp_db, monkeypatch) -> None:
"""Judge projection includes the skill name so heuristic arg_patterns
can match on it and the audit row records which persona was chosen.
Mirrors the long-standing ``spawn_workstream`` projection at
session.py:4603 without it, policy rules targeting risky
skills via ``task_agent`` silently no-op."""
session = _make_session()
fake_verdict = MagicMock()
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
fake_judge = MagicMock()
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
skill = {"name": "research", "content": "# Research", "enabled": True}
with patch("turnstone.core.session.get_skill_by_name", return_value=skill):
item = session._prepare_task("c1", {"prompt": "investigate X", "skill": "research"})
session._evaluate_intent([item])
fa = item["func_args"]
assert fa["skill"] == "research"
assert fa["prompt"] == "investigate X"
def test_evaluate_intent_projects_empty_skill_when_omitted(self, tmp_db, monkeypatch) -> None:
"""Symmetric regression guard: no-skill case projects skill="" so
the func_args shape is stable across both branches (the judge can
always read ``func_args["skill"]`` without a KeyError)."""
session = _make_session()
fake_verdict = MagicMock()
fake_verdict.to_dict.return_value = {"verdict_id": "v0", "tier": "heuristic"}
fake_judge = MagicMock()
fake_judge.evaluate.side_effect = lambda items, *_a, **_kw: [fake_verdict] * len(items)
monkeypatch.setattr(session, "_ensure_judge", lambda: fake_judge)
item = session._prepare_task("c1", {"prompt": "do x"})
session._evaluate_intent([item])
fa = item["func_args"]
assert fa["skill"] == ""
assert fa["prompt"] == "do x"
# ---------------------------------------------------------------------------
# Per-call model override on plan_agent / task_agent
# ---------------------------------------------------------------------------
@@ -498,9 +711,51 @@ class TestAgentModelOverride:
assert item.get("needs_approval") is False
assert "error" in item
assert "unknown model alias 'bogus'" in item["error"]
# The error guidance must list the available aliases so the LLM can retry.
for alias in ("default", "smart", "fast"):
# Error guidance lists the aliases the LLM may retry, intentionally
# excluding ``default`` — that alias is operator-only (see
# ``test_prepare_plan_default_model_rejected``). Surfacing it here
# would re-enable the per-role-override bypass even though the
# tool description hides it.
for alias in ("smart", "fast"):
assert alias in item["error"]
assert "default" not in item["error"]
def test_prepare_plan_default_model_rejected(self, tmp_db) -> None:
"""``model="default"`` is rejected even when the alias exists in
the registry bypasses the operator-configured ``plan_alias``."""
session = _make_session(registry=self._registry(), model_alias="default")
item = session._prepare_plan("c1", {"goal": "do x", "model": "default"})
assert item.get("needs_approval") is False
assert "error" in item
assert "'default' is not a selectable model alias" in item["error"]
assert "Omit `model=`" in item["error"]
def test_prepare_plan_default_model_rejected_with_whitespace(self, tmp_db) -> None:
"""The ``default`` rejection runs after ``strip()`` so leading/
trailing whitespace can't sneak the alias past the carve-out."""
session = _make_session(registry=self._registry(), model_alias="default")
item = session._prepare_plan("c1", {"goal": "do x", "model": " default "})
assert item.get("needs_approval") is False
assert "'default' is not a selectable model alias" in item["error"]
def test_prepare_plan_unknown_model_with_only_default_in_registry(self, tmp_db) -> None:
"""When the registry holds only the reserved ``default`` alias
(single-CLI-model back-compat), the unknown-alias error must say
'(no alternative aliases configured — omit `model=`)' not the
misleading '(no registry configured)' that suggests routing isn't
wired up at all."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={"default": ModelConfig("default", "x", "x", "m")},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
item = session._prepare_plan("c1", {"goal": "do x", "model": "bogus"})
assert item.get("needs_approval") is False
assert "unknown model alias 'bogus'" in item["error"]
assert "no alternative aliases configured" in item["error"]
assert "no registry configured" not in item["error"]
# ---- _prepare_task ----
@@ -520,6 +775,15 @@ class TestAgentModelOverride:
assert item.get("needs_approval") is False
assert "error" in item
assert "unknown model alias 'bogus'" in item["error"]
assert "default" not in item["error"]
def test_prepare_task_default_model_rejected(self, tmp_db) -> None:
"""Symmetric carve-out for task_agent — see
``test_prepare_plan_default_model_rejected``."""
session = _make_session(registry=self._registry(), model_alias="default")
item = session._prepare_task("c1", {"prompt": "do x", "model": "default"})
assert item.get("needs_approval") is False
assert "'default' is not a selectable model alias" in item["error"]
# ---- tool description rendering ----
@@ -538,8 +802,11 @@ class TestAgentModelOverride:
tool = self._agent_tool(session, name)
assert tool is not None, f"{name} missing from session tools"
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
for alias in ("default", "smart", "fast"):
for alias in ("smart", "fast"):
assert f"`{alias}`" in desc, f"alias {alias} missing from {desc!r}"
# ``default`` is intentionally hidden — see
# ``test_render_omits_default_alias_from_description``.
assert "`default`" not in desc
def test_render_no_op_without_registry(self, tmp_db) -> None:
"""No registry → leave the placeholder description untouched."""
@@ -570,6 +837,82 @@ class TestAgentModelOverride:
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
assert "`bigboi`" in desc
def test_render_omits_default_alias_from_description(self, tmp_db) -> None:
"""The ``default`` alias is filtered from the LLM-facing alias list.
Reading "default" as English ("use the default") and passing it
explicitly bypasses the operator-configured per-role plan_alias /
task_alias. The LLM should reach the per-role default by omitting
``model=`` instead.
"""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={
"default": ModelConfig("default", "x", "x", "m"),
"gh200": ModelConfig("gh200", "x", "x", "m"),
"opus-4.7": ModelConfig("opus-4.7", "x", "x", "m"),
},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
for name in ("plan_agent", "task_agent"):
tool = self._agent_tool(session, name)
assert tool is not None
desc = tool["function"]["parameters"]["properties"]["model"]["description"]
assert "`gh200`" in desc
assert "`opus-4.7`" in desc
assert "`default`" not in desc
def test_render_falls_back_to_base_when_only_default_alias(self, tmp_db) -> None:
"""Single-CLI-model registries (only ``default`` in registry) leave
the base description untouched the LLM sees ``"No alternative
aliases configured"`` rather than an empty alias list."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={"default": ModelConfig("default", "x", "x", "m")},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
plan_tool = self._agent_tool(session, "plan_agent")
assert plan_tool is not None
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
assert "No alternative aliases configured" in desc
def test_refresh_into_only_default_resets_to_base(self, tmp_db) -> None:
"""A reload that drops the registry to only ``default`` must clear
stale alias names from the previously-rendered tool descriptions
not return early and leave them in place."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
reg = ModelRegistry(
models={
"default": ModelConfig("default", "x", "x", "m"),
"smart": ModelConfig("smart", "x", "x", "m"),
"fast": ModelConfig("fast", "x", "x", "m"),
},
default="default",
)
session = _make_session(registry=reg, model_alias="default")
# Sanity: initial render carries the non-default aliases.
plan_tool = self._agent_tool(session, "plan_agent")
assert plan_tool is not None
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
assert "`smart`" in desc and "`fast`" in desc
# Reload the registry down to only ``default`` (admin removed
# every other model definition).
reg.reload({"default": ModelConfig("default", "x", "x", "m")}, "default")
session.refresh_agent_tool_schemas()
plan_tool = self._agent_tool(session, "plan_agent")
assert plan_tool is not None
desc = plan_tool["function"]["parameters"]["properties"]["model"]["description"]
assert "`smart`" not in desc, f"stale alias survived reload: {desc!r}"
assert "`fast`" not in desc, f"stale alias survived reload: {desc!r}"
assert "No alternative aliases configured" in desc
def test_module_level_constants_not_mutated(self, tmp_db) -> None:
"""Rendering must not pollute the module-level TOOLS list shared
across all sessions."""
@@ -1968,6 +2311,202 @@ class TestCoordinatorMemoryScope:
assert scopes == ["workstream", "user", "global"]
class TestMemoryToolAudit:
"""Mutating memory tool actions emit audit rows.
Closes the gap that masked the May 2026 vllm_fork_overlay_pattern
investigation: only the admin-console DELETE route emitted
``memory.delete``, so a long-running session whose memory was
deleted via the admin UI couldn't tell from logs alone whether the
row had been deleted out-of-band, never persisted, or was never
visible. Read actions (get/search/list) intentionally stay
un-audited auditing reads would multiply audit volume without
forensic value.
"""
@staticmethod
def _audit_rows(action: str) -> list[dict]:
from turnstone.core.storage._registry import get_storage
return get_storage().list_audit_events(action=action)
def test_save_new_emits_memory_save(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"call_1",
{
"action": "save",
"name": "fact_one",
"content": "alpha content",
"scope": "user",
"type": "reference",
},
)
assert "error" not in item
session._exec_memory(item)
rows = self._audit_rows("memory.save")
assert len(rows) == 1
row = rows[0]
assert row["user_id"] == "user-1"
assert row["resource_type"] == "memory"
assert row["resource_id"] # memory_id was populated
detail = json.loads(row["detail"])
assert detail["name"] == "fact_one"
assert detail["scope"] == "user"
assert detail["scope_id"] == "user-1"
assert detail["type"] == "reference"
assert detail["ws_id"] == "ws-1"
# The "create" path must NOT also stamp an update row.
assert self._audit_rows("memory.update") == []
def test_save_global_scope_emits_empty_scope_id(self, tmp_db):
"""Global memories have no scope_id — the audit row's detail
must still carry the key (with value ``""``) so a forensic
consumer can distinguish ``scope='global'`` from a row that
forgot to populate ``scope_id`` for a scoped write."""
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"call_1",
{
"action": "save",
"name": "fact_global",
"content": "shared content",
"scope": "global",
},
)
assert "error" not in item
session._exec_memory(item)
rows = self._audit_rows("memory.save")
assert len(rows) == 1
detail = json.loads(rows[0]["detail"])
assert detail["scope"] == "global"
assert detail["scope_id"] == ""
assert detail["ws_id"] == "ws-1"
def test_save_upsert_emits_memory_update(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
for content in ("first", "second"):
item = session._prepare_memory(
"call_x",
{
"action": "save",
"name": "fact_one",
"content": content,
"scope": "user",
"type": "reference",
},
)
session._exec_memory(item)
saves = self._audit_rows("memory.save")
updates = self._audit_rows("memory.update")
assert len(saves) == 1
assert len(updates) == 1
# Same memory_id on both rows — the update audits the row save created.
assert saves[0]["resource_id"] == updates[0]["resource_id"]
def test_delete_emits_memory_delete(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
save_item = session._prepare_memory(
"call_1",
{
"action": "save",
"name": "fact_one",
"content": "alpha",
"scope": "user",
"type": "reference",
},
)
session._exec_memory(save_item)
saved_memory_id = self._audit_rows("memory.save")[0]["resource_id"]
delete_item = session._prepare_memory(
"call_2",
{"action": "delete", "name": "fact_one", "scope": "user"},
)
_, msg = session._exec_memory(delete_item)
assert "Deleted memory" in msg
rows = self._audit_rows("memory.delete")
assert len(rows) == 1
# resource_id must point at the same row save audited — proves
# delete-by-name resolved to the right row before recording.
assert rows[0]["resource_id"] == saved_memory_id
detail = json.loads(rows[0]["detail"])
assert detail["name"] == "fact_one"
assert detail["scope"] == "user"
assert detail["type"] == "reference"
def test_delete_not_found_emits_no_audit(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
delete_item = session._prepare_memory(
"call_1",
{"action": "delete", "name": "no_such_mem", "scope": "user"},
)
_, msg = session._exec_memory(delete_item)
assert "not found" in msg
assert self._audit_rows("memory.delete") == []
def test_reads_emit_no_audit(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
session._exec_memory(
session._prepare_memory(
"call_save",
{
"action": "save",
"name": "fact_one",
"content": "alpha",
"scope": "user",
},
)
)
for spec in (
{"action": "get", "name": "fact_one", "scope": "user"},
{"action": "search", "query": "fact"},
{"action": "list"},
):
item = session._prepare_memory("call_read", spec)
assert "error" not in item
session._exec_memory(item)
# Only the save above should have audited.
save_count = len(self._audit_rows("memory.save"))
update_count = len(self._audit_rows("memory.update"))
delete_count = len(self._audit_rows("memory.delete"))
assert (save_count, update_count, delete_count) == (1, 0, 0)
def test_audit_failure_does_not_break_tool_call(self, tmp_db):
"""A blow-up inside record_audit must not propagate to the LLM.
Auditing is best-effort instrumentation; a storage hiccup that
prevents the audit row from landing must not also lose the
save/delete the user actually asked for.
"""
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"call_1",
{
"action": "save",
"name": "fact_one",
"content": "alpha",
"scope": "user",
},
)
with patch(
"turnstone.core.audit.record_audit",
side_effect=RuntimeError("audit storage exploded"),
):
_, msg = session._exec_memory(item)
assert "Saved memory 'fact_one'" in msg
# The save itself still landed.
from turnstone.core.memory import get_structured_memory_by_name
assert get_structured_memory_by_name("fact_one", "user", "user-1") is not None
class TestPerKindToolVariants:
"""Verify the ``kind_variants`` metadata applies per-kind tool overrides.
+327
View File
@@ -0,0 +1,327 @@
"""Tests for :meth:`ChatSession._format_backend_error`.
The helper turns bare backend-boundary exceptions (httpx ``ReadTimeout``,
OpenAI SDK ``APITimeoutError`` / ``APIConnectionError`` /
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``) into
operator-actionable messages that include the provider, base URL, and
model. We bind the method to lightweight stubs rather than constructing
a full :class:`ChatSession`: the helper only reads ``self.client``,
``self._provider``, ``self.model``, and ``self._model_alias``, so a
SimpleNamespace stub exercises the same surface without dragging in the
storage / prompt composition fixtures.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
import pytest
from turnstone.core.session import ChatSession
def _stub(
*,
base_url: str = "http://192.168.0.5:8000/v1",
provider_name: str = "openai-compatible",
model: str = "flatspark",
model_alias: str | None = "flatspark",
client_attr: str = "base_url",
) -> Any:
"""Build a minimal session-like stub for ``_format_backend_error``.
``client_attr`` selects which attribute on the client carries the
URL both ``base_url`` (OpenAI / Anthropic SDK public surface) and
``_base_url`` (httpx fallback) are exercised by the helper.
"""
client_kwargs: dict[str, Any] = {client_attr: base_url}
return SimpleNamespace(
client=SimpleNamespace(**client_kwargs),
_provider=SimpleNamespace(provider_name=provider_name),
model=model,
_model_alias=model_alias,
)
def _format(stub: Any, exc: BaseException) -> str | None:
"""Invoke the method as if on a real session — ``__func__`` skips
the descriptor protocol so we can pass any object as ``self``."""
return ChatSession._format_backend_error(stub, exc) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# Synthetic exception classes — class name is what the helper matches on,
# so we don't need real httpx / openai imports here.
# ---------------------------------------------------------------------------
# N818 (Error suffix on Exception names) is intentionally suppressed
# for the four classes below — they exist to impersonate httpx /
# Anthropic SDK exception class names verbatim, since the formatter
# matches by class name. Renaming them defeats the test.
class ReadTimeout(Exception): # noqa: N818
pass
class WriteTimeout(Exception): # noqa: N818
pass
class APITimeoutError(Exception):
pass
class ConnectError(Exception): # noqa: N818
pass
class ConnectTimeout(Exception): # noqa: N818
pass
class APIConnectionError(Exception):
pass
class NotFoundError(Exception):
pass
class AuthenticationError(Exception):
pass
class PermissionDeniedError(Exception):
pass
class RateLimitError(Exception):
pass
class SomeUnrelatedError(Exception):
"""Outside the recognised set — should fall through to ``None``."""
# ---------------------------------------------------------------------------
# Known categories — each branch produces an operator-actionable message
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("exc_cls", [ReadTimeout, WriteTimeout, APITimeoutError])
def test_timeout_message_names_backend_and_model(exc_cls):
msg = _format(_stub(), exc_cls())
assert msg is not None
assert "Backend timeout" in msg
assert exc_cls.__name__ in msg
assert "openai-compatible" in msg
assert "http://192.168.0.5:8000/v1" in msg
assert "model=flatspark" in msg
assert "wedged" in msg
@pytest.mark.parametrize("exc_cls", [ConnectError, ConnectTimeout, APIConnectionError])
def test_connect_message_says_unreachable(exc_cls):
msg = _format(_stub(), exc_cls("dial tcp: i/o timeout"))
assert msg is not None
assert "Backend unreachable" in msg
assert exc_cls.__name__ in msg
assert "http://192.168.0.5:8000/v1" in msg
# Raw exception text is preserved as a tail for grep-correlation.
assert "dial tcp: i/o timeout" in msg
def test_not_found_points_at_model_name_mismatch():
msg = _format(_stub(model="flatspark"), NotFoundError("model flatspark not found"))
assert msg is not None
assert "Backend reports model not loaded" in msg
assert "no model named 'flatspark'" in msg
assert "/v1/models" in msg # operator hint
@pytest.mark.parametrize("exc_cls", [AuthenticationError, PermissionDeniedError])
def test_auth_message_mentions_api_key(exc_cls):
msg = _format(_stub(), exc_cls("invalid api key"))
assert msg is not None
assert "Backend rejected credentials" in msg
assert "API key" in msg
def test_rate_limit_message():
msg = _format(_stub(), RateLimitError("limit exceeded"))
assert msg is not None
assert "Backend rate-limited" in msg
assert "limit exceeded" in msg
# ---------------------------------------------------------------------------
# Fall-through + degradation behaviour
# ---------------------------------------------------------------------------
def test_unknown_exception_returns_none():
assert _format(_stub(), SomeUnrelatedError("anything")) is None
def test_unknown_exception_value_error_returns_none():
assert _format(_stub(), ValueError("not a backend error")) is None
def test_trailing_slash_and_query_string_stripped():
msg = _format(
_stub(base_url="http://node-a:8000/v1/?api_key=secret&foo=1"),
ReadTimeout(),
)
assert msg is not None
assert "http://node-a:8000/v1" in msg
# Query string (which may carry credentials) is stripped before the
# message is built — sanitize_error_text is a second line of defence
# but the helper itself must not embed query params verbatim.
assert "api_key" not in msg
assert "secret" not in msg
def test_missing_provider_degrades_to_placeholder():
stub = _stub()
stub._provider = None
msg = _format(stub, ReadTimeout())
assert msg is not None
# No exception, no NoneType formatting leaking through.
assert "Backend timeout" in msg
assert "from ?" in msg or "openai-compatible" not in msg
def test_client_base_url_raises_degrades_gracefully():
class _BadClient:
@property
def base_url(self) -> str:
raise RuntimeError("boom")
stub = SimpleNamespace(
client=_BadClient(),
_provider=SimpleNamespace(provider_name="openai-compatible"),
model="flatspark",
_model_alias="flatspark",
)
msg = _format(stub, ReadTimeout())
assert msg is not None
assert "Backend timeout" in msg
# base_url accessor blew up — message still renders with placeholder.
assert "at ?" in msg
def test_httpx_underscore_base_url_fallback():
# httpx client carries ``_base_url`` on some versions instead of
# ``base_url`` — the helper checks both.
stub = _stub(base_url="http://alt-host:9000", client_attr="_base_url")
# SimpleNamespace exposes the attr; remove the public one so the
# fallback path is exercised.
delattr(stub.client, "base_url") if hasattr(stub.client, "base_url") else None
msg = _format(stub, ReadTimeout())
assert msg is not None
assert "http://alt-host:9000" in msg
# ---------------------------------------------------------------------------
# Integration with _record_fatal_error — original bare-class string is
# replaced by the enriched message when the exception type is recognised.
# ---------------------------------------------------------------------------
def _record_fatal_stub(ui: Any, captured: dict[str, str]) -> Any:
"""Build a stub for the ``_record_fatal_error`` integration tests.
``_record_fatal_error`` calls ``self._format_backend_error(...)``
internally, so the stub binds the unbound method to itself rather
than relying on Python's descriptor protocol (which only kicks in
when ``self`` is a real instance of the class)."""
stub = SimpleNamespace(
client=SimpleNamespace(base_url="http://192.168.0.5:8000/v1"),
_provider=SimpleNamespace(provider_name="openai-compatible"),
model="flatspark",
_model_alias="flatspark",
_ws_id="ws-test",
_has_persisted_error=False,
ui=ui,
_emit_state=lambda state: captured.setdefault("state", state),
)
stub._format_backend_error = lambda exc: ChatSession._format_backend_error(stub, exc)
return stub
def test_record_fatal_uses_enriched_message_for_known(monkeypatch):
"""End-to-end: a recognised exception flows through
``_record_fatal_error`` and the enriched text reaches both the UI
and the persist hook."""
captured: dict[str, str] = {}
def fake_persist(ws_id: str, msg: str) -> None:
captured["persist"] = msg
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
# Skip the credential-redaction module (and its module-level
# regex compile) by returning the input verbatim — the helper
# under test produces no credentials.
return text
import turnstone.core.memory as memory_mod
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
class _UI:
def __init__(self) -> None:
self.errors: list[str] = []
def on_error(self, msg: str) -> None:
self.errors.append(msg)
ui = _UI()
stub = _record_fatal_stub(ui, captured)
ChatSession._record_fatal_error(stub, ReadTimeout()) # type: ignore[arg-type]
assert ui.errors, "UI never received error"
assert "Backend timeout" in ui.errors[0]
assert "ReadTimeout" in ui.errors[0]
assert captured["persist"] == ui.errors[0]
assert captured["state"] == "error"
assert stub._has_persisted_error is True
def test_record_fatal_falls_back_for_unknown(monkeypatch):
"""An unrecognised exception keeps the legacy
``f"{type(exc).__name__}: {exc}"`` shape so we don't regress
existing call sites that grep on it."""
captured: dict[str, str] = {}
def fake_persist(ws_id: str, msg: str) -> None:
captured["persist"] = msg
def fake_sanitize(text: str, *, max_len: int = 1024) -> str:
return text
import turnstone.core.memory as memory_mod
monkeypatch.setattr(memory_mod, "persist_last_error", fake_persist)
monkeypatch.setattr(memory_mod, "sanitize_error_text", fake_sanitize)
class _UI:
def __init__(self) -> None:
self.errors: list[str] = []
def on_error(self, msg: str) -> None:
self.errors.append(msg)
ui = _UI()
stub = _record_fatal_stub(ui, captured)
ChatSession._record_fatal_error(stub, ValueError("plain old error")) # type: ignore[arg-type]
assert ui.errors == ["ValueError: plain old error"]
assert captured["persist"] == "ValueError: plain old error"
+648
View File
@@ -0,0 +1,648 @@
"""Tests for session-level ``replay_reasoning_to_model`` plumbing.
Phase 2 of optional reasoning persistence reads the per-model
``ModelConfig.replay_reasoning_to_model`` flag at the wire-build call
site and threads it through ``provider.create_streaming`` /
``provider.create_completion``. These tests pin:
1. The resolver helper (``ChatSession._resolve_replay_reasoning_to_model``)
walks the registry correctly and falls back to ``False`` (the
conservative default matching the migration server_default) when
the lookup fails.
2. The streaming wire-build call site at ``session.py:_try_stream``
actually passes the resolved flag down without this, the Phase
2 work is dead code (the strip-when-False predicate never fires).
3. The non-streaming wire-build call site at
``session.py:_utility_completion`` does the same.
Drives through the real ``ChatSession._resolve_replay_reasoning_to_model``
with a stub registry, then captures the kwarg passed to a mock provider
to verify the flow end-to-end.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests._session_helpers import make_session as _make_session
def _registry_with_flag(persist: bool = True, replay: bool = False) -> Any:
"""Stub registry returning a ModelConfig-shaped object with the
flags under test."""
return SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
surface_persisted_reasoning=persist,
replay_reasoning_to_model=replay,
)
)
class TestResolveReplayReasoningToModel:
"""Direct unit tests for the resolver."""
def test_returns_false_when_no_registry(self) -> None:
session = _make_session()
session._registry = None
session._model_alias = "anything"
assert session._resolve_replay_reasoning_to_model() is False
def test_returns_false_when_no_alias(self) -> None:
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = ""
assert session._resolve_replay_reasoning_to_model() is False
def test_returns_false_default(self) -> None:
session = _make_session()
session._registry = _registry_with_flag(replay=False)
session._model_alias = "claude-opus-4-7"
assert session._resolve_replay_reasoning_to_model() is False
def test_returns_true_when_flag_set(self) -> None:
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = "claude-opus-4-7"
assert session._resolve_replay_reasoning_to_model() is True
def test_explicit_alias_arg_overrides_default(self) -> None:
session = _make_session()
def per_alias(alias: str) -> Any:
return SimpleNamespace(
replay_reasoning_to_model=(alias == "needs-replay"),
)
session._registry = SimpleNamespace(get_config=per_alias)
session._model_alias = "primary"
# Default reads session._model_alias → False.
assert session._resolve_replay_reasoning_to_model() is False
# Explicit alias arg → True for "needs-replay".
assert session._resolve_replay_reasoning_to_model("needs-replay") is True
def test_returns_false_on_registry_exception(self) -> None:
session = _make_session()
def boom(alias: str) -> Any:
raise KeyError(alias)
session._registry = SimpleNamespace(get_config=boom)
session._model_alias = "missing"
# Conservative fallback — losing the strip is a UX nuisance,
# but accepting wire-side reasoning replay against an unknown
# operator preference is a worse default.
assert session._resolve_replay_reasoning_to_model() is False
def test_caps_none_preserves_back_compat(self) -> None:
# When ``caps`` is omitted, the resolver returns the operator
# flag unchanged — matching pre-PR behaviour for any caller
# that hasn't been updated to thread caps yet.
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = "claude-opus-4-7"
assert session._resolve_replay_reasoning_to_model() is True
assert session._resolve_replay_reasoning_to_model(caps=None) is True
def test_caps_supports_replay_true_passes_through(self) -> None:
from turnstone.core.providers._protocol import ModelCapabilities
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = "claude-opus-4-7"
caps = ModelCapabilities(supports_reasoning_replay=True)
assert session._resolve_replay_reasoning_to_model(caps=caps) is True
def test_caps_supports_replay_false_blocks_replay(self) -> None:
from turnstone.core.providers._protocol import ModelCapabilities
# Operator flipped replay=True but the model's capability
# advertises supports_reasoning_replay=False — AND-gate blocks
# replay so the strip predicate runs at the wire build.
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = "hypothetical-no-replay-claude"
caps = ModelCapabilities(supports_reasoning_replay=False)
assert session._resolve_replay_reasoning_to_model(caps=caps) is False
def test_caps_supports_replay_true_does_not_force_replay(self) -> None:
from turnstone.core.providers._protocol import ModelCapabilities
# Capability True but operator flag False — result must be
# False (the AND has to be False on either side).
session = _make_session()
session._registry = _registry_with_flag(replay=False)
session._model_alias = "claude-opus-4-7"
caps = ModelCapabilities(supports_reasoning_replay=True)
assert session._resolve_replay_reasoning_to_model(caps=caps) is False
class TestStreamingCallSitePassesFlag:
"""Pin that ``_try_stream`` actually passes the resolved flag to
``provider.create_streaming`` without this the Phase 2 work is
dead code at the call site."""
def test_replay_true_propagates_to_provider(self) -> None:
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = "claude-opus-4-7"
# Stub provider: capture the kwargs passed to create_streaming.
captured: dict[str, Any] = {}
def capture_streaming(**kwargs: Any) -> Any:
captured.update(kwargs)
return iter([])
mock_provider = MagicMock()
mock_provider.create_streaming = capture_streaming
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
session._try_stream(
client=MagicMock(),
model="claude-opus-4-7",
msgs=[{"role": "user", "content": "hi"}],
provider=mock_provider,
model_alias="claude-opus-4-7",
)
assert captured["replay_reasoning_to_model"] is True
def test_replay_false_propagates_to_provider(self) -> None:
session = _make_session()
session._registry = _registry_with_flag(replay=False)
session._model_alias = "claude-opus-4-7"
captured: dict[str, Any] = {}
def capture_streaming(**kwargs: Any) -> Any:
captured.update(kwargs)
return iter([])
mock_provider = MagicMock()
mock_provider.create_streaming = capture_streaming
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
session._try_stream(
client=MagicMock(),
model="claude-opus-4-7",
msgs=[{"role": "user", "content": "hi"}],
provider=mock_provider,
model_alias="claude-opus-4-7",
)
assert captured["replay_reasoning_to_model"] is False
def test_fallback_alias_uses_its_own_flag(self) -> None:
# When the primary fails and we fall back to an alias with a
# different flag, the flag MUST track the resolved alias —
# not the session's primary alias.
session = _make_session()
def per_alias(alias: str) -> Any:
return SimpleNamespace(
replay_reasoning_to_model=(alias == "fallback-with-replay"),
)
session._registry = SimpleNamespace(get_config=per_alias)
session._model_alias = "primary" # primary has replay=False
captured: dict[str, Any] = {}
def capture_streaming(**kwargs: Any) -> Any:
captured.update(kwargs)
return iter([])
mock_provider = MagicMock()
mock_provider.create_streaming = capture_streaming
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
session._try_stream(
client=MagicMock(),
model="fallback-model",
msgs=[{"role": "user", "content": "hi"}],
provider=mock_provider,
model_alias="fallback-with-replay",
)
# Resolved against the FALLBACK alias, not the session's primary.
assert captured["replay_reasoning_to_model"] is True
class TestSessionToWireBoundaryIntegration:
"""End-to-end integration: session._try_stream -> real
AnthropicProvider.create_streaming -> captured Anthropic SDK
boundary call. Verifies the strip-when-False predicate actually
fires at the wire payload, not just at the captured kwarg.
The bare-function-stub tests above (TestStreamingCallSitePassesFlag)
pin that ``_try_stream`` PASSES the flag; this test pins that the
real provider USES it. Together they catch:
- kwarg renamed at provider boundary -> stub-tests still pass,
this one fails on its real-provider assertion.
- _convert_messages stops reading the kwarg -> stub-tests still
pass, this one fails because the wire payload still carries
the thinking block.
- _try_stream stops calling create_streaming -> stub-tests fail
on the captured kwarg, this one fails because the SDK boundary
was never reached.
Drives through the real ``AnthropicProvider`` with a mock client
whose ``client.messages.stream`` is captured the smallest possible
surface that crosses the session->provider->wire boundary chain.
Negative-tested: temporarily reverting
``_anthropic.py:create_streaming``'s
``self._convert_messages(messages, replay_reasoning_to_model=...)``
call to drop the kwarg makes the wire payload carry the thinking
block again; ``test_replay_false_strips_thinking_at_wire`` then
fails with ``Strip predicate did not fire at wire boundary``.
Restoring the kwarg makes it pass confirming the test gates the
actual wire-build invariant rather than the captured kwarg.
"""
def _stub_anthropic_client(self) -> tuple[MagicMock, dict[str, object]]:
"""Build a mock Anthropic client + captured-kwargs dict.
``client.messages.stream(**kwargs)`` returns a context manager
whose ``__enter__`` yields an iterable of zero events enough
to satisfy the ``_iter_with_cleanup`` shape without exercising
actual streaming protocol.
"""
captured: dict[str, object] = {}
def stream(**kwargs: object) -> object:
captured.update(kwargs)
cm = MagicMock()
cm.__enter__ = MagicMock(return_value=iter([]))
cm.__exit__ = MagicMock(return_value=False)
return cm
client = MagicMock()
client.messages.stream = stream
return client, captured
def _drive_session_through_anthropic(
self,
replay_flag: bool,
msgs: list[dict[str, object]],
) -> dict[str, object]:
"""Run session._try_stream against a real AnthropicProvider with
the resolver pre-set to *replay_flag*. Returns the kwargs
dict that reached the (mocked) Anthropic SDK boundary.
"""
pytest.importorskip("anthropic")
from turnstone.core.providers._anthropic import AnthropicProvider
session = _make_session()
session._registry = _registry_with_flag(replay=replay_flag)
session._model_alias = "claude-opus-4-7"
client, captured = self._stub_anthropic_client()
real_provider = AnthropicProvider()
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
stream = session._try_stream(
client=client,
model="claude-opus-4-7",
msgs=msgs,
provider=real_provider,
model_alias="claude-opus-4-7",
)
# Iterate the stream to drain the (empty) generator and ensure
# _ensure_anthropic / convert / build_kwargs all ran.
list(stream)
return captured
def test_replay_false_strips_thinking_at_wire(self) -> None:
msgs: list[dict[str, object]] = [
{"role": "user", "content": "hello"},
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "secret reasoning", "signature": "s"},
{"type": "text", "text": "Final answer."},
],
},
{"role": "user", "content": "ack"},
]
captured = self._drive_session_through_anthropic(False, msgs)
# Anthropic SDK was called.
wire_msgs = captured.get("messages")
assert isinstance(wire_msgs, list), (
f"Expected messages= list at SDK boundary, got {captured}"
)
# Walk the wire payload — the thinking block must NOT be present
# in the assistant turn's content blocks.
assistant = next(m for m in wire_msgs if m["role"] == "assistant")
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
assert "thinking" not in block_types, (
f"Strip predicate did not fire at wire boundary: blocks={block_types}"
)
# Defense-in-depth: the secret reasoning text must not appear
# anywhere in the wire payload.
flat = repr(captured)
assert "secret reasoning" not in flat, "Reasoning text leaked into the SDK boundary payload"
def test_replay_true_preserves_thinking_at_wire(self) -> None:
msgs: list[dict[str, object]] = [
{"role": "user", "content": "hello"},
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "kept reasoning", "signature": "s"},
{"type": "text", "text": "Final answer."},
],
},
{"role": "user", "content": "ack"},
]
captured = self._drive_session_through_anthropic(True, msgs)
wire_msgs = captured.get("messages")
assert isinstance(wire_msgs, list)
assistant = next(m for m in wire_msgs if m["role"] == "assistant")
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
assert "thinking" in block_types, (
f"Replay-true did not preserve thinking at wire: blocks={block_types}"
)
def test_capability_false_strips_thinking_even_when_operator_flag_true(self) -> None:
# Mirror of the OpenAI Responses ``test_capability_false_omits_
# include_even_when_flag_true`` test below: operator flips
# replay=True but the model's capability advertises
# supports_reasoning_replay=False. AND-gate at the resolver
# blocks replay, so the strip predicate fires at the wire and
# the thinking block does NOT reach the SDK boundary.
pytest.importorskip("anthropic")
from turnstone.core.providers._anthropic import AnthropicProvider
from turnstone.core.providers._protocol import ModelCapabilities
msgs: list[dict[str, object]] = [
{"role": "user", "content": "hello"},
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{"type": "thinking", "thinking": "secret reasoning", "signature": "s"},
{"type": "text", "text": "Final answer."},
],
},
{"role": "user", "content": "ack"},
]
session = _make_session()
session._registry = _registry_with_flag(replay=True) # operator opted in
session._model_alias = "hypothetical-no-replay-claude"
caps = ModelCapabilities(supports_reasoning_replay=False)
client, captured = self._stub_anthropic_client()
real_provider = AnthropicProvider()
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
stream = session._try_stream(
client=client,
model="hypothetical-no-replay-claude",
msgs=msgs,
provider=real_provider,
capabilities=caps,
model_alias="hypothetical-no-replay-claude",
)
list(stream)
wire_msgs = captured.get("messages")
assert isinstance(wire_msgs, list), (
f"Expected messages= list at SDK boundary, got {captured}"
)
assistant = next(m for m in wire_msgs if m["role"] == "assistant")
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
assert "thinking" not in block_types, (
"Capability gate did not block replay: thinking block reached the wire "
f"despite supports_reasoning_replay=False (blocks={block_types})"
)
flat = repr(captured)
assert "secret reasoning" not in flat, (
"Reasoning text leaked into the SDK boundary payload despite capability gate"
)
class TestSessionToOpenAIResponsesBoundaryIntegration:
"""End-to-end integration: session._try_stream -> real
OpenAIResponsesProvider.create_streaming -> captured Responses
SDK boundary call. Mirrors the AnthropicProvider test above
but for the path-2 (Responses API) replay flow.
Pins the include= request kwarg + reasoning input-item emission
actually fire at the wire boundary when the operator flag and
model capability both allow.
"""
def _stub_responses_client(self) -> tuple[MagicMock, dict[str, object]]:
"""Mock OpenAI Responses client. ``client.responses.create``
captures kwargs and returns an empty stream iterator."""
captured: dict[str, object] = {}
def create(**kwargs: object) -> object:
captured.update(kwargs)
return iter([])
client = MagicMock()
client.responses.create = create
return client, captured
def _registry_with_reasoning_capability(
self, replay: bool = True, supports_replay: bool = True
) -> Any:
from turnstone.core.providers._protocol import ModelCapabilities
return SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
replay_reasoning_to_model=replay,
capabilities={}, # no overrides
),
_caps=ModelCapabilities(
context_window=400000,
supports_temperature=False,
reasoning_effort_values=("low", "medium", "high"),
default_reasoning_effort="medium",
supports_reasoning_replay=supports_replay,
),
)
def test_replay_true_adds_include_to_responses_request(self) -> None:
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
registry = self._registry_with_reasoning_capability(replay=True, supports_replay=True)
session = _make_session()
session._registry = registry
session._model_alias = "gpt-5"
client, captured = self._stub_responses_client()
real_provider = OpenAIResponsesProvider()
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
stream = session._try_stream(
client=client,
model="gpt-5",
msgs=[{"role": "user", "content": "hi"}],
provider=real_provider,
capabilities=registry._caps,
model_alias="gpt-5",
)
list(stream)
assert captured.get("include") == ["reasoning.encrypted_content"]
def test_replay_false_omits_include(self) -> None:
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
registry = self._registry_with_reasoning_capability(replay=False, supports_replay=True)
session = _make_session()
session._registry = registry
session._model_alias = "gpt-5"
client, captured = self._stub_responses_client()
real_provider = OpenAIResponsesProvider()
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
stream = session._try_stream(
client=client,
model="gpt-5",
msgs=[{"role": "user", "content": "hi"}],
provider=real_provider,
capabilities=registry._caps,
model_alias="gpt-5",
)
list(stream)
assert "include" not in captured
def test_capability_false_omits_include_even_when_flag_true(self) -> None:
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
# Operator flips replay=True but the model has
# supports_reasoning_replay=False (e.g. gpt-4o via Responses).
# Capability gate prevents the include= from being sent.
registry = self._registry_with_reasoning_capability(replay=True, supports_replay=False)
session = _make_session()
session._registry = registry
session._model_alias = "gpt-4o"
client, captured = self._stub_responses_client()
real_provider = OpenAIResponsesProvider()
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
stream = session._try_stream(
client=client,
model="gpt-4o",
msgs=[{"role": "user", "content": "hi"}],
provider=real_provider,
capabilities=registry._caps,
model_alias="gpt-4o",
)
list(stream)
assert "include" not in captured
def test_replay_true_emits_reasoning_input_item(self) -> None:
from turnstone.core.providers._openai_responses import OpenAIResponsesProvider
registry = self._registry_with_reasoning_capability(replay=True, supports_replay=True)
session = _make_session()
session._registry = registry
session._model_alias = "gpt-5"
client, captured = self._stub_responses_client()
real_provider = OpenAIResponsesProvider()
# Multi-turn conversation with stored reasoning on assistant turn.
msgs: list[dict[str, object]] = [
{"role": "user", "content": "explain"},
{
"role": "assistant",
"content": "Final answer.",
"_provider_content": [
{
"type": "reasoning",
"id": "r_xyz",
"summary": [{"type": "summary_text", "text": "I thought"}],
"encrypted_content": "blob",
}
],
},
{"role": "user", "content": "follow-up"},
]
with (
patch.object(session, "_get_active_tools", return_value=None),
patch.object(session, "_provider_extra_params", return_value=None),
patch.object(session, "_get_deferred_names", return_value=frozenset()),
patch.object(session, "_check_cancelled"),
):
stream = session._try_stream(
client=client,
model="gpt-5",
msgs=msgs,
provider=real_provider,
capabilities=registry._caps,
model_alias="gpt-5",
)
list(stream)
# Walk the wire input items — one of them must be the reasoning
# round-trip (id matches what we stored).
wire_input = captured.get("input")
assert isinstance(wire_input, list)
reasoning_items = [it for it in wire_input if it.get("type") == "reasoning"]
assert len(reasoning_items) == 1
assert reasoning_items[0]["id"] == "r_xyz"
assert reasoning_items[0]["encrypted_content"] == "blob"
class TestUtilityCompletionPassesFlag:
"""Non-streaming utility path (title gen, compaction, extraction) —
same plumbing requirement as streaming."""
def test_utility_completion_passes_resolved_flag(self) -> None:
from turnstone.core.providers._protocol import ModelCapabilities
session = _make_session()
session._registry = _registry_with_flag(replay=True)
session._model_alias = "claude-opus-4-7"
captured: dict[str, Any] = {}
def capture_completion(**kwargs: Any) -> Any:
captured.update(kwargs)
return SimpleNamespace(content="title", finish_reason="stop", usage=None)
mock_provider = MagicMock()
mock_provider.create_completion = capture_completion
session._provider = mock_provider
caps = ModelCapabilities(max_output_tokens=0, supports_reasoning_replay=True)
with (
patch.object(session, "_get_capabilities", return_value=caps),
patch.object(session, "_provider_extra_params", return_value=None),
):
session._utility_completion(
messages=[{"role": "user", "content": "summarize"}],
max_tokens=512,
temperature=0.3,
)
assert captured["replay_reasoning_to_model"] is True
+355
View File
@@ -0,0 +1,355 @@
"""Tests for ChatSession synthetic ``reasoning_text`` block stamping (Phase 3 path 3).
Path 3 covers OpenAI Chat Completions endpoints vLLM with
``--reasoning-parser``, llama.cpp with ``reasoning_format``, Gemini's
``/v1beta/openai/`` endpoint, and any other server that surfaces
``delta.reasoning_content`` Pydantic extras. These have no native
provider_blocks shape on the wire, so ``ChatSession._stream_response``
captures the streamed reasoning text into ``reasoning_parts`` and
``_maybe_synth_reasoning_block`` stamps it onto ``_provider_content``
as a synthetic ``{type: "reasoning_text"}`` block at the end of the
turn.
These tests pin:
1. The synthesizer fires only when no native blocks were emitted AND
reasoning was captured (Anthropic + OpenAI Responses bypass it).
2. ``source`` field is tagged with the active model's server_type
(informational; pulled from ``server_compat.server_type``).
3. ``OpenAIChatCompletionsProvider.extract_reasoning_text`` round-trips
the synthetic block on history rehydration.
4. The synthetic shape is NOT in ``ANTHROPIC_VALID_BLOCK_TYPES`` so
cross-model resumption (local-model Anthropic) falls through
cleanly to the text+tool_calls rebuild path.
"""
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
from tests._session_helpers import make_session as _make_session
from turnstone.core.providers._anthropic import (
ANTHROPIC_VALID_BLOCK_TYPES,
AnthropicProvider,
)
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
class TestMaybeSynthReasoningBlock:
"""Direct unit tests for ``ChatSession._maybe_synth_reasoning_block``."""
def test_no_synth_when_provider_blocks_present(self) -> None:
# Anthropic / OpenAI Responses path — native blocks already
# carry the reasoning, no synth needed.
session = _make_session()
existing = [{"type": "thinking", "thinking": "x"}]
out = session._maybe_synth_reasoning_block(existing, ["should not be added"])
assert out is existing
def test_no_synth_when_reasoning_parts_empty(self) -> None:
session = _make_session()
out = session._maybe_synth_reasoning_block([], [])
assert out == []
def test_no_synth_when_reasoning_parts_only_whitespace(self) -> None:
session = _make_session()
out = session._maybe_synth_reasoning_block([], [" ", "\n\t"])
assert out == []
def test_synth_creates_reasoning_text_block(self) -> None:
session = _make_session()
out = session._maybe_synth_reasoning_block([], ["thought ", "process"])
assert len(out) == 1
assert out[0]["type"] == "reasoning_text"
assert out[0]["text"] == "thought process"
def test_synth_omits_source_when_no_server_type(self) -> None:
session = _make_session()
# No registry / no server_compat → source field omitted.
out = session._maybe_synth_reasoning_block([], ["text"])
assert "source" not in out[0]
def test_synth_includes_source_when_server_type_resolvable(self) -> None:
session = _make_session()
session._registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
capabilities={"server_compat": {"server_type": "vllm"}},
)
)
session._model_alias = "qwen3-32b"
out = session._maybe_synth_reasoning_block([], ["text"])
assert out[0]["source"] == "vllm"
def test_synth_handles_registry_exception(self) -> None:
# _resolve_server_type silently returns "" on any lookup error
# — synth still fires but omits the source field.
class BrokenRegistry:
def get_config(self, alias: str) -> Any:
raise KeyError(alias)
session = _make_session()
session._registry = BrokenRegistry()
session._model_alias = "missing"
out = session._maybe_synth_reasoning_block([], ["text"])
assert out[0]["text"] == "text"
assert "source" not in out[0]
def test_synth_appends_when_provider_blocks_are_non_reasoning(self) -> None:
# GoogleProvider attaches raw tool_call dicts as provider_blocks
# on the finish chunk (for thought_signature round-trip). When
# the same turn streamed reasoning_delta (Gemini's reasoning_
# content extra), the synthesizer must APPEND the synthetic
# reasoning block rather than skip synthesis — otherwise the
# reasoning text is shown live but lost on page reload.
session = _make_session()
existing = [
{
"id": "call_1",
"type": "function",
"function": {"name": "search", "arguments": "{}"},
"thought_signature": "sig123",
}
]
out = session._maybe_synth_reasoning_block(existing, ["I should search"])
assert len(out) == 2
assert out[0] is existing[0] # tool_call fidelity block survives intact
assert out[1]["type"] == "reasoning_text"
assert out[1]["text"] == "I should search"
def test_no_synth_when_openai_responses_reasoning_already_present(self) -> None:
# OpenAI Responses native reasoning item — synth must NOT fire
# even though provider_blocks contains ALSO non-reasoning items
# (e.g. message blocks). The reasoning-bearing block satisfies
# the persistence contract on its own.
session = _make_session()
existing = [
{"type": "reasoning", "summary": [{"text": "openai reasoning"}]},
{"type": "message", "role": "assistant", "content": "answer"},
]
out = session._maybe_synth_reasoning_block(existing, ["live reasoning text"])
assert out is existing
def test_no_synth_when_non_reasoning_blocks_but_reasoning_parts_empty(self) -> None:
# Google tool_calls with no reasoning streamed — return as-is.
session = _make_session()
existing = [
{"id": "call_1", "type": "function", "function": {"name": "f", "arguments": "{}"}}
]
out = session._maybe_synth_reasoning_block(existing, [])
assert out is existing
class TestSyntheticBlockShapeContract:
"""The synthetic block shape MUST stay outside Anthropic's valid
block types so cross-model resumption falls through cleanly."""
def test_reasoning_text_not_in_anthropic_valid_types(self) -> None:
# If this assertion ever fails, the cross-model resumption
# safety story breaks: a synthetic block from a local-model
# session would reach Anthropic's wire as a malformed block.
assert "reasoning_text" not in ANTHROPIC_VALID_BLOCK_TYPES
def test_synthetic_block_falls_through_anthropic_shape_filter(self) -> None:
# Cross-model resumption regression: turn 1 was on a local
# model (synthetic block stamped), then the operator switched
# to Anthropic. The shape filter must reject the synthetic
# block and fall through to text+tool_calls rebuild.
provider = AnthropicProvider()
msg = {
"role": "assistant",
"content": "spoken answer",
"_provider_content": [
{"type": "reasoning_text", "text": "synth thought", "source": "vllm"},
],
}
_, converted = provider._convert_messages([msg])
assistant = next(m for m in converted if m["role"] == "assistant")
block_types = [b.get("type") for b in assistant["content"] if isinstance(b, dict)]
# Foreign block did NOT reach Anthropic's wire. Rebuilt from
# text only.
assert "reasoning_text" not in block_types
assert assistant["content"] == [{"type": "text", "text": "spoken answer"}]
class TestOpenAIChatExtractReasoningText:
"""``OpenAIChatCompletionsProvider.extract_reasoning_text`` reads
the synthetic block back out for UI rehydration."""
def test_reads_synthetic_reasoning_text_block(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [{"type": "reasoning_text", "text": "captured thought"}]
assert provider.extract_reasoning_text(blocks) == "captured thought"
def test_concatenates_multiple_blocks(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [
{"type": "reasoning_text", "text": "first"},
{"type": "reasoning_text", "text": "second"},
]
assert provider.extract_reasoning_text(blocks) == "first\nsecond"
def test_skips_other_block_types(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [
{"type": "thinking", "thinking": "anth"},
{"type": "reasoning", "summary": [{"text": "openai"}]},
{"type": "reasoning_text", "text": "chat"},
]
assert provider.extract_reasoning_text(blocks) == "chat"
def test_handles_empty_text_field(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [
{"type": "reasoning_text", "text": ""},
{"type": "reasoning_text", "text": "kept"},
]
assert provider.extract_reasoning_text(blocks) == "kept"
def test_handles_missing_text_field(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [
{"type": "reasoning_text"}, # no text
{"type": "reasoning_text", "text": "kept"},
]
assert provider.extract_reasoning_text(blocks) == "kept"
def test_returns_empty_for_no_synth_blocks(self) -> None:
provider = OpenAIChatCompletionsProvider()
blocks = [{"type": "thinking", "thinking": "x"}]
assert provider.extract_reasoning_text(blocks) == ""
class TestStreamResponseSynthBlockIntegration:
"""Integration test: drives a fake reasoning-emitting stream
through ``ChatSession._stream_response`` and asserts the
synthesizer wires up correctly. Pins the call site at
``session.py`` (where ``_maybe_synth_reasoning_block`` is invoked
on the assembled provider_blocks before stamping ``_provider_content``)
without this, a future refactor that drops the synthesizer call
would silently break path-3 capture (vLLM/llama.cpp/Gemini-compat
reasoning would be visible live but invisible on history reload).
"""
def _make_stream(self, content: str, reasoning: str) -> Any:
"""Build an iterator of StreamChunks that mimic a path-3
capture (reasoning_delta chunks, content chunks, no
provider_blocks emitted).
"""
from turnstone.core.providers._protocol import StreamChunk, UsageInfo
chunks = []
# Reasoning first (matches live SSE order).
if reasoning:
chunks.append(StreamChunk(reasoning_delta=reasoning, is_first=True))
# Content next.
if content:
chunks.append(
StreamChunk(
content_delta=content,
is_first=not reasoning,
)
)
# Final chunk with finish_reason + usage.
chunks.append(
StreamChunk(
finish_reason="stop",
usage=UsageInfo(prompt_tokens=10, completion_tokens=20, total_tokens=30),
)
)
return iter(chunks)
def test_stream_response_stamps_synth_block_when_path3_reasoning_captured(
self,
) -> None:
"""Drive a fake stream emitting reasoning_delta chunks (no
native provider_blocks) through ``_stream_response``; assert
the resulting assistant_msg carries a synthetic reasoning_text
block stamped onto ``_provider_content``."""
session = _make_session()
# No registry → source field omitted from synth block.
stream = self._make_stream(content="Final answer.", reasoning="path-3 reasoning")
msg = session._stream_response(stream)
assert msg["role"] == "assistant"
assert msg["content"] == "Final answer."
# Synthetic block should be stamped onto _provider_content.
provider_content = msg.get("_provider_content")
assert isinstance(provider_content, list)
assert len(provider_content) == 1
assert provider_content[0]["type"] == "reasoning_text"
assert provider_content[0]["text"] == "path-3 reasoning"
def test_stream_response_no_synth_when_no_reasoning_captured(self) -> None:
"""Stream emits only content (no reasoning_delta). No synth
block stamped _provider_content key absent on assistant_msg."""
session = _make_session()
stream = self._make_stream(content="just content", reasoning="")
msg = session._stream_response(stream)
assert msg["content"] == "just content"
# No synth block (and no native blocks either) → key absent.
assert "_provider_content" not in msg
def test_stream_response_synth_block_carries_source_when_server_type_resolvable(
self,
) -> None:
"""When the active model has server_compat.server_type set,
the synth block carries it as the ``source`` field."""
session = _make_session()
session._registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
capabilities={"server_compat": {"server_type": "vllm"}},
)
)
session._model_alias = "qwen3-32b"
stream = self._make_stream(content="answer", reasoning="reasoning text")
msg = session._stream_response(stream)
provider_content = msg.get("_provider_content")
assert isinstance(provider_content, list)
assert provider_content[0]["source"] == "vllm"
class TestResolveServerType:
"""Direct unit tests for the helper that pulls server_type from
the active model's capabilities dict."""
def test_returns_empty_when_no_registry(self) -> None:
session = _make_session()
session._registry = None
assert session._resolve_server_type() == ""
def test_returns_empty_when_no_alias(self) -> None:
session = _make_session()
session._registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(capabilities={})
)
session._model_alias = ""
assert session._resolve_server_type() == ""
def test_returns_server_type_when_present(self) -> None:
session = _make_session()
session._registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
capabilities={"server_compat": {"server_type": "llama.cpp"}}
)
)
session._model_alias = "local-model"
assert session._resolve_server_type() == "llama.cpp"
def test_returns_empty_when_server_compat_missing(self) -> None:
session = _make_session()
session._registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
capabilities={"context_window": 32768},
)
)
session._model_alias = "local-model"
assert session._resolve_server_type() == ""
def test_returns_empty_on_exception(self) -> None:
class BrokenRegistry:
def get_config(self, alias: str) -> Any:
raise RuntimeError("boom")
session = _make_session()
session._registry = BrokenRegistry()
session._model_alias = "x"
assert session._resolve_server_type() == ""
+598 -2
View File
@@ -167,8 +167,8 @@ def test_on_intent_verdict_persists_verdict_row() -> None:
}
with _patch_get_storage(storage):
ui.on_intent_verdict(verdict)
storage.create_intent_verdict.assert_called_once()
kwargs = storage.create_intent_verdict.call_args.kwargs
storage.upsert_intent_verdict.assert_called_once()
kwargs = storage.upsert_intent_verdict.call_args.kwargs
assert kwargs["verdict_id"] == "v1"
assert kwargs["ws_id"] == "ws-1"
assert kwargs["call_id"] == "c1"
@@ -352,6 +352,192 @@ def test_resolve_approval_stamps_all_pending_verdicts() -> None:
assert ui._last_verdict_decision == "denied"
# ---------------------------------------------------------------------------
# user_decision value space — pending / approved / denied / timeout
# / auto-approve reasons (policy / blanket / skill / always / auto_approve_tools).
# Guards the "user_decision is never empty for new rows" invariant.
# ---------------------------------------------------------------------------
def test_resolve_approval_timeout_kwarg_writes_timeout_value() -> None:
"""``resolve_approval(False, ..., timeout=True)`` writes
``user_decision="timeout"`` so the audit trail can distinguish a
passive timeout expiry from an active user denial the feedback
string used to carry this distinction but the column alone could not."""
storage = MagicMock()
ui = _make_ui()
with _patch_get_storage(storage):
ui.on_intent_verdict({"verdict_id": "v1", "call_id": "c1"})
with _patch_get_storage(storage):
ui.resolve_approval(False, "expired", timeout=True)
storage.update_intent_verdict.assert_any_call("v1", user_decision="timeout")
assert ui._last_verdict_decision == "timeout"
def test_resolve_approval_timeout_with_approved_raises() -> None:
"""``timeout=True`` is mutually exclusive with ``approved=True`` —
the combination would land a row whose audit column says
``"timeout"`` while the SSE event reports ``approved=True``. Fail
loud so the inconsistency can't ship silently."""
import pytest
ui = _make_ui()
with pytest.raises(ValueError, match="timeout"):
ui.resolve_approval(True, timeout=True)
def test_record_auto_approves_populates_reason_lookup() -> None:
"""``_record_auto_approves`` must seed
``_auto_approve_reasons[call_id]`` with the per-item reason so a
late-arriving LLM judge verdict can recover the auto-approve
reason via ``on_intent_verdict``."""
storage = MagicMock()
ui = _make_ui()
items = [
{
"call_id": "c-policy",
"func_name": "bash",
"auto_approved": True,
"auto_approve_reason": "policy",
},
{
"call_id": "c-blanket",
"func_name": "list_workstreams",
"auto_approved": True,
"auto_approve_reason": "blanket",
},
]
with _patch_get_storage(storage):
ui._record_auto_approves(items)
assert "c-policy" in ui._auto_approve_reasons
assert "c-blanket" in ui._auto_approve_reasons
assert ui._auto_approve_reasons["c-policy"][0] == "policy"
assert ui._auto_approve_reasons["c-blanket"][0] == "blanket"
def test_on_intent_verdict_consumes_auto_approve_reason() -> None:
"""A late LLM verdict for a previously auto-approved call_id picks
up the reason from ``_auto_approve_reasons``, stamps it on the
verdict before persist, and pops the entry so re-use isn't
possible. Closes the misdiagnosis bug where auto-approved tools
landed verdict rows with ``user_decision=""``."""
storage = MagicMock()
ui = _make_ui()
ui._auto_approve_reasons["c-x"] = ("auto_approve_tools", 0.0)
with _patch_get_storage(storage):
ui.on_intent_verdict({"verdict_id": "v-x", "call_id": "c-x"})
storage.upsert_intent_verdict.assert_called_once()
kwargs = storage.upsert_intent_verdict.call_args.kwargs
assert kwargs["user_decision"] == "auto_approve_tools"
# Consumed on read so the same call_id can't double-stamp later.
assert "c-x" not in ui._auto_approve_reasons
# Auto-stamped verdicts must NOT join _pending_verdicts — the
# row's final decision is already set; appending would let a
# later resolve_approval overwrite the auto-reason with the
# manual decision (real audit-trail clobber bug).
assert ui._pending_verdicts == []
def test_on_intent_verdict_auto_reason_survives_resolve_cycle() -> None:
"""Mixed-batch case: one tool was auto-approved (policy), another
needs manual approval. The LLM judge fires for the auto-approved
sibling DURING the manual-approval wait. The verdict must land
with ``user_decision="policy"`` and stay that way even after
``resolve_approval`` fires for the pending sibling the prior
bug was that the auto-stamped row got overwritten with
``"approved"``/``"denied"`` by the resolve path."""
storage = MagicMock()
ui = _make_ui()
ui._auto_approve_reasons["c-auto"] = ("policy", 0.0)
with _patch_get_storage(storage):
# LLM verdict fires for the auto-approved sibling.
ui.on_intent_verdict({"verdict_id": "v-auto", "call_id": "c-auto"})
# Now the pending sibling gets a verdict + manual resolve.
ui.on_intent_verdict({"verdict_id": "v-pending", "call_id": "c-pending"})
ui.resolve_approval(True, "looks good")
# Only the pending verdict should be UPDATEd to "approved" — the
# auto-stamped one stays "policy" via its INSERT.
update_calls = {
c.args[0]: c.kwargs.get("user_decision")
for c in storage.update_intent_verdict.call_args_list
}
assert update_calls == {"v-pending": "approved"}
# The auto verdict's INSERT carried the policy reason.
insert_calls = {
c.kwargs["verdict_id"]: c.kwargs["user_decision"]
for c in storage.upsert_intent_verdict.call_args_list
}
assert insert_calls["v-auto"] == "policy"
assert insert_calls["v-pending"] == "pending"
def test_persist_auto_approved_heuristic_verdicts_stamps_reason() -> None:
"""The auto-approve early-return branches in ``approve_tools`` used
to drop heuristic verdicts on the floor auditors couldn't tell
whether the judge ran or the call was simply silently auto-approved.
``_persist_auto_approved_heuristic_verdicts`` closes that gap and
stamps each verdict with the item's reason."""
storage = MagicMock()
ui = _make_ui()
items = [
{
"call_id": "c-1",
"auto_approved": True,
"auto_approve_reason": "blanket",
"_heuristic_verdict": {
"verdict_id": "v-1",
"call_id": "c-1",
"risk_level": "low",
"recommendation": "review",
},
},
# No _heuristic_verdict — skipped (judge didn't run for this item).
{"call_id": "c-2", "auto_approved": True, "auto_approve_reason": "blanket"},
# Not auto_approved — skipped (this helper only handles auto-approved).
{
"call_id": "c-3",
"_heuristic_verdict": {"verdict_id": "v-3", "call_id": "c-3"},
},
]
with _patch_get_storage(storage):
ui._persist_auto_approved_heuristic_verdicts(items)
storage.create_intent_verdicts_bulk.assert_called_once()
rows = storage.create_intent_verdicts_bulk.call_args.args[0]
assert len(rows) == 1
assert rows[0]["verdict_id"] == "v-1"
assert rows[0]["user_decision"] == "blanket"
def test_auto_approve_reasons_ttl_prune_drops_stale_entries() -> None:
"""Lazy TTL eviction at write time: entries older than
``_AUTO_APPROVE_REASON_TTL`` are pruned on the next
``_record_auto_approves`` call. Without this, a session with the
LLM judge disabled would accumulate entries that never get
consumed."""
import time as time_module
storage = MagicMock()
ui = _make_ui()
# Seed two stale entries (well past the TTL).
stale_ts = time_module.time() - ui._AUTO_APPROVE_REASON_TTL - 30.0
ui._auto_approve_reasons["c-stale-1"] = ("policy", stale_ts)
ui._auto_approve_reasons["c-stale-2"] = ("blanket", stale_ts)
items = [
{
"call_id": "c-fresh",
"auto_approved": True,
"auto_approve_reason": "skill",
"func_name": "bash",
}
]
with _patch_get_storage(storage):
ui._record_auto_approves(items)
# Stale entries pruned; only the fresh one remains.
assert "c-stale-1" not in ui._auto_approve_reasons
assert "c-stale-2" not in ui._auto_approve_reasons
assert "c-fresh" in ui._auto_approve_reasons
# ---------------------------------------------------------------------------
# Output guard persistence
# ---------------------------------------------------------------------------
@@ -872,3 +1058,413 @@ def test_concurrent_enqueue_and_listener_registration() -> None:
# intent survives optimization-mode assertion stripping.
assert not producer.is_alive()
assert all(not s.is_alive() for s in subscribers)
# ---------------------------------------------------------------------------
# Per-turn inflight buffers — SSE refresh-resume snapshot path
# ---------------------------------------------------------------------------
def test_on_content_token_writes_to_both_buffers() -> None:
"""``on_content_token`` writes to the multi-turn buffer (IDLE
piggyback) AND the per-turn inflight buffer (SSE snapshot)."""
ui = _make_ui()
ui.on_content_token("hello")
assert ui._ws_turn_content == ["hello"]
assert ui._ws_inflight_content == ["hello"]
assert ui._ws_inflight_seq == 1
def test_on_reasoning_token_writes_to_inflight_buffer_only() -> None:
"""Reasoning has no multi-turn IDLE piggyback — only the inflight
buffer + the seq counter."""
ui = _make_ui()
ui.on_reasoning_token("thinking...")
assert ui._ws_inflight_reasoning == ["thinking..."]
assert ui._ws_inflight_seq == 1
# Multi-turn buffer is content-only and untouched by reasoning.
assert ui._ws_turn_content == []
def test_inflight_seq_advances_on_every_emit_even_at_cap() -> None:
"""Cap-hit content tokens MUST advance ``_ws_inflight_seq``,
even though the buffer rejected the append. If seq stalled at
high-water-pre-cap, a subscriber registering AFTER the cap is
hit would capture ``snap_seq == stalled_seq`` and every
subsequent live token (also tagged with the stalled seq) would
be filter-dropped by the events handler silently losing the
rest of the stream. The cap is a buffer-size limit, not a
"stop streaming" signal."""
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
ui = _make_ui()
chunk = "x" * 1024
while ui._ws_inflight_content_size < _MAX_TURN_CONTENT_CHARS:
ui.on_content_token(chunk)
seq_at_cap = ui._ws_inflight_seq
# Cap-hit token: seq MUST advance (no buffer append, but the
# event still gets a fresh seq for the dedup filter).
ui.on_content_token(chunk)
assert ui._ws_inflight_seq == seq_at_cap + 1
# Buffer remains bounded — the cap-hit token is NOT in inflight.
assert ui._ws_inflight_content_size <= _MAX_TURN_CONTENT_CHARS + len(chunk)
def test_subscriber_after_cap_hit_receives_subsequent_tokens() -> None:
"""Regression for Copilot's cap+seq finding: a subscriber that
connects AFTER the inflight buffer is at cap must still receive
live tokens past the cap. Past-cap tokens are absent from
``snap.content`` (the snapshot text was truncated at cap) but
the live stream past them must NOT be filter-dropped."""
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
ui = _make_ui()
chunk = "x" * 1024
while ui._ws_inflight_content_size < _MAX_TURN_CONTENT_CHARS:
ui.on_content_token(chunk)
# Stream a few tokens PAST the cap before subscribing.
for _ in range(3):
ui.on_content_token(chunk)
lq, snap = ui.register_listener_with_in_progress_snapshot()
snap_seq = snap["seq"]
# Live token past cap.
ui.on_content_token(chunk)
ev = lq.get_nowait()
assert ev["type"] == "content"
# The critical invariant: seq advances per-emit, so the new
# event's _seq is strictly greater than the snap_seq the
# subscriber captured. Without this, the events handler's
# ``seq <= snap_seq`` filter would drop every token past the
# cap (silent token loss for refresh-past-cap).
assert ev["_seq"] > snap_seq, (
f"Token past cap has _seq={ev['_seq']} which is <= "
f"snap_seq={snap_seq} — would be silently dropped after a "
f"refresh past the cap."
)
def test_subscriber_after_reasoning_cap_hit_receives_subsequent_tokens() -> None:
"""Same invariant as content cap: reasoning subscribers past
cap must keep receiving live reasoning tokens."""
from turnstone.core.session_ui_base import _MAX_TURN_CONTENT_CHARS
ui = _make_ui()
chunk = "x" * 1024
while ui._ws_inflight_reasoning_size < _MAX_TURN_CONTENT_CHARS:
ui.on_reasoning_token(chunk)
for _ in range(3):
ui.on_reasoning_token(chunk)
lq, snap = ui.register_listener_with_in_progress_snapshot()
snap_seq = snap["seq"]
ui.on_reasoning_token(chunk)
ev = lq.get_nowait()
assert ev["type"] == "reasoning"
assert ev["_seq"] > snap_seq
def test_on_turn_committed_resets_inflight_after_commit() -> None:
"""``on_turn_committed`` fires immediately after each
``messages.append(assistant_msg)`` in the send loop. Without it,
the inflight buffer keeps the just-committed turn's content
during the post-commit tool-execution window and a refresh in
that window would show the assistant turn TWICE (history list
+ in_progress_snapshot)."""
ui = _make_ui()
ui.on_content_token("Just-finished turn ")
ui.on_reasoning_token("Reasoning for the turn ")
# Sanity: buffer is populated pre-commit.
assert ui._ws_inflight_content == ["Just-finished turn "]
assert ui._ws_inflight_reasoning == ["Reasoning for the turn "]
ui.on_turn_committed()
# Inflight content + reasoning reset; seq stays monotonic.
assert ui._ws_inflight_content == []
assert ui._ws_inflight_reasoning == []
# Multi-turn buffer is NOT reset by commit (it drains at idle).
assert ui._ws_turn_content == ["Just-finished turn "]
def test_inflight_snapshot_empty_during_post_commit_tool_window() -> None:
"""Models the user-reported bug: refresh during a tool-execution
window between commit and the next stream. Pre-fix: snapshot has
the just-committed turn's text → double-renders against history.
Post-fix: snapshot is empty no double-render. Seq stays
monotonic (carries the high-water mark across turn boundaries)."""
ui = _make_ui()
ui.on_content_token("Calling tool with these args: ")
seq_pre_commit = ui._ws_inflight_seq
ui.on_turn_committed() # session.py fires this after messages.append
# We're now in the tool-execution window. A reconnecting client
# would call register_listener_with_in_progress_snapshot.
_, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == ""
assert snap["reasoning"] == ""
# Seq did NOT reset — must remain monotonic across turns.
assert snap["seq"] == seq_pre_commit
def test_on_turn_start_resets_inflight_content_and_reasoning() -> None:
"""``on_turn_start`` clears the per-turn content + reasoning
buffers but does NOT touch the multi-turn ``_ws_turn_content``
(which the dashboard's IDLE-piggyback payload depends on) and
does NOT reset the seq counter (must remain monotonic across
turn boundaries see ``test_inflight_seq_monotonic_across_turn_boundaries``)."""
ui = _make_ui()
ui.on_content_token("turn-1 ")
ui.on_reasoning_token("reasoning-1 ")
multi_pre = list(ui._ws_turn_content)
multi_pre_size = ui._ws_turn_content_size
ui.on_turn_start()
assert ui._ws_inflight_content == []
assert ui._ws_inflight_content_size == 0
assert ui._ws_inflight_reasoning == []
assert ui._ws_inflight_reasoning_size == 0
# Multi-turn untouched.
assert ui._ws_turn_content == multi_pre
assert ui._ws_turn_content_size == multi_pre_size
def test_register_listener_with_in_progress_snapshot_empty() -> None:
ui = _make_ui()
lq, snap = ui.register_listener_with_in_progress_snapshot()
assert isinstance(lq, queue.Queue)
assert lq in ui._listeners
assert snap == {"content": "", "reasoning": "", "seq": 0}
def test_register_listener_with_in_progress_snapshot_populated() -> None:
ui = _make_ui()
ui.on_content_token("Hello, ")
ui.on_content_token("world!")
ui.on_reasoning_token("planning a greeting")
lq, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == "Hello, world!"
assert snap["reasoning"] == "planning a greeting"
# seq counts every successful append across BOTH buffers.
assert snap["seq"] == 3
# Listener is registered — later live tokens land in lq.
ui.on_content_token(" Goodbye.")
ev = lq.get_nowait()
assert ev["type"] == "content"
assert ev["text"] == " Goodbye."
assert ev["_seq"] == 4
def test_register_listener_with_in_progress_snapshot_only_inflight_not_multi_turn() -> None:
"""The snapshot reflects the in-progress turn only — anything
cleared by ``on_turn_start`` (a prior committed turn within the
same send) must NOT appear in the snapshot, even though the
multi-turn buffer still has it."""
ui = _make_ui()
ui.on_content_token("PRIOR_TURN ")
ui.on_turn_start() # commit boundary — inflight reset
ui.on_content_token("CURRENT")
_, snap = ui.register_listener_with_in_progress_snapshot()
assert snap["content"] == "CURRENT"
# Multi-turn buffer still has both turns (drives the IDLE piggyback).
assert "".join(ui._ws_turn_content) == "PRIOR_TURN CURRENT"
def test_seq_filter_dedup_round_trip() -> None:
"""End-to-end dedup invariant: every token appears exactly once
when reconstructing from snapshot + listener queue under live
writes that race the registration. Models the events handler."""
ui = _make_ui()
for ch in "abcde":
ui.on_content_token(ch)
lq, snap = ui.register_listener_with_in_progress_snapshot()
for ch in "fgh":
ui.on_content_token(ch)
reconstructed = snap["content"]
while True:
try:
ev = lq.get_nowait()
except queue.Empty:
break
if ev.get("_seq", 0) <= snap["seq"]:
continue
reconstructed += ev["text"]
assert reconstructed == "abcdefgh"
def test_seq_filter_drops_overlap_when_register_lands_after_writer() -> None:
"""Race: writer appends + emits while a second register snapshots
after the writer. The live event has _seq <= snap.seq must be
dropped to avoid double-render."""
ui = _make_ui()
# Register a first listener so the writer's enqueue lands somewhere.
lq1, _ = ui.register_listener_with_in_progress_snapshot()
ui.on_content_token("X")
# Second register snapshots AFTER the write — snap has "X" AND
# the writer's enqueue is in lq1.
_, snap2 = ui.register_listener_with_in_progress_snapshot()
assert snap2["content"] == "X"
# Drain lq1 with the filter against snap2.seq — duplicate dropped.
duped: list[str] = []
while True:
try:
ev = lq1.get_nowait()
except queue.Empty:
break
if ev.get("_seq", 0) <= snap2["seq"]:
continue
duped.append(ev["text"])
assert duped == []
def test_inflight_seq_monotonic_across_turn_boundaries() -> None:
"""Regression: a subscriber registered mid-turn-N must still
receive turn N+1's tokens. The seq counter is monotonic across
turn boundaries resetting it at on_turn_committed/on_turn_start
would silently drop turn N+1's first M tokens (M = the snap_seq
captured mid-turn-N) via the events handler's `seq <= snap_seq`
filter."""
ui = _make_ui()
# Turn N: stream tokens, register a listener mid-turn.
ui.on_content_token("turn-N tok1 ")
ui.on_content_token("turn-N tok2 ")
lq, snap = ui.register_listener_with_in_progress_snapshot()
snap_seq = snap["seq"]
assert snap_seq == 2
# Turn N completes, turn N+1 begins.
ui.on_turn_committed()
ui.on_turn_start()
# Turn N+1's first content token. With the q-1 fix, seq is
# monotonic (3), not reset to 1. The events handler's
# `seq <= snap_seq` filter must NOT swallow it.
ui.on_content_token("turn-N+1 tok1 ")
ev = lq.get_nowait()
assert ev["type"] == "content"
assert ev["text"] == "turn-N+1 tok1 "
assert ev["_seq"] > snap_seq, (
f"Token from turn N+1 has _seq={ev['_seq']} which is <= "
f"snap_seq={snap_seq} — the events handler's dedup filter "
f"would silently drop it on a long-lived SSE subscription."
)
def test_snapshot_and_consume_drains_inflight_at_idle() -> None:
"""Regression for the cancel/error path: ``on_turn_committed`` is
NOT called from cancel handlers, but every exit path eventually
fires ``_emit_state("idle")`` (cancel) or ``_emit_state("error")``
(exception). The IDLE/ERROR branches of
``snapshot_and_consume_state_payload`` must drain the inflight
buffers so a refresh post-cancel doesn't double-render the
cancelled fragment against history's marker'd version."""
ui = _make_ui()
ui.on_content_token("partial cancelled text ")
ui.on_reasoning_token("partial reasoning ")
assert ui._ws_inflight_content_size > 0
assert ui._ws_inflight_reasoning_size > 0
ui.snapshot_and_consume_state_payload("idle")
assert ui._ws_inflight_content == []
assert ui._ws_inflight_content_size == 0
assert ui._ws_inflight_reasoning == []
assert ui._ws_inflight_reasoning_size == 0
def test_snapshot_and_consume_drains_inflight_at_error() -> None:
"""Regression for the exception path: ERROR-branch must drain
inflight too (parallel to the IDLE branch)."""
ui = _make_ui()
ui.on_content_token("partial errored text ")
ui.on_reasoning_token("partial errored reasoning ")
ui.snapshot_and_consume_state_payload("error")
assert ui._ws_inflight_content == []
assert ui._ws_inflight_reasoning == []
def test_snapshot_and_consume_does_not_reset_seq_at_idle_or_error() -> None:
"""The IDLE/ERROR drain clears content + reasoning but must NOT
reset the seq counter long-lived subscribers' snap_seq must
stay valid across turn boundaries (see the q-1 invariant test)."""
ui = _make_ui()
ui.on_content_token("a")
ui.on_content_token("b")
assert ui._ws_inflight_seq == 2
ui.snapshot_and_consume_state_payload("idle")
assert ui._ws_inflight_seq == 2
ui.snapshot_and_consume_state_payload("error")
assert ui._ws_inflight_seq == 2
def test_listeners_share_dict_reference_warning() -> None:
"""Pinning the shape that necessitated the events-handler shallow
copy: ``_enqueue`` puts ONE dict reference into every listener
queue. If multiple SSE coroutines mutate (e.g. ``del event[\"_seq\"]``)
without copying first, they corrupt each other's view. The fix
in make_events_handler is ``event = dict(event)`` immediately
after ``client_queue.get`` verify the underlying invariant
here so a future refactor of ``_enqueue`` can't silently break
the assumption the events handler relies on."""
ui = _make_ui()
lq1, _ = ui.register_listener_with_in_progress_snapshot()
lq2, _ = ui.register_listener_with_in_progress_snapshot()
ui.on_content_token("X")
ev1 = lq1.get_nowait()
ev2 = lq2.get_nowait()
# Same reference today — consumers MUST shallow-copy before any
# mutation. If a future _enqueue change makes this no longer
# true, the events handler's defensive copy becomes redundant
# but harmless; if this assertion suddenly fails the underlying
# invariant has shifted and the handler comment should be updated.
assert ev1 is ev2
def test_concurrent_writer_and_register_with_snapshot_no_loss_no_dup() -> None:
"""Stress: many tokens streaming + a register_with_snapshot landing
at a random point. End state: snapshot filtered_live == every
token written, exactly once."""
ui = _make_ui()
n_tokens = 500
snap_box: dict[str, Any] = {}
lq_box: dict[str, queue.Queue[Any]] = {}
def _writer() -> None:
for i in range(n_tokens):
ui.on_content_token(f"{i},")
def _registrar() -> None:
# Tiny sleep so the writer is mid-flight.
threading.Event().wait(0.001)
lq, snap = ui.register_listener_with_in_progress_snapshot()
snap_box["snap"] = snap
lq_box["lq"] = lq
w = threading.Thread(target=_writer)
r = threading.Thread(target=_registrar)
w.start()
r.start()
w.join()
r.join()
snap = snap_box["snap"]
lq = lq_box["lq"]
reconstructed = snap["content"]
while True:
try:
ev = lq.get_nowait()
except queue.Empty:
break
if ev.get("_seq", 0) <= snap["seq"]:
continue
reconstructed += ev["text"]
expected = "".join(f"{i}," for i in range(n_tokens))
assert reconstructed == expected, (
f"reconstruction mismatch: len(rec)={len(reconstructed)}, len(exp)={len(expected)}"
)
@@ -24,6 +24,12 @@ from turnstone.core.storage._registry import get_storage
class NullUI:
"""UI adapter that discards all output."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
+6
View File
@@ -34,6 +34,12 @@ from turnstone.core.storage._sqlite import SQLiteBackend
class NullUI:
"""UI adapter that discards all output."""
def on_turn_start(self):
pass
def on_turn_committed(self):
pass
def on_thinking_start(self):
pass
+222
View File
@@ -0,0 +1,222 @@
"""Tests for the storage layer's cross-process ``notify`` / ``listen`` API.
Covers SQLite (synthetic-sweep + in-process fan-out) and PostgreSQL
(real ``LISTEN``/``NOTIFY``). The PG-only cases are gated on the
``--storage-backend=postgresql`` flag so they no-op on default CI runs.
"""
from __future__ import annotations
import threading
import time
import pytest
def _drain_until(stream, predicate, deadline_sec: float = 5.0):
"""Poll ``stream`` until ``predicate`` matches one of the drained notifies.
Returns the matching notify or raises ``TimeoutError``. Tests use
this so timing flakes against the bounded-blocking ``poll`` shape
don't masquerade as logic bugs.
"""
deadline = time.monotonic() + deadline_sec
while time.monotonic() < deadline:
remaining = max(0.05, deadline - time.monotonic())
for n in stream.poll(min(0.5, remaining)):
if predicate(n):
return n
msg = "no matching notify drained before deadline"
raise TimeoutError(msg)
class TestSqliteNotify:
"""SQLite path: in-process fan-out + synthetic sweep."""
def test_notify_no_listeners_is_noop(self, storage):
# No exception, no side effect — safe to always call from dispatch.
storage.notify("services", '{"op": "INSERT"}')
def test_notify_delivers_to_in_process_listener(self, storage):
with storage.listen(["services"]) as stream:
storage.notify("services", '{"op": "INSERT"}')
got = _drain_until(stream, lambda n: n.payload == '{"op": "INSERT"}')
assert got.channel == "services"
# ``pid`` is 0 on the SQLite synthetic path and the sending
# backend's PID on Postgres — both are valid notify shapes,
# so don't assert on the value here.
def test_notify_filters_by_channel(self, storage):
with storage.listen(["services"]) as stream:
storage.notify("other_channel", "ignored")
storage.notify("services", "wanted")
got = _drain_until(stream, lambda n: True)
assert got.payload == "wanted"
def test_multiple_listeners_each_get_event(self, storage):
# Two streams open on the same channel; each gets its own copy.
with storage.listen(["services"]) as s1, storage.listen(["services"]) as s2:
storage.notify("services", "broadcast")
got1 = _drain_until(s1, lambda n: True)
got2 = _drain_until(s2, lambda n: True)
assert got1.payload == "broadcast"
assert got2.payload == "broadcast"
def test_close_stops_stream(self, storage):
with storage.listen(["services"]) as stream:
pass
# After context exit, the stream is closed; poll returns [] without
# blocking. A second close() is idempotent.
assert stream.poll(0.05) == []
stream.close()
def test_synthetic_sweep_emits_after_interval(self, storage, _is_sqlite):
# Synthetic sweep is fundamentally SQLite-specific — the PG path
# uses real ``LISTEN``/``NOTIFY`` and has no sweep tick. Gate
# so the test doesn't false-fail by waiting for a "sweep" notify
# that the PG stream will never produce.
with storage.listen(["services"], sweep_interval=0.1) as stream:
# First poll: not yet at the interval, so likely empty.
stream.poll(0.05)
# Wait past the interval, then poll again — should emit a
# synthetic-sweep notify per declared channel.
time.sleep(0.15)
got = _drain_until(stream, lambda n: n.payload == "sweep")
assert got.channel == "services"
assert got.payload == "sweep"
def test_empty_channel_list_yields_empty_stream(self, storage):
with storage.listen([]) as stream:
# No channels — poll returns [] regardless of how long we wait.
assert stream.poll(0.05) == []
# ---------------------------------------------------------------------------
# PostgreSQL path — gated on --storage-backend=postgresql.
# ---------------------------------------------------------------------------
@pytest.fixture
def _is_postgres(storage):
"""Skip the wrapped test when the active backend isn't Postgres."""
if storage.__class__.__name__ != "PostgreSQLBackend":
pytest.skip("PostgreSQL-specific test")
return True
@pytest.fixture
def _is_sqlite(storage):
"""Skip the wrapped test when the active backend isn't SQLite."""
if storage.__class__.__name__ != "SQLiteBackend":
pytest.skip("SQLite-specific test")
return True
class TestPostgresNotify:
def test_round_trip(self, storage, _is_postgres):
# Open a listener, fire a notify on a regular pooled connection,
# drain the listener within a reasonable bound (PG NOTIFY is
# typically sub-100ms on a local socket).
with storage.listen(["pytest_round_trip"]) as stream:
# Tiny sleep so the LISTEN settles before the NOTIFY fires —
# otherwise the notify can arrive on the connection before
# the LISTEN is registered (race only visible in tests).
time.sleep(0.05)
storage.notify("pytest_round_trip", '{"hello": "world"}')
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
assert got.channel == "pytest_round_trip"
assert got.payload == '{"hello": "world"}'
assert got.pid > 0
def test_concurrent_notifies_all_arrive(self, storage, _is_postgres):
with storage.listen(["pytest_concurrent"]) as stream:
time.sleep(0.05)
for i in range(5):
storage.notify("pytest_concurrent", str(i))
seen: set[str] = set()
deadline = time.monotonic() + 3.0
while len(seen) < 5 and time.monotonic() < deadline:
for n in stream.poll(0.2):
seen.add(n.payload)
assert seen == {"0", "1", "2", "3", "4"}
def test_close_aborts_blocked_poll(self, storage, _is_postgres):
# poll() should return promptly once close() runs on another thread.
with storage.listen(["pytest_close"]) as stream:
done = threading.Event()
result: list[list] = []
def _poll_long():
result.append(stream.poll(5.0))
done.set()
t = threading.Thread(target=_poll_long, daemon=True)
t.start()
time.sleep(0.1)
stream.close()
assert done.wait(2.0), "close() did not unblock poll()"
# No notify arrived, so the polled batch is empty — but the
# poll loop must have exited well under the 5 s timeout.
assert result == [[]]
class TestServicesTriggerFilter:
"""Migration 053's trigger: fires on real changes, quiet on heartbeats.
PG-only the SQLite path has no trigger and is covered by
:class:`TestSqliteNotify`. Verifies the in-trigger ``IS NOT DISTINCT
FROM`` filter a heartbeat-only UPDATE (same url + same metadata,
only ``last_heartbeat`` changed) must NOT emit a NOTIFY, since
``register_service`` runs the same UPSERT on every 30 s tick × N
nodes and the channel would otherwise flood.
"""
def test_insert_fires_notify(self, storage, _is_postgres):
with storage.listen(["services"]) as stream:
time.sleep(0.05)
storage.register_service("server", "pytest-trigger-node", "http://127.0.0.1:1")
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
assert got.channel == "services"
assert '"op": "INSERT"' in got.payload or "INSERT" in got.payload
# Cleanup so concurrent suites don't pick up the row.
storage.deregister_service("server", "pytest-trigger-node")
def test_delete_fires_notify(self, storage, _is_postgres):
storage.register_service("server", "pytest-trigger-node-del", "http://127.0.0.1:2")
with storage.listen(["services"]) as stream:
time.sleep(0.05)
storage.deregister_service("server", "pytest-trigger-node-del")
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
assert "DELETE" in got.payload
def test_url_change_update_fires_notify(self, storage, _is_postgres):
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:3")
with storage.listen(["services"]) as stream:
time.sleep(0.05)
# UPSERT with different url — UPDATE path with url diff,
# trigger must fire.
storage.register_service("server", "pytest-trigger-node-url", "http://127.0.0.1:9")
got = _drain_until(stream, lambda n: True, deadline_sec=3.0)
assert "UPDATE" in got.payload
storage.deregister_service("server", "pytest-trigger-node-url")
def test_heartbeat_only_update_is_quiet(self, storage, _is_postgres):
# Open the LISTEN session FIRST so PG delivers the INSERT NOTIFY
# to this connection — pg_notify routes only to sessions that
# have LISTENed at COMMIT time, so an INSERT committed before the
# listen opens would be lost and the drain would time out instead
# of exercising the heartbeat-quiet check below.
with storage.listen(["services"]) as stream:
time.sleep(0.05)
storage.register_service("server", "pytest-trigger-node-hb", "http://127.0.0.1:4")
# Drain the INSERT notify so subsequent polls see only what
# heartbeats emit (if anything).
_drain_until(stream, lambda n: True, deadline_sec=2.0)
# Now fire a heartbeat tick — same url + same metadata,
# only last_heartbeat updates. Trigger must NOT emit.
storage.heartbeat_service("server", "pytest-trigger-node-hb")
# Poll long enough that any spurious notify would have
# arrived; the channel must stay silent.
spurious = stream.poll(0.5)
assert spurious == [], f"heartbeat-only update emitted unexpected notify: {spurious}"
storage.deregister_service("server", "pytest-trigger-node-hb")
+32 -39
View File
@@ -232,59 +232,52 @@ class TestSoftCap:
# ---------------------------------------------------------------------------
# valid_until predicate
# Predicate independence
# ---------------------------------------------------------------------------
class TestValidUntil:
"""The ``valid_until`` predicate captured at dispatch time re-checks
the watch's ``active`` flag at drain time, so a cancelled watch's
last splat doesn't ride out a future wake.
class TestPredicateIndependence:
"""The watch closure does NOT wire a ``valid_until`` predicate.
Earlier the closure wired ``_still_active`` (re-reading
``is_watch_active`` at drain time). That predicate raced
``WatchRunner._poll_watch``'s commit of ``active=False`` and silently
dropped every terminal fire. The closure now enqueues without a
predicate; entries survive drain regardless of the row's ``active``
column state.
"""
def test_valid_until_drops_when_watch_inactive(self, tmp_db, monkeypatch):
def test_drain_delivers_even_when_storage_reports_inactive(self, tmp_db, monkeypatch):
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
# Storage stub returns False at drain time.
is_active_calls = patch_session_storage(monkeypatch, active=False)
dispatch(_reminder("body"), "watch-1")
# Drain fires the predicate; entry should NOT be delivered.
out = session._nudge_queue.drain({"any"})
assert out == []
# Predicate ran once with the dispatched watch_id.
assert is_active_calls == ["watch-1"]
def test_valid_until_drops_when_storage_raises(self, tmp_db, monkeypatch):
"""The closure's broad-except in the predicate translates a
storage-layer exception to ``False`` so the drain doesn't
propagate; the predicate captured ``watch_id`` correctly
(otherwise storage wouldn't even be touched).
"""
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
patch_session_storage(monkeypatch, raise_on_is_active=True)
dispatch(_reminder("body"), "watch-bound-id")
out = session._nudge_queue.drain({"any"})
assert out == []
def test_valid_until_delivers_when_watch_active(self, tmp_db, monkeypatch):
"""Happy-path counter-test for the predicate above: the entry
DOES drain when the watch is still active.
"""
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
patch_session_storage(monkeypatch, active=True)
# Even if storage reports active=False, the entry should still
# drain — no predicate to drop it.
patch_session_storage(monkeypatch, active=False)
dispatch(_reminder("body"), "watch-1")
out = session._nudge_queue.drain({"any"})
assert len(out) == 1
assert out[0][0] == "watch_triggered"
def test_dispatch_never_calls_is_watch_active(self, tmp_db, monkeypatch):
"""Pin the invariant directly: the closure must NOT consult
``storage.is_watch_active`` anywhere along the enqueue + drain
path. Without this assertion, a future change that re-wires
an ``is_watch_active`` predicate would silently bring back the
bug that motivates this whole module.
"""
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
is_active_calls = patch_session_storage(monkeypatch, active=True)
dispatch(_reminder("body"), "watch-bound-id")
session._nudge_queue.drain({"any"})
assert is_active_calls == [], (
f"watch closure must not call is_watch_active; got {is_active_calls!r}"
)
# ---------------------------------------------------------------------------
# Concurrency
+284
View File
@@ -24,11 +24,15 @@ the structural integration gate for the watch switchover.
from __future__ import annotations
import contextlib
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests._helpers import patch_session_storage
from turnstone.core.session import ChatSession
from turnstone.core.storage import get_storage
from turnstone.core.watch import WatchRunner
@@ -272,3 +276,283 @@ def test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session(tmp_db, m
# Original session's queue stays empty — the dispatch did NOT
# accidentally route back to it.
assert len(original._nudge_queue) == 0
@pytest.mark.parametrize(
("stop_on", "max_polls", "label"),
[
('"HIT" in output', 100, "stop_on_fired"),
(None, 1, "max_polls_reached"),
],
)
def test_poll_watch_terminal_fire_survives_drain(
tmp_db: str,
monkeypatch: pytest.MonkeyPatch,
stop_on: str | None,
max_polls: int,
label: str,
) -> None:
"""Regression for the dispatch-ordering bug.
With the broken ordering (``update_watch(active=False)`` before
``_dispatch_result``) plus the ``_still_active`` ``valid_until``
predicate that re-reads ``is_watch_active`` at drain time, every
terminal watch fire was silently dropped the closure enqueued
the entry but the predicate immediately invalidated it because
the row's ``active`` flag had already been flipped to ``0`` in
the same poll. The model never saw the fire.
This test drives a REAL ``WatchRunner._poll_watch`` against a real
``tmp_db`` watch row (no ``patch_session_storage(active=True)``
stub that stub is exactly what masked the bug in earlier tests).
Covers both terminal paths: ``stop_on`` condition matched and
``poll_count >= max_polls`` reached.
"""
session = _make_session()
storage = get_storage()
runner = WatchRunner(storage=storage, node_id="test-node")
session.set_watch_runner(runner)
storage.create_watch(
watch_id=f"w-regression-{label}",
ws_id=session._ws_id,
node_id="test-node",
name=f"regression-{label}",
command="echo HIT",
interval_secs=10.0,
stop_on=stop_on,
max_polls=max_polls,
created_by="model",
next_poll="1970-01-01T00:00:00",
)
# Spy ``enqueue`` so the assertion can distinguish "dispatch never
# called" (a different bug class) from "dispatch enqueued but the
# predicate dropped it at drain" (this bug).
enqueue_calls: list[tuple[str, str, str]] = []
real_enqueue = session._nudge_queue.enqueue
def _spy_enqueue(*args: Any, **kwargs: Any) -> None:
enqueue_calls.append((args[0], args[1][:40], args[2]))
return real_enqueue(*args, **kwargs)
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
# For the max_polls=1 case the first poll has prev_output=None and
# would not normally fire on output change; the max_polls branch
# at watch.py:412-414 still marks is_final=True so dispatch runs.
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == f"w-regression-{label}"]
assert len(matching) == 1, f"watch row not picked up by list_due_watches: {due!r}"
runner._poll_watch(matching[0])
assert len(enqueue_calls) == 1, (
f"_poll_watch did not enqueue exactly one fire (got {enqueue_calls!r}); "
"this is a different bug from the predicate-drop regression"
)
assert enqueue_calls[0][0] == "watch_triggered"
assert storage.is_watch_active(f"w-regression-{label}") is False, (
"terminal fire should have committed active=False on the row"
)
# The key assertion: drain delivers the entry. Pre-fix this
# returned ``[]`` because the ``_still_active`` predicate re-read
# ``active=0``. Post-fix the watch closure no longer wires a
# predicate and the entry survives.
out = session._nudge_queue.drain({"any"})
assert len(out) == 1, (
"Watch fire was enqueued but never reached drain — dispatch-ordering "
"regression. Check that WatchRunner._poll_watch dispatches BEFORE "
"committing active=False, and that the watch closure in "
"ChatSession.set_watch_runner does not wire an is_watch_active "
"predicate."
)
nt, text, _meta = out[0]
assert nt == "watch_triggered"
assert "HIT" in text
def test_cancel_reports_already_completed_for_auto_cancelled_watch(tmp_db: str) -> None:
"""After a watch fires and auto-cancels, the cancel-by-name path
should report 'already completed' rather than 'not found'.
Pre-fix, ``_exec_watch`` cancel looked the watch up via
``list_watches_for_ws`` which filters ``active==1``, so a recently-
auto-cancelled row was invisible and the model got the same
'not found' message it would for a typo'd name. Post-fix the
cancel path uses ``find_watch_by_name`` (no active filter) and
branches on ``row["active"]``.
"""
session = _make_session()
storage = get_storage()
storage.create_watch(
watch_id="w-completed-1",
ws_id=session._ws_id,
node_id="test-node",
name="completed-watch",
command="echo x",
interval_secs=10.0,
stop_on=None,
max_polls=100,
created_by="model",
next_poll="",
)
# Simulate the post-fire state.
storage.update_watch("w-completed-1", active=False, next_poll="")
_call_id, msg = session._exec_watch(
{"call_id": "c1", "action": "cancel", "watch_name": "completed-watch"}
)
assert "not found" not in msg.lower()
assert "completed" in msg.lower()
def test_cancel_reports_not_found_for_unknown_watch(tmp_db: str) -> None:
"""The 'not found' message still applies when the watch genuinely
does not exist make sure the new ``find_watch_by_name`` path
didn't accidentally turn every cancel into 'already completed'.
"""
session = _make_session()
_call_id, msg = session._exec_watch(
{"call_id": "c1", "action": "cancel", "watch_name": "ghost-watch"}
)
assert "not found" in msg.lower()
def test_poll_watch_retry_deactivate_after_update_watch_failure(
tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``_terminal_dispatched`` lifecycle: if ``update_watch`` raises
AFTER ``_dispatch_result`` shipped the reminder for a terminal
fire, the next ``_poll_watch`` tick MUST retry the row write
(so the row stops appearing in ``list_due_watches``) and MUST NOT
re-dispatch the reminder the model already saw.
This is the keystone path that prevents duplicate-fire under
transient storage failure. Pre-this-test, the entire branch was
unexercised.
"""
session = _make_session()
storage = get_storage()
runner = WatchRunner(storage=storage, node_id="test-node")
session.set_watch_runner(runner)
watch_id = "w-retry-1"
storage.create_watch(
watch_id=watch_id,
ws_id=session._ws_id,
node_id="test-node",
name="retry-watch",
command="echo HIT",
interval_secs=10.0,
stop_on='"HIT" in output',
max_polls=100,
created_by="model",
next_poll="1970-01-01T00:00:00",
)
enqueue_calls: list[tuple[str, str]] = []
real_enqueue = session._nudge_queue.enqueue
def _spy_enqueue(*args: Any, **kwargs: Any) -> None:
enqueue_calls.append((args[0], args[1][:32]))
return real_enqueue(*args, **kwargs)
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
# Stage 1 — first poll. ``update_watch`` raises AFTER dispatch.
real_update = storage.update_watch
update_raise = {"armed": True}
def _failing_update(wid: str, **fields: Any) -> bool:
if update_raise["armed"]:
raise RuntimeError("simulated transient storage failure")
return real_update(wid, **fields)
monkeypatch.setattr(storage, "update_watch", _failing_update)
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == watch_id]
assert len(matching) == 1
# ``_poll_watch`` doesn't catch the storage error; the outer
# ``_tick`` would log it. Suppress here so the test owns the
# boundary and continues to its assertions.
with contextlib.suppress(RuntimeError):
runner._poll_watch(matching[0])
# Dispatch ran exactly once and the watch_id sits in the
# terminal-dispatched set awaiting retry.
assert len(enqueue_calls) == 1
assert enqueue_calls[0][0] == "watch_triggered"
assert watch_id in runner._terminal_dispatched
# The row is still active=1 because update_watch raised. It
# would re-appear in list_due_watches on the next tick.
assert storage.is_watch_active(watch_id) is True
# Stage 2 — second poll. Storage now succeeds; retry-deactivate
# branch must commit active=False WITHOUT re-dispatching.
update_raise["armed"] = False
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == watch_id]
assert len(matching) == 1
runner._poll_watch(matching[0])
# Exactly one dispatch in total — the retry path took the
# short-circuit return at the top of _poll_watch.
assert len(enqueue_calls) == 1, f"retry-deactivate must not re-dispatch; got {enqueue_calls!r}"
# Row is now inactive (the retry path's update_watch landed).
assert storage.is_watch_active(watch_id) is False
# Set is cleared so future watches with the same id (unlikely) /
# process memory doesn't accumulate.
assert watch_id not in runner._terminal_dispatched
def test_cancel_clears_pending_terminal_dispatched_entry(
tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
"""If ``update_watch`` raised after dispatch, leaving a pending
entry in ``_terminal_dispatched``, and the user then cancels the
watch out-of-band, the retry-deactivate branch never gets to run
(the cancel sets ``next_poll=""`` which removes the row from
``list_due_watches``). The cancel path itself must discard the
pending entry; otherwise the runner leaks ``watch_id``s for the
process lifetime.
"""
session = _make_session()
storage = get_storage()
runner = WatchRunner(storage=storage, node_id="test-node")
session.set_watch_runner(runner)
watch_id = "w-leak-1"
storage.create_watch(
watch_id=watch_id,
ws_id=session._ws_id,
node_id="test-node",
name="leak-watch",
command="echo x",
interval_secs=10.0,
stop_on=None,
max_polls=100,
created_by="model",
next_poll="",
)
# Simulate: dispatch shipped, update_watch raised, watch_id sits
# in the runner's pending set.
with runner._terminal_dispatched_lock:
runner._terminal_dispatched.add(watch_id)
# User cancels. Because the cancel writes active=False, next_poll="",
# the row leaves list_due_watches and the runner's retry-deactivate
# branch never executes for it. The cancel must discard the entry.
storage.update_watch(watch_id, active=False, next_poll="")
session._exec_watch({"call_id": "c1", "action": "cancel", "watch_name": "leak-watch"})
assert watch_id not in runner._terminal_dispatched
+89
View File
@@ -2,6 +2,10 @@
from __future__ import annotations
import sqlalchemy as sa
from turnstone.core.storage._schema import watches as watches_table
def _make_watch_kwargs(**overrides):
"""Build default kwargs for create_watch."""
@@ -101,6 +105,91 @@ class TestWatchListQueries:
db.update_watch("w1", active=False)
assert db.list_watches_for_ws("ws-1") == []
def test_find_by_name_returns_inactive(self, db):
"""``find_watch_by_name`` ignores the active filter — that is
what lets the cancel-by-name UX distinguish 'already completed'
from 'no such watch.'
"""
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="completed"))
db.update_watch("w1", active=False)
row = db.find_watch_by_name("ws-1", "completed")
assert row is not None
assert row["watch_id"] == "w1"
assert not row["active"]
def test_find_by_name_matches_watch_id_prefix(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="abcdef123", ws_id="ws-1", name="x"))
row = db.find_watch_by_name("ws-1", "abc")
assert row is not None
assert row["watch_id"] == "abcdef123"
def test_find_by_name_scoped_to_ws(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="shared"))
db.create_watch(**_make_watch_kwargs(watch_id="w2", ws_id="ws-2", name="shared"))
row = db.find_watch_by_name("ws-1", "shared")
assert row is not None
assert row["watch_id"] == "w1"
def test_find_by_name_returns_none_when_missing(self, db):
assert db.find_watch_by_name("ws-1", "ghost") is None
def test_find_by_name_empty_input_returns_none(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="x"))
assert db.find_watch_by_name("ws-1", "") is None
def test_find_by_name_treats_percent_as_literal(self, db):
"""A model-supplied '%' must NOT match arbitrary watch_ids.
Pre-escape, ``watch_id.like(f"{name_or_prefix}%")`` would
interpret '%' as 'match anything' and pick up the first row in
the workstream regardless of name.
"""
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="real-watch"))
assert db.find_watch_by_name("ws-1", "%") is None
def test_find_by_name_treats_underscore_as_literal(self, db):
"""Same as the '%' case for the single-char LIKE wildcard."""
db.create_watch(**_make_watch_kwargs(watch_id="abcd", ws_id="ws-1", name="real-watch"))
# '_' would otherwise match any single char, picking up
# watch_ids beginning with 'a', 'b', etc.
assert db.find_watch_by_name("ws-1", "_") is None
def test_find_by_name_prefers_active_over_newer_inactive(self, db):
"""If a same-name pair exists where the inactive row is NEWER
than the active row, find_watch_by_name must still return the
active row. Pre-fix the query was ``ORDER BY created DESC
LIMIT 1`` which would return the newer inactive row and
cause the cancel UX to report 'already completed' for a name
whose live row is still polling.
Reachable in practice because storage allows out-of-band
writes (e.g. ``delete_watches_for_ws`` cleanup followed by
re-create, an admin manually flipping ``active``, or test
scaffolding) that bypass the create-time duplicate-name
guard.
"""
# Older active watch.
db.create_watch(**_make_watch_kwargs(watch_id="w-active", ws_id="ws-1", name="recurring"))
# Newer inactive watch with the same name. ``create_watch``
# stamps ``created`` to ``now`` at second resolution, so we
# bypass the API to give the inactive row a deterministically
# later timestamp.
db.create_watch(**_make_watch_kwargs(watch_id="w-inactive", ws_id="ws-1", name="recurring"))
with db._conn() as conn:
conn.execute(
sa.update(watches_table)
.where(watches_table.c.watch_id == "w-inactive")
.values(active=0, next_poll="", created="2099-01-01T00:00:00")
)
conn.commit()
row = db.find_watch_by_name("ws-1", "recurring")
assert row is not None
assert row["watch_id"] == "w-active"
assert row["active"]
def test_list_for_node(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="w1", node_id="n1"))
db.create_watch(**_make_watch_kwargs(watch_id="w2", node_id="n1"))
+5 -5
View File
@@ -29,7 +29,7 @@ class TestVersionHtml:
def test_vendored_katex_skipped(self):
from turnstone.core.web_helpers import version_html
html = '<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">'
html = '<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">'
result = version_html(html)
assert result == html # unchanged
@@ -43,14 +43,14 @@ class TestVersionHtml:
def test_vendored_mermaid_skipped(self):
from turnstone.core.web_helpers import version_html
html = '<script src="/shared/mermaid-11.14.0/mermaid.min.js"></script>'
html = '<script src="/shared/mermaid-11.15.0/mermaid.min.js"></script>'
result = version_html(html)
assert result == html # unchanged
def test_vendored_hls_skipped(self):
from turnstone.core.web_helpers import version_html
html = '<script src="/shared/hls-1.6.15/hls.min.js"></script>'
html = '<script src="/shared/hls-1.6.16/hls.min.js"></script>'
result = version_html(html)
assert result == html # unchanged
@@ -76,7 +76,7 @@ class TestVersionHtml:
html = (
'<link rel="stylesheet" href="/shared/base.css">\n'
'<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">\n'
'<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">\n'
'<link rel="stylesheet" href="/static/style.css">\n'
'<script src="/shared/utils.js"></script>\n'
'<script src="/shared/hljs-11.11.1/highlight.min.js"></script>\n'
@@ -88,7 +88,7 @@ class TestVersionHtml:
assert f'/shared/utils.js?v={__version__}"' in result
assert f'/static/app.js?v={__version__}"' in result
# Vendored libs unchanged
assert '/shared/katex-0.16.44/katex.min.css"' in result
assert '/shared/katex-0.16.47/katex.min.css"' in result
assert '/shared/hljs-11.11.1/highlight.min.js"' in result
def test_version_matches_package(self):
+154
View File
@@ -2,7 +2,9 @@
from __future__ import annotations
import json
import queue
from types import SimpleNamespace
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
@@ -1762,3 +1764,155 @@ class TestTenantCheckOnReadEndpoints:
assert cold_check in offloaded, (
f"tenant_check must be invoked through asyncio.to_thread; got {offloaded}"
)
class TestHistoryReasoningRehydration:
"""The lifted ``GET /v1/api/workstreams/{ws_id}/history`` surfaces
stored Anthropic thinking blocks on assistant messages so a page
refresh re-renders the reasoning bubble. Drives through the real
``AnthropicProvider.extract_reasoning_text`` and the storage
``reconstruct_messages`` boundary that JSON-decodes
``provider_data`` into ``_provider_content``.
"""
def test_history_handler_surfaces_reasoning_for_anthropic_thinking(self, _inject_storage):
ws_id = "ws-reason-1"
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
provider_data = json.dumps(
[
{"type": "thinking", "thinking": "let me reason", "signature": "s"},
{"type": "text", "text": "Final answer."},
]
)
_inject_storage.save_message(
ws_id, "assistant", "Final answer.", provider_data=provider_data
)
# No live session — exercises the storage-only path which
# falls back to default surface_persisted_reasoning=True.
mock_mgr = MagicMock()
mock_mgr.get.return_value = None
client = _build_history_app(mock_mgr, _inject_storage)
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
assert r.status_code == 200
msgs = r.json()["messages"]
assistant = next(m for m in msgs if m.get("role") == "assistant")
assert assistant["reasoning"] == "let me reason"
def test_history_handler_strips_provider_content(self, _inject_storage):
ws_id = "ws-reason-2"
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
provider_data = json.dumps([{"type": "thinking", "thinking": "x", "signature": "s"}])
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
mock_mgr = MagicMock()
mock_mgr.get.return_value = None
client = _build_history_app(mock_mgr, _inject_storage)
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
assert r.status_code == 200
for m in r.json()["messages"]:
assert "_provider_content" not in m
def test_history_handler_with_persist_flag_false_via_live_session(self, _inject_storage):
"""Operator-flipped ``surface_persisted_reasoning=False`` on the active
model suppresses the reasoning field even when the data is
stored. ``_provider_content`` is still stripped from the wire.
"""
ws_id = "ws-reason-3"
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
provider_data = json.dumps([{"type": "thinking", "thinking": "hidden", "signature": "s"}])
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
live_session = SimpleNamespace(
id=ws_id,
_registry=SimpleNamespace(
get_config=lambda alias: SimpleNamespace(surface_persisted_reasoning=False)
),
_model_alias="claude-opus-4-7",
)
mock_mgr = MagicMock()
mock_mgr.get.return_value = live_session
client = _build_history_app(mock_mgr, _inject_storage)
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
assert r.status_code == 200
for m in r.json()["messages"]:
if m.get("role") == "assistant":
assert "reasoning" not in m
assert "_provider_content" not in m
def test_history_handler_cold_workstream_resolves_via_workstream_config(self, _inject_storage):
"""Cold workstream (no live session) — the handler walks
``workstream_config.model_alias`` (persisted at first send by
the SessionManager rehydrate path) and looks up the active
model's ``surface_persisted_reasoning`` flag through the global registry
on ``app.state``. Operator flag-flip is honored uniformly
across live and cold workstreams.
"""
ws_id = "ws-reason-cold"
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
# Simulate the model alias persisted by the rehydrate path
# (session_manager.py:628-629 reads it back via the same key).
_inject_storage.save_workstream_config(ws_id, {"model_alias": "claude-opus-4-7"})
provider_data = json.dumps(
[{"type": "thinking", "thinking": "should not surface", "signature": "s"}]
)
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
# No live session — handler falls back to workstream_config + registry.
mock_mgr = MagicMock()
mock_mgr.get.return_value = None
# Build the app with a global registry that reports persist=False
# for the saved alias.
cfg = _interactive_endpoint_cfg(mock_mgr)
handler = make_history_handler(cfg)
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route(
"/api/workstreams/{ws_id}/history",
handler,
methods=["GET"],
),
],
),
],
middleware=[Middleware(_InjectAuthMiddleware)],
)
app.state.workstreams = mock_mgr
app.state.auth_storage = _inject_storage
app.state.registry = SimpleNamespace(
get_config=lambda alias: SimpleNamespace(
surface_persisted_reasoning=(alias != "claude-opus-4-7"),
)
)
client = TestClient(app)
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
assert r.status_code == 200
# Flag-flip on the saved alias is honored: reasoning suppressed.
for m in r.json()["messages"]:
if m.get("role") == "assistant":
assert "reasoning" not in m
assert "_provider_content" not in m
def test_history_handler_cold_workstream_no_alias_defaults_true(self, _inject_storage):
"""A workstream that pre-dates the rehydrate-time alias persist
(or one that simply has no workstream_config row) falls through
to the conservative default ``True``. Reasoning surfaces.
"""
ws_id = "ws-reason-cold-no-alias"
_inject_storage.register_workstream(ws_id, kind="interactive", user_id="test-user")
provider_data = json.dumps(
[{"type": "thinking", "thinking": "default-true wins", "signature": "s"}]
)
_inject_storage.save_message(ws_id, "assistant", "Answer.", provider_data=provider_data)
mock_mgr = MagicMock()
mock_mgr.get.return_value = None
client = _build_history_app(mock_mgr, _inject_storage)
r = client.get(f"/v1/api/workstreams/{ws_id}/history")
assert r.status_code == 200
assistant = next(m for m in r.json()["messages"] if m.get("role") == "assistant")
assert assistant["reasoning"] == "default-true wins"
+7
View File
@@ -62,6 +62,13 @@
[database]
# url = "" # postgres://user:pass@host/db or /path/to.db
# env: TURNSTONE_DB_URL
# listen_url = "" # direct-to-postgres URL for the console's
# dedicated LISTEN connection. Set this when
# `url` points at pgbouncer in transaction
# pooling mode (LISTEN holds session state and
# is incompatible with transaction pooling —
# see docs/pgbouncer.md). Defaults to `url`
# when unset. env: TURNSTONE_DB_LISTEN_URL
# SSL params (passed through to SQLAlchemy connection):
# sslmode = "prefer" # disable, allow, prefer, require, verify-ca, verify-full
# sslrootcert = "" # path to CA cert for verify-ca/verify-full
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "1.5.10"
__version__ = "1.5.18"
+40 -14
View File
@@ -12,14 +12,36 @@ import uuid
from typing import Any
def _get_storage() -> Any:
"""Initialize and return the storage backend."""
def _get_storage(args: argparse.Namespace) -> Any:
"""Initialize and return the storage backend.
Precedence (matches turnstone-server): CLI / config.toml ``[database]``
> ``TURNSTONE_DB_*`` env vars > hardcoded defaults.
"""
from turnstone.core.storage import init_storage
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
db_url = os.environ.get("TURNSTONE_DB_URL", "")
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
return init_storage(db_backend, path=db_path, url=db_url)
def _pick(arg_name: str, env_name: str, default: str = "") -> Any:
# `is not None` (not truthy) so a legitimate falsy TOML value
# like `pool_size = 0` or `url = ""` still beats the env fallback.
val = getattr(args, arg_name, None)
if val is not None:
return val
return os.environ.get(env_name, default)
db_backend = str(_pick("db_backend", "TURNSTONE_DB_BACKEND", "sqlite"))
db_url = str(_pick("db_url", "TURNSTONE_DB_URL"))
db_path = str(_pick("db_path", "TURNSTONE_DB_PATH"))
db_pool_size = int(_pick("db_pool_size", "TURNSTONE_DB_POOL_SIZE", "2"))
return init_storage(
db_backend,
path=db_path,
url=db_url,
pool_size=db_pool_size,
sslmode=str(_pick("db_sslmode", "TURNSTONE_DB_SSLMODE")),
sslrootcert=str(_pick("db_sslrootcert", "TURNSTONE_DB_SSLROOTCERT")),
sslcert=str(_pick("db_sslcert", "TURNSTONE_DB_SSLCERT")),
sslkey=str(_pick("db_sslkey", "TURNSTONE_DB_SSLKEY")),
)
def _cmd_create_user(args: argparse.Namespace) -> None:
@@ -37,7 +59,7 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
print("Error: invalid username (1-64 chars: letters, digits, . _ -)", file=sys.stderr)
sys.exit(1)
storage = _get_storage()
storage = _get_storage(args)
user_id = uuid.uuid4().hex
# Prompt for password
@@ -76,7 +98,7 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
def _cmd_create_token(args: argparse.Namespace) -> None:
from turnstone.core.auth import generate_token, hash_token, token_prefix
storage = _get_storage()
storage = _get_storage(args)
if storage.get_user(args.user) is None:
print(f"Error: user {args.user} not found", file=sys.stderr)
@@ -110,7 +132,7 @@ def _cmd_create_token(args: argparse.Namespace) -> None:
def _cmd_list_users(args: argparse.Namespace) -> None:
storage = _get_storage()
storage = _get_storage(args)
users = storage.list_users()
if not users:
print("No users found.")
@@ -120,7 +142,7 @@ def _cmd_list_users(args: argparse.Namespace) -> None:
def _cmd_list_tokens(args: argparse.Namespace) -> None:
storage = _get_storage()
storage = _get_storage(args)
tokens = storage.list_api_tokens(args.user)
if not tokens:
print(f"No tokens found for user {args.user}.")
@@ -134,7 +156,7 @@ def _cmd_list_tokens(args: argparse.Namespace) -> None:
def _cmd_revoke_token(args: argparse.Namespace) -> None:
storage = _get_storage()
storage = _get_storage(args)
if storage.delete_api_token(args.token_id):
print(f"Revoked token {args.token_id}")
else:
@@ -297,7 +319,7 @@ def _cmd_list_node_metadata(args: argparse.Namespace) -> None:
"""List metadata for a node."""
import json
storage = _get_storage()
storage = _get_storage(args)
rows = storage.get_node_metadata(args.node_id)
if not rows:
print(f"No metadata for node: {args.node_id}")
@@ -324,7 +346,7 @@ def _cmd_set_node_metadata(args: argparse.Namespace) -> None:
"""Set a metadata key on a node."""
import json
storage = _get_storage()
storage = _get_storage(args)
# Check for auto-source conflict
existing = storage.get_node_metadata(args.node_id)
@@ -345,7 +367,7 @@ def _cmd_set_node_metadata(args: argparse.Namespace) -> None:
def _cmd_delete_node_metadata(args: argparse.Namespace) -> None:
"""Delete a metadata key from a node."""
storage = _get_storage()
storage = _get_storage(args)
existing = storage.get_node_metadata(args.node_id)
for r in existing:
@@ -395,6 +417,10 @@ def main() -> None:
prog="turnstone-admin",
description="Turnstone user and token administration",
)
from turnstone.core.config import add_config_arg, apply_config
add_config_arg(parser)
apply_config(parser, ["database"])
sub = parser.add_subparsers(dest="command")
p_cu = sub.add_parser("create-user", help="Create a new user")
+8
View File
@@ -908,6 +908,8 @@ class ModelDefinitionInfo(BaseModel):
temperature: float | None = None
max_tokens: int | None = None
reasoning_effort: str | None = None
surface_persisted_reasoning: bool = True
replay_reasoning_to_model: bool = False
source: str = ""
created_by: str = ""
created: str = ""
@@ -926,6 +928,8 @@ class CreateModelDefinitionRequest(BaseModel):
temperature: float | None = None
max_tokens: int | None = None
reasoning_effort: str | None = None
surface_persisted_reasoning: bool = True
replay_reasoning_to_model: bool = False
class UpdateModelDefinitionRequest(BaseModel):
@@ -940,6 +944,8 @@ class UpdateModelDefinitionRequest(BaseModel):
temperature: float | None = None
max_tokens: int | None = None
reasoning_effort: str | None = None
surface_persisted_reasoning: bool | None = None
replay_reasoning_to_model: bool | None = None
class ListModelDefinitionsResponse(BaseModel):
@@ -990,6 +996,8 @@ class ListAvailableModelsResponse(BaseModel):
models: list[AvailableModelInfo] = Field(default_factory=list)
default_alias: str = ""
channel_default_alias: str = ""
coordinator_default_alias: str = ""
judge_default_alias: str = ""
# ---------------------------------------------------------------------------
+8
View File
@@ -110,6 +110,14 @@ class TerminalUI(SessionUI):
self.auto_approve = False
self.auto_approve_tools: set[str] = set()
def on_turn_start(self) -> None:
# Terminal UI has no inflight buffer to reset.
pass
def on_turn_committed(self) -> None:
# Terminal UI has no inflight buffer to reset.
pass
def on_thinking_start(self) -> None:
self.spinner = Spinner("Thinking")
self.spinner.start()
+40
View File
@@ -25,9 +25,13 @@ import httpx_sse
from turnstone.core.workstream import WorkstreamKind
if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.console.metrics import ConsoleMetrics
from turnstone.console.notify_dispatcher import NotifyDispatcher
from turnstone.console.router import ConsoleRouter
from turnstone.core.auth import ServiceTokenManager
from turnstone.core.storage._notify import Notify
from turnstone.core.storage._protocol import StorageBackend
log = logging.getLogger("turnstone.console.collector")
@@ -73,6 +77,7 @@ class ClusterCollector:
tls_cert: tuple[str, str] | None = None,
router: ConsoleRouter | None = None,
console_metrics: ConsoleMetrics | None = None,
notify_dispatcher: NotifyDispatcher | None = None,
):
self._storage = storage
self._discovery_interval = discovery_interval
@@ -82,6 +87,8 @@ class ClusterCollector:
self._console_metrics = console_metrics
self._tls_verify = tls_verify
self._tls_cert = tls_cert
self._notify_dispatcher = notify_dispatcher
self._notify_unsubscribe: Callable[[], None] | None = None
self._lock = threading.Lock()
self._nodes: dict[str, NodeSnapshot] = {}
@@ -128,6 +135,15 @@ class ClusterCollector:
def start(self) -> None:
"""Start background threads."""
self._running = True
# Subscribe to the ``services`` channel for reactive node discovery.
# NOTIFY-driven wake-ups bring new-node visibility from up-to-60 s
# (next discovery tick) down to ~500 ms on Postgres; the 60 s
# discovery loop still runs as the backstop for crash-shaped node
# loss (NOTIFY only fires on actual writes, not on crash exits).
if self._notify_dispatcher is not None:
self._notify_unsubscribe = self._notify_dispatcher.subscribe(
"services", self._on_services_notify
)
for target, name in [
(self._discovery_loop, "console-discovery"),
(self._sse_manager_thread, "console-sse"),
@@ -145,6 +161,10 @@ class ClusterCollector:
its ``finally`` cleanup (cancel tasks, close AsyncClient).
"""
self._running = False
if self._notify_unsubscribe is not None:
with contextlib.suppress(Exception):
self._notify_unsubscribe()
self._notify_unsubscribe = None
# Request cancellation of all SSE tasks so they don't block the
# manager's cleanup. The manager coroutine exits when _running is
# False and handles remaining task cancellation in its finally block.
@@ -156,6 +176,26 @@ class ClusterCollector:
t.join(timeout=5)
log.info("ClusterCollector stopped")
def _on_services_notify(self, notify: Notify) -> None:
"""Run a discovery tick when the ``services`` channel fires.
The dispatcher delivers both real Postgres notifications and
synthetic ``reconcile`` wake-ups after a reconnect both shape
the same way: re-read ``services`` and diff against in-memory
state. Re-uses :meth:`_discover_nodes` so the timer-driven
backstop and the NOTIFY-driven fast-path share one code path.
"""
from turnstone.core.storage._registry import StorageUnavailableError
if not self._running:
return
try:
self._discover_nodes()
except StorageUnavailableError:
pass # already logged by storage layer
except Exception:
log.exception("Node discovery error (notify-driven)")
def _fanout(self, event: dict[str, Any]) -> None:
"""Copy an event to all registered SSE listener queues."""
with self._listeners_lock:
+27
View File
@@ -19,6 +19,7 @@ from typing import TYPE_CHECKING, Any
from turnstone.core import session_worker
from turnstone.core.adapters._ui_cleanup import cleanup_session_ui
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.child_source import ClusterChildSource
from turnstone.core.children_registry import ChildrenRegistry
from turnstone.core.log import get_logger
@@ -67,6 +68,14 @@ class CoordinatorAdapter:
# method names which still exist as thin shims for the
# cluster-routing + cleanup callers.
self._registry = ChildrenRegistry()
# In-process wakeup primitive for ``wait_for_workstream``. The
# dispatch sink (:meth:`_dispatch_child_event`) calls
# ``notify(child_ws_id)`` after each translated child event;
# waiters block on per-call ``threading.Event``s instead of
# polling storage. Owned by the adapter so the manager-level
# exposure can simply delegate; ``CoordinatorClient`` picks it
# up via the coord client factory closure.
self._child_event_bus = ChildEventBus()
# Cross-node child events arrive via ``ClusterChildSource``
# (Stage 3 Step 2): a strategy that subscribes to the
# collector's listener channel and runs a daemon thread that
@@ -77,6 +86,17 @@ class CoordinatorAdapter:
# the collector reference is available.
self._child_source: ClusterChildSource | None = None
@property
def child_event_bus(self) -> ChildEventBus:
"""In-process wakeup bus consumed by ``wait_for_workstream``.
Exposed so the coord client factory in the console bootstrap
can pass it to :class:`CoordinatorClient` without reaching
into a private attr, and so :class:`SessionManager` can
delegate its own ``child_event_bus`` property here.
"""
return self._child_event_bus
def attach(self, manager: SessionManager) -> None:
"""Late-bind the owning :class:`SessionManager`.
@@ -639,6 +659,13 @@ class CoordinatorAdapter:
"detail": event.get("detail") or {},
}
_enqueue_on_ui(owning_ws.ui, coord_id, child_event)
# Wake any in-process ``wait_for_workstream`` subscriber on
# this child. Notify runs AFTER the UI enqueue so the SSE
# fan-out keeps priority (a wait that wakes early sees the
# state already enqueued for its owning dashboard). The bus
# is a no-op when no waiter is registered — the steady
# state for the hot dispatch path.
self._child_event_bus.notify(ws_id)
def _enqueue_on_ui(ui: Any, coord_ws_id: str, payload: dict[str, Any]) -> None:
+88
View File
@@ -0,0 +1,88 @@
"""Coordinator alias resolution shared by the placeholder API and the
session factory.
Both ``/v1/api/models`` (advertises the resolved default to the home
composer) and ``console/session_factory.py:factory`` (resolves the
alias new coordinator sessions launch on) walk the same three-tier
chain. Centralising it here means the tier names and the tier-2
validation policy live once the prior arrangement was two
implementations coupled by a "keep these in sync" comment, which is
exactly the drift trap that produced the historical bug where the
home composer advertised one alias while sessions ran on another.
Tiers, in priority order:
1. **Explicit pin** per-call ``model_alias`` arg (factory only) or
the ``coordinator.model_alias`` ConfigStore setting.
2. **System default** ``model.default_alias`` ConfigStore setting,
admin-managed in the Models tab. Validated against
``registry.has_alias()`` a stale or typo'd value falls through
with a logged warning rather than 503ing.
3. **Registry default** ``registry.default`` (config.toml
``[model].default``), guaranteed by the registry to resolve.
Tier 1 is intentionally passed through unvalidated by default: an
explicit operator pin should surface as 503 at ``registry.resolve``
when stale, not silently fall through to a different alias. Callers
that need stricter filtering (the placeholder API restricts to
enabled DB rows so the home composer doesn't advertise a model the
workstream picker can't offer) supply an ``alias_filter`` predicate
applied to every tier.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from turnstone.core.log import get_logger
if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.core.config_store import ConfigStore
from turnstone.core.model_registry import ModelRegistry
log = get_logger(__name__)
def resolve_coordinator_alias(
*,
explicit: str | None,
config_store: ConfigStore,
registry: ModelRegistry,
alias_filter: Callable[[str], bool] | None = None,
) -> str:
"""Resolve the effective coordinator alias through the three tiers.
See module docstring for the full chain. Returns the concrete
alias name, or ``""`` when every tier failed (rare only when
``registry.default`` itself fails the filter).
"""
def _accept(alias: str) -> bool:
if not alias:
return False
return alias_filter(alias) if alias_filter is not None else True
explicit_alias = (explicit or "").strip()
if not explicit_alias:
explicit_alias = (config_store.get("coordinator.model_alias") or "").strip()
if _accept(explicit_alias):
return explicit_alias
fallback_alias = (config_store.get("model.default_alias") or "").strip()
if fallback_alias and not registry.has_alias(fallback_alias):
log.warning(
"coord_alias.model_default_alias_unknown alias=%r "
"— falling through to registry.default",
fallback_alias,
)
fallback_alias = ""
if _accept(fallback_alias):
return fallback_alias
registry_default = registry.default or ""
if _accept(registry_default):
return registry_default
return ""
+473 -74
View File
@@ -73,12 +73,27 @@ WAIT_MAX_WS_IDS: int = 32
# wait_for_workstream again with the same ws_ids — each call re-arms freshly.
WAIT_MAX_TIMEOUT: float = 600.0
# Storage-poll cadence. 500ms is short enough that the wait terminates
# promptly after a child finishes (well under the human-perceptible-latency
# floor), and long enough that a 60s wait incurs at most 120 cheap row
# reads — still cheaper than the 20+ inspect_workstream model turns the
# tool replaces.
WAIT_POLL_INTERVAL: float = 0.5
# Maximum ``event.wait`` interval in the bus-driven wait loop.
# A long-running stuck child would otherwise look dead in the sidebar UI
# because the ``wait_progress`` SSE emission piggybacks on the wait loop
# — capping at 2 s keeps the heartbeat visible without flooding storage.
# Today's polling effectively snapshots every 500 ms; 2 s preserves a
# similar liveness feel while cutting per-listener SSE traffic ~4x in the
# steady-state-quiescent case. Tunable post-merge if profiling shows
# storage-read pressure on state-change wakes.
#
# **Worst-case completion latency**: 2 s. ``SessionManager.set_state``
# buffers non-ERROR storage writes through ``StateWriter`` (async-flushed
# at ~1 s cadence) while ``emit_state`` fans the event out immediately —
# a bus-driven wake can therefore beat the flusher and read pre-transition
# state on a terminal transition, then re-block on ``event.wait`` until
# the heartbeat cap fires. Pre-bus the 0.5 s poll bounded this at 0.5 s.
# Going to 2 s is intentional: the 4x SSE-traffic reduction in the
# steady-state-quiescent case outweighs the worst-case latency
# regression on the most common terminal transition, and a model issuing
# a follow-up ``inspect_workstream`` (the pre-bus pattern this tool
# replaces) was already paying multi-second model-turn latency per probe.
WAIT_HEARTBEAT_INTERVAL: float = 2.0
# Per-ws cap on the inline ``message`` field bundled into wait_for_workstream
# results. Sized so a fan-out of 32 children at the cap is ~320 KiB of
@@ -184,6 +199,7 @@ def load_task_envelope(storage: Any, ws_id: str) -> tuple[dict[str, Any], bool]:
if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.core.child_event_bus import ChildEventBus
from turnstone.core.storage._protocol import StorageBackend
log = get_logger(__name__)
@@ -313,6 +329,7 @@ class CoordinatorClient:
user_id: str,
timeout: float = 30.0,
http_client: httpx.Client | None = None,
child_event_bus: ChildEventBus,
) -> None:
self._base_url = console_base_url.rstrip("/")
self._storage = storage
@@ -325,6 +342,12 @@ class CoordinatorClient:
# with the coordinator session.
self._http = http_client or httpx.Client(timeout=timeout)
self._owns_http = http_client is None
# In-process wakeup bus for ``wait_for_workstream``. The wait
# loop blocks on a ``threading.Event`` keyed by ws_id and only
# re-snapshots storage on state-change wakes or the heartbeat
# cap. Owned by ``CoordinatorAdapter`` in production; tests
# pass their own instance.
self._child_event_bus = child_event_bus
# tasks per-ws lock cache — populated lazily by _task_lock().
# Single-session so a plain dict behind a coarse lock is fine;
# WeakValueDictionary isn't needed (entries live as long as the
@@ -447,6 +470,27 @@ class CoordinatorClient:
return False
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
def _row_in_own_subtree(self, ws_id: str, row: dict[str, Any] | None) -> bool:
"""Row-level subtree predicate sharing one home for read paths.
Both :meth:`wait_for_workstream`'s pre-loop ownership filter and
its inner ``_snapshot_all`` already have the workstream row in
hand (from ``get_workstreams_batch``). Funneling them through
the same 4-line check keeps the predicate in lockstep with
:meth:`_is_own_subtree` (used by mutating ops) both require
``parent_ws_id`` AND ``user_id`` parity so a corrupted or
forged ``parent_ws_id`` alone can't satisfy the gate on either
path. Returns False on a missing / None row so callers can
safely pass ``rows.get(wid)``.
"""
if ws_id == self._coord_ws_id:
return True
if row is None:
return False
if row.get("parent_ws_id") != self._coord_ws_id:
return False
return bool(row.get("user_id")) and row.get("user_id") == self._user_id
# -- model-invoked mutating ops (HTTP) ---------------------------------
def spawn(
@@ -562,7 +606,7 @@ class CoordinatorClient:
_WAIT_TERMINAL_STATES: ClassVar[frozenset[str]] = WAIT_TERMINAL_STATES
_WAIT_MAX_WS_IDS: ClassVar[int] = WAIT_MAX_WS_IDS
_WAIT_MAX_TIMEOUT: ClassVar[float] = WAIT_MAX_TIMEOUT
_WAIT_POLL_INTERVAL: ClassVar[float] = WAIT_POLL_INTERVAL
_WAIT_HEARTBEAT_INTERVAL: ClassVar[float] = WAIT_HEARTBEAT_INTERVAL
def wait_for_workstream(
self,
@@ -721,12 +765,7 @@ class CoordinatorClient:
snaps: dict[str, dict[str, Any]] = {}
for wid in cleaned:
row = rows.get(wid)
if row is None:
snaps[wid] = {"state": "denied", "tokens": 0}
continue
is_self = wid == self._coord_ws_id
is_own_child = row.get("parent_ws_id") == self._coord_ws_id
if not (is_self or is_own_child):
if row is None or not self._row_in_own_subtree(wid, row):
snaps[wid] = {"state": "denied", "tokens": 0}
continue
snaps[wid] = {
@@ -760,54 +799,119 @@ class CoordinatorClient:
last_results: dict[str, dict[str, Any]] = {}
complete = False
while True:
results = _snapshot_all()
last_results = results
if progress_callback is not None:
try:
progress_callback(results, time.monotonic() - start)
except Exception:
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
settled = [_is_settled(snap) for snap in results.values()]
# ``since`` — orthogonal to mode. If the caller supplied a
# prior snapshot, any diff on a ws_id that IS in ``since_map``
# exits the wait so a follow-up call doesn't re-count
# already-terminal children. ws_ids absent from ``since_map``
# are ignored for the diff-exit check — they fall through to
# the normal mode='any' / mode='all' conditions below. This
# prevents a disjoint since-dict from exiting on tick one
# with complete=True (previous shape did, silently).
if since_map and any(
_diff_since(snap, since_map[wid])
for wid, snap in results.items()
if wid in since_map
):
complete = True
break
if mode == "any":
if any(real_terminal):
# Subscribe to in-process state-change events for the watched
# ws_ids when the bus is wired. ``register_waiter`` returns a
# single ``threading.Event`` registered against every id so a
# wait on [A, B, C] wakes on any of A/B/C changing. Bus is
# optional so test fixtures that don't wire it fall back to the
# legacy ``time.sleep`` cadence with no behaviour change.
#
# **Defense-in-depth ownership filter**: ``_dispatch_child_event``
# fires ``bus.notify(ws_id)`` for every ws_id in *any* coord's
# registry on this console process, so a foreign ws_id passed by
# an untrusted coord LLM (prompt injection) would otherwise leak
# wake-up timing as a side channel — _snapshot_all returns
# ``denied`` for the content, but the *time* at which the wait
# un-blocked would correlate with the foreign ws_id's next
# state-class event. Filter ``cleaned`` to own-subtree ids
# before registering; foreign / missing ws_ids stay in the
# snapshot list so they still surface as ``denied`` in
# ``_snapshot_all`` and exit via the pure-denied short-circuit
# below. Predicate shared with ``_snapshot_all`` via
# :meth:`_row_in_own_subtree`.
try:
pre_rows = self._storage.get_workstreams_batch(cleaned)
except Exception:
log.debug("coord_client.wait.ownership_filter_failed", exc_info=True)
pre_rows = {wid: None for wid in cleaned}
own_subtree = [wid for wid in cleaned if self._row_in_own_subtree(wid, pre_rows.get(wid))]
bus = self._child_event_bus
wake_event = bus.register_waiter(own_subtree) if own_subtree else None
try:
while True:
# Clear BEFORE the storage snapshot to close the
# subscribe/check race: any ``notify`` between clear
# and the next ``wake_event.wait`` leaves the Event
# set, so the wait returns immediately and the loop
# re-snapshots without losing the wake-up.
if wake_event is not None:
wake_event.clear()
results = _snapshot_all()
last_results = results
if progress_callback is not None:
try:
progress_callback(results, time.monotonic() - start)
except Exception:
log.debug("coord_client.wait.progress_cb_failed", exc_info=True)
real_terminal = [_is_real_terminal(snap) for snap in results.values()]
settled = [_is_settled(snap) for snap in results.values()]
# ``since`` — orthogonal to mode. If the caller supplied a
# prior snapshot, any diff on a ws_id that IS in ``since_map``
# exits the wait so a follow-up call doesn't re-count
# already-terminal children. ws_ids absent from ``since_map``
# are ignored for the diff-exit check — they fall through to
# the normal mode='any' / mode='all' conditions below. This
# prevents a disjoint since-dict from exiting on tick one
# with complete=True (previous shape did, silently).
if since_map and any(
_diff_since(snap, since_map[wid])
for wid, snap in results.items()
if wid in since_map
):
complete = True
break
# Pure-denied list: every snap is settled but none is a
# real terminal — no work to wait for. Short-circuit so
# the model sees the denied results immediately rather
# than spinning the timeout (``complete=False`` because
# the wait condition never had a real chance to fire).
if all(settled):
if mode == "any":
if any(real_terminal):
complete = True
break
# Pure-denied list: every snap is settled but none is a
# real terminal — no work to wait for. Short-circuit so
# the model sees the denied results immediately rather
# than spinning the timeout (``complete=False`` because
# the wait condition never had a real chance to fire).
if all(settled):
break
else: # mode == "all"
if all(settled):
# Every ws_id is settled (real-terminal or denied).
# The wait condition is met — the model gets the
# full results dict and decides what each terminal
# state means.
complete = True
break
remaining = deadline - time.monotonic()
if remaining <= 0:
break
else: # mode == "all"
if all(settled):
# Every ws_id is settled (real-terminal or denied).
# The wait condition is met — the model gets the
# full results dict and decides what each terminal
# state means.
complete = True
break
remaining = deadline - time.monotonic()
if remaining <= 0:
break
time.sleep(min(self._WAIT_POLL_INTERVAL, remaining))
if wake_event is not None:
# Block until a child state-change notify fires OR
# the heartbeat cap expires (so a stuck child still
# emits a periodic ``wait_progress`` for the
# sidebar UX). Heartbeat cap is the only timer —
# the bus is the wake source. See
# ``WAIT_HEARTBEAT_INTERVAL`` (module top) for the
# worst-case completion-latency rationale: 2 s is
# a deliberate 4x trade vs the pre-bus 0.5 s poll.
wake_event.wait(min(remaining, self._WAIT_HEARTBEAT_INTERVAL))
else:
# Pure-foreign / pure-denied list: every cleaned
# ws_id was filtered out of ``own_subtree`` so the
# bus has nothing to wake on. The pure-denied
# short-circuit above exits ``mode='any'`` on the
# first tick; ``mode='all'`` falls through to here
# and must burn the timeout. Use the heartbeat
# cadence for the deadline carve-up so
# ``progress_callback`` keeps firing.
time.sleep(min(self._WAIT_HEARTBEAT_INTERVAL, remaining))
finally:
# Always unregister so a crash mid-wait can't leak the
# registration past one wait's lifetime. Bus discards
# empty buckets so long-lived buses don't accumulate dead
# keys after many waits. Unregister against the same
# ``own_subtree`` list the register call used — passing
# ``cleaned`` here would silently no-op for foreign ids
# but pass an unknown bucket to ``unregister_waiter``.
if wake_event is not None:
bus.unregister_waiter(own_subtree, wake_event)
# Bundle each terminal child's last assistant message inline so the
# coordinator LLM doesn't have to follow up with one
# ``inspect_workstream`` per ws. Only ``idle`` / ``error`` ws_ids
@@ -816,7 +920,7 @@ class CoordinatorClient:
# subset across a small thread pool — at the WAIT_MAX_WS_IDS=32
# cap, 8 workers cuts a worst-case all-idle fan-out from 32
# sequential storage round-trips down to 4 batches, which lands
# inside the WAIT_POLL_INTERVAL the model already tolerates
# inside the WAIT_HEARTBEAT_INTERVAL the model already tolerates
# between ticks. Storage backends use SQLAlchemy with
# ``check_same_thread=False`` (SQLite) / a connection pool
# (Postgres), so concurrent reads from the worker pool are safe.
@@ -1174,21 +1278,29 @@ class CoordinatorClient:
allowed_tools: list[str] = [str(t) for t in allowed_full[:_SKILL_TOOLS_PROJECTION_CAP]]
if len(allowed_full) > _SKILL_TOOLS_PROJECTION_CAP:
allowed_tools.append(f"+{len(allowed_full) - _SKILL_TOOLS_PROJECTION_CAP} more")
skills.append(
{
"name": r.get("name") or "",
"category": r.get("category") or "",
"tags": tags,
"version": r.get("version") or "",
"description": r.get("description") or "",
"model": r.get("model") or "",
"enabled": bool(r.get("enabled")),
"risk_level": r.get("risk_level") or "",
"activation": r.get("activation") or "",
"kind": r["kind"],
"allowed_tools": allowed_tools,
}
)
skill_row: dict[str, Any] = {
"name": r.get("name") or "",
"category": r.get("category") or "",
"tags": tags,
"version": r.get("version") or "",
"description": r.get("description") or "",
"model": r.get("model") or "",
"enabled": bool(r.get("enabled")),
"risk_level": r.get("risk_level") or "",
"activation": r.get("activation") or "",
"kind": r["kind"],
}
# Omit ``allowed_tools`` when empty: an empty list reads as
# "no tools are usable by this skill" to a model that doesn't
# know the semantics, but the actual meaning is "no tools are
# pre-approved (auto-approve exemption list)". Real
# misdiagnosis happened in testing when a code-review skill
# with no auto-approve allowlist looked like it had been
# spawned with zero tool access. Dropping the key altogether
# when empty removes the ambiguity at the source.
if allowed_tools:
skill_row["allowed_tools"] = allowed_tools
skills.append(skill_row)
return {"skills": skills, "truncated": truncated}
# ------------------------------------------------------------------
@@ -1687,6 +1799,293 @@ def _serialize_verdicts(rows: list[Any]) -> list[dict[str, Any]]:
return out
# ---------------------------------------------------------------------------
# inspect_workstream — tiered output compression
# ---------------------------------------------------------------------------
#
# A coord doing a fan-out wave of inspect_workstream calls against
# tool-heavy children can blow the context budget on raw output alone
# (one child with a 100 KB bash result × N children). The previous
# safety net was ``_truncate_output``'s head+tail strategy, which
# silently drops *middle* messages — exactly the wrong shape for a
# coordinator trying to understand a child's trajectory (the LAST
# message tells the model what the child concluded; the FIRST sets
# the brief; the middle is the connective tissue).
#
# The three-tier degradation pattern matches the ``search`` tool's
# Tier-1/Tier-2/Tier-3 ladder at ``session.py:_format_search_results``.
# First tier whose serialized size fits the budget wins; the LLM
# learns which tier it got via the ``_tier`` field in the response
# (no API change to the coordinator tool).
#
# Budget chosen well under ``tool_truncation`` (typically 256 KB+) so
# the head+tail safety net never fires for inspect_workstream — that
# strategy silently drops middle messages, which is exactly the
# pathology this formatter exists to avoid.
_INSPECT_OUTPUT_BUDGET: int = 32_768
# Per-message head/tail snip when Tier 2 needs to compress content.
# Head dominates because the first ~600 chars of an assistant message
# usually contains the conclusion / direction; the tail is the
# follow-through. Tool results compress similarly: head shows what
# the tool was asked / what it found at the top; tail shows the final
# state / error suffix.
_INSPECT_MSG_CONTENT_HEAD: int = 600
_INSPECT_MSG_CONTENT_TAIL: int = 300
# Skeleton-tier preview length on the last assistant message. Single
# value because the skeleton wants ONE meaningful signal ("what did
# the child last say"), not a head/tail snip.
_INSPECT_SKELETON_LAST_PREVIEW: int = 400
# Snip lengths for tool-call ``function.arguments`` strings on
# assistant turns. Tighter than content snipping because tool calls
# often appear in clusters (10+ per turn for a fan-out) and the
# arguments JSON is dense — keep just enough to see what was invoked
# and the head of the args structure.
_INSPECT_TOOL_ARG_HEAD: int = 300
_INSPECT_TOOL_ARG_TAIL: int = 100
# Bytes ``_snip_head_tail`` reserves for the elision marker itself
# (``\n...[N chars elided]...\n``). A text shorter than
# ``head + tail + this margin`` passes through unsnipped — snipping
# would cost more bytes (the marker) than it saves.
_INSPECT_ELISION_MARGIN: int = 64
# Message-list trim ladder for the compact tier when per-message
# content snipping alone doesn't free enough budget. Each rung is
# ``(head_count, tail_count)`` — keep the first N + last M messages,
# elide the middle as ``{"_omitted": K}``. Tail-weighted because the
# last assistant turn carries the load-bearing "what did the child
# conclude" signal (same rationale as ``_inspect_skeleton``'s
# last-assistant preview). Tried in order; first rung whose
# serialized emission fits the budget wins. Mirrors the per-file
# sample ladder in ``_format_search_results`` at session.py:254.
_INSPECT_LIST_TRIM_LADDER: tuple[tuple[int, int], ...] = ((20, 30), (10, 20), (5, 10))
def _snip_head_tail(text: str, head: int, tail: int) -> str:
"""Head/tail snip with elision marker; passthrough when shorter than threshold."""
if not isinstance(text, str) or len(text) <= head + tail + _INSPECT_ELISION_MARGIN:
return text
elided = len(text) - head - tail
return text[:head] + f"\n...[{elided} chars elided]...\n" + text[-tail:]
def _compact_tool_calls(tool_calls: Any) -> Any:
"""Snip ``function.arguments`` on each tool-call entry; keep ``id`` and
``function.name`` verbatim.
OpenAI shape: ``[{"id": ..., "type": "function", "function":
{"name": ..., "arguments": "<json-string>"}}, ...]``. The
arguments string is the dominant size term on a fan-out turn that
issued many tool calls with multi-KB JSON arguments each;
preserving them verbatim re-opens the same size pressure the
compact tier is trying to relieve. Non-list / non-dict entries
pass through so a future shape change doesn't crash the formatter.
"""
if not isinstance(tool_calls, list):
return tool_calls
out: list[Any] = []
for call in tool_calls:
if not isinstance(call, dict):
out.append(call)
continue
compact_call: dict[str, Any] = {}
for k in ("id", "type"):
v = call.get(k)
if v:
compact_call[k] = v
func = call.get("function")
if isinstance(func, dict):
compact_func: dict[str, Any] = {}
name = func.get("name")
if name:
compact_func["name"] = name
args = func.get("arguments", "")
if args:
compact_func["arguments"] = _snip_head_tail(
args, _INSPECT_TOOL_ARG_HEAD, _INSPECT_TOOL_ARG_TAIL
)
compact_call["function"] = compact_func
out.append(compact_call)
return out
def _compact_message(msg: dict[str, Any]) -> dict[str, Any]:
"""Tier-2 per-message projection: keep role + identifier keys, snip content + tool_calls.
Tool-call linkage is the load-bearing "what happened" signal:
``tool_call_id`` on the result side matches an ``id`` in
``tool_calls`` on the issuing assistant turn. Stripping
``tool_calls`` (the pre-fix shape) left tool results dangling
against an invisible call the audit reader could see "bash
returned X" but not "the assistant asked for ``ls /tmp``". The
``arguments`` string is the size offender, so we snip it head/tail
rather than dropping the call entirely.
"""
content = msg.get("content", "")
snipped = _snip_head_tail(content, _INSPECT_MSG_CONTENT_HEAD, _INSPECT_MSG_CONTENT_TAIL)
compact: dict[str, Any] = {"role": msg.get("role"), "content": snipped}
# Tool-result linkage (result-side keys).
for k in ("tool_name", "tool_call_id", "name"):
v = msg.get(k)
if v:
compact[k] = v
# Tool-call request linkage (issuing-side list), snipped per-call.
tool_calls = msg.get("tool_calls")
if tool_calls:
compact["tool_calls"] = _compact_tool_calls(tool_calls)
return compact
def _inspect_skeleton(result: dict[str, Any]) -> dict[str, Any]:
"""Tier-3 fallback: state + counts + last assistant preview + terminal info.
Drops every message, keeping only aggregate signal: state, message
count, role distribution, verdict count + risk distribution, and a
short preview of the most recent assistant turn (the "what did this
child last say" signal). Terminal-state fields (``close_reason``,
``last_error``) and the ``live`` block pass through unchanged
because they're already small and load-bearing.
"""
messages = result.get("messages") or []
verdicts = result.get("verdicts") or []
role_counts: dict[str, int] = {}
for m in messages:
role = m.get("role") if isinstance(m, dict) else None
if role:
role_counts[role] = role_counts.get(role, 0) + 1
verdicts_by_risk: dict[str, int] = {}
for v in verdicts:
if isinstance(v, dict):
risk = v.get("risk_level") or "unknown"
verdicts_by_risk[risk] = verdicts_by_risk.get(risk, 0) + 1
last_preview = ""
for m in reversed(messages):
if not isinstance(m, dict) or m.get("role") != "assistant":
continue
c = m.get("content", "")
if isinstance(c, str) and c:
last_preview = c[:_INSPECT_SKELETON_LAST_PREVIEW]
if len(c) > _INSPECT_SKELETON_LAST_PREVIEW:
last_preview += "..."
break
skeleton: dict[str, Any] = {
# Storage row keys verbatim from ``get_workstreams_batch``
# (the projection backing ``get_workstream`` → ``inspect()``):
# ``ws_id``, ``skill_id``. No fallback to ``id`` / ``skill``
# — fail loud on storage column drift rather than silently
# emitting null.
"ws_id": result["ws_id"],
"state": result.get("state"),
"title": result.get("title"),
"skill": result["skill_id"],
"message_count": len(messages),
"roles": role_counts,
"verdict_count": len(verdicts),
"verdicts_by_risk": verdicts_by_risk,
"last_assistant_preview": last_preview,
"_tier": "skeleton",
"_tier_note": (
"Output exceeded the inspect_workstream budget at both full and compact "
"tiers; skeleton-only. Re-call with a smaller ``message_limit`` to fit "
"the compact tier, or read individual messages via the storage admin path."
),
}
for k in ("close_reason", "last_error", "live"):
v = result.get(k)
if v:
skeleton[k] = v
return skeleton
def _format_inspect_tiered(result: dict[str, Any], *, budget: int = _INSPECT_OUTPUT_BUDGET) -> str:
"""Serialize an ``inspect_workstream`` result with tiered degradation.
Tier 1 (full): every message verbatim used when the size fits.
Tier 2 (compact): per-message ``{role, head/tail-snipped content,
tool linkage, snipped tool_calls.arguments}`` for
every message, then a head+tail message-list trim
ladder when content snipping alone doesn't free
enough budget.
Tier 3 (skeleton): no messages counts + last assistant preview only.
First emission whose JSON serialization fits ``budget`` wins.
``_tier`` appears on every non-error emission so the coordinator
LLM (and any audit reader) can see which compression rung the
output landed on without inferring from length. Error-shape
results (missing or cross-tenant ws_id) bypass tiering entirely
they're already small and the ``error`` key signals the shape.
The intermediate Tier-2 list-trim rungs exist because content
snipping alone fails on workloads where many small messages
overflow the budget by sheer count (``message_limit=200`` × a few
hundred chars each). In that regime, dropping content-snipping
saves zero bytes per message, so without the list-trim ladder
Tier-2 produces output strictly larger than Tier-1 (added
``_tier_note``) and the formatter fell through to skeleton
losing every message when a head+tail message-list trim would
have preserved dozens. Mirrors the per-file sample ladder in
``_format_search_results`` (session.py:_SEARCH_TIER2_SAMPLE_LADDER).
"""
if "error" in result:
# Cross-tenant guard / not-found responses — pass through.
return json.dumps(result, default=str, separators=(",", ":"))
tier1 = {**result, "_tier": "full"}
out1 = json.dumps(tier1, default=str, separators=(",", ":"))
if len(out1) <= budget:
return out1
messages = result.get("messages") or []
compact_msgs = [_compact_message(m) if isinstance(m, dict) else m for m in messages]
tier2_note_full = (
"Output exceeded the inspect_workstream budget at the full tier; messages "
"are head/tail-snipped at "
f"{_INSPECT_MSG_CONTENT_HEAD}/{_INSPECT_MSG_CONTENT_TAIL} chars. Re-call "
"with a smaller ``message_limit`` for a tighter tail, or include_provider_"
"content=False if it was on."
)
tier2 = {
**result,
"messages": compact_msgs,
"_tier": "compact",
"_tier_note": tier2_note_full,
}
out2 = json.dumps(tier2, default=str, separators=(",", ":"))
if len(out2) <= budget:
return out2
# Tier-2 list-trim ladder: keep head N + tail M, elide the middle.
# Tail-weighted because the recent turns carry the load-bearing
# signal ("what did the child conclude") — same reason
# ``_inspect_skeleton`` keeps a last-assistant preview rather than
# a first-user preview.
total = len(compact_msgs)
for head_n, tail_n in _INSPECT_LIST_TRIM_LADDER:
if head_n + tail_n >= total:
# Rung doesn't actually trim — would re-emit Tier-2 verbatim.
continue
omitted = total - head_n - tail_n
trimmed: list[Any] = (
compact_msgs[:head_n] + [{"_omitted": omitted}] + compact_msgs[-tail_n:]
)
tier2_trim_note = (
f"Output exceeded the inspect_workstream budget at the compact tier; "
f"keeping first {head_n} + last {tail_n} of {total} messages, eliding "
f"{omitted} middle messages. Re-call with a smaller ``message_limit`` "
"to fit the full compact tier."
)
tier2_trim = {
**result,
"messages": trimmed,
"_tier": "compact",
"_tier_note": tier2_trim_note,
}
out2_trim = json.dumps(tier2_trim, default=str, separators=(",", ":"))
if len(out2_trim) <= budget:
return out2_trim
skeleton = _inspect_skeleton(result)
return json.dumps(skeleton, default=str, separators=(",", ":"))
# ---------------------------------------------------------------------------
# wait_for_workstream — last-message extraction
# ---------------------------------------------------------------------------
+362
View File
@@ -0,0 +1,362 @@
"""Console-side multiplexer for PostgreSQL ``LISTEN``/``NOTIFY`` events.
Holds a single dedicated listen connection (via :meth:`StorageBackend.listen`),
drains it on a listener thread, and fans notifications out to per-channel
handlers on a dedicated dispatch thread so a slow handler doesn't back up
the connection.
Consumers register at construction time by passing their channel in
:attr:`channels`, then call :meth:`subscribe` to attach a handler.
Registering an undeclared channel raises the construction list is the
single source of truth so wire-in is explicit (each future consumer
touches the dispatcher construction call site at
``turnstone/console/server.py::main`` to add its channel).
On connection loss the listener wakes its handlers with a synthetic
``Notify(channel, payload="reconcile", pid=0)`` so every consumer
re-reads the underlying rows; their normal "reconcile on any wake-up"
code path covers both real notifications and reconnect recovery
identically.
"""
from __future__ import annotations
import contextlib
import queue
import threading
import time
from typing import TYPE_CHECKING
from turnstone.core.log import get_logger
from turnstone.core.storage._notify import Notify, NotifyConnectionError
if TYPE_CHECKING:
from collections.abc import Callable, Iterable
from turnstone.core.storage._protocol import StorageBackend
log = get_logger(__name__)
# Backoff (seconds) between reconnect attempts after :class:`NotifyConnectionError`.
# Doubles each failure, capped at the max — long enough that a Postgres outage
# doesn't burn CPU on reconnect spins, short enough that recovery is fast.
_RECONNECT_BACKOFF_INITIAL: float = 1.0
_RECONNECT_BACKOFF_MAX: float = 30.0
# Poll cadence on the listener thread. Short enough that ``stop`` lands
# promptly without joining a long-blocked notifies() call; long enough
# that we don't burn CPU on empty polls.
_LISTENER_POLL_TIMEOUT: float = 1.0
# Cap on the inter-thread dispatch queue. Drops oldest if a slow handler
# falls behind (logs once per drop bucket). Sized larger than the expected
# steady-state notification rate (services trigger fires only on
# register/restart/deregister — order of hundreds per hour at the 100-node
# design ceiling).
_DISPATCH_QUEUE_MAX: int = 1024
class NotifyDispatcher:
"""Holds the dedicated listen connection and fans events to handlers.
Lifecycle: construct with the declared channel list, attach
handlers via :meth:`subscribe`, then call :meth:`start`. :meth:`stop`
closes the connection and joins the worker threads. Idempotent in
both directions so console teardown can call stop unconditionally.
"""
def __init__(self, storage: StorageBackend, channels: Iterable[str]) -> None:
ch_list = [str(c) for c in channels if c]
if not ch_list:
msg = "NotifyDispatcher requires at least one declared channel"
raise ValueError(msg)
self._storage = storage
self._channels: list[str] = list(dict.fromkeys(ch_list)) # de-dupe, preserve order
self._handlers: dict[str, list[Callable[[Notify], None]]] = {
ch: [] for ch in self._channels
}
self._handlers_lock = threading.Lock()
self._lifecycle_lock = threading.Lock()
self._started = False
self._stopping = threading.Event()
self._listener_thread: threading.Thread | None = None
self._dispatch_thread: threading.Thread | None = None
self._dispatch_queue: queue.Queue[Notify | None] = queue.Queue(maxsize=_DISPATCH_QUEUE_MAX)
self._drop_count = 0
# Set inside :meth:`_listener_loop` after each successful
# ``storage.listen`` open; cleared on disconnect. Callers use
# :meth:`wait_until_ready` after :meth:`start` to block until the
# listener is actually listening (matters when the next caller
# action is a ``notify`` whose delivery requires the LISTEN to
# already be in place — e.g. tests, or any startup-path traffic
# that should be reactive from the first event).
self._listener_ready = threading.Event()
@property
def channels(self) -> list[str]:
"""Snapshot copy of declared channels."""
return list(self._channels)
def subscribe(self, channel: str, handler: Callable[[Notify], None]) -> Callable[[], None]:
"""Attach ``handler`` to ``channel``; return an unsubscribe callable.
Safe to call before or after :meth:`start`. Raises if the
channel was not declared at construction time the channel
list is fixed so the dispatcher knows up-front which LISTENs
to issue (consumers added in follow-up PRs touch the
construction call site).
"""
if channel not in self._handlers:
msg = (
f"channel {channel!r} not declared at construction; "
f"declared channels: {sorted(self._handlers)}"
)
raise ValueError(msg)
with self._handlers_lock:
self._handlers[channel].append(handler)
def _unsubscribe() -> None:
with self._handlers_lock, contextlib.suppress(ValueError):
self._handlers[channel].remove(handler)
return _unsubscribe
def start(self) -> None:
"""Open the listen stream and start the listener + dispatch threads.
Idempotent repeat calls log a debug line and return without
spawning a second listener.
"""
with self._lifecycle_lock:
if self._started:
log.debug("notify_dispatcher.start_noop_already_started")
return
self._started = True
self._stopping.clear()
# Clear ready so a stop/start cycle's wait_until_ready only
# returns True after the new listener has actually opened.
self._listener_ready.clear()
self._listener_thread = threading.Thread(
target=self._listener_loop,
name="notify-dispatcher-listener",
daemon=True,
)
self._dispatch_thread = threading.Thread(
target=self._dispatch_loop,
name="notify-dispatcher-dispatch",
daemon=True,
)
self._listener_thread.start()
self._dispatch_thread.start()
log.info(
"notify_dispatcher.started",
channels=self._channels,
)
def wait_until_ready(self, timeout: float = 5.0) -> bool:
"""Block until the listener has opened its stream, or ``timeout`` elapses.
Returns ``True`` when the listener is ready (``LISTEN`` issued
for every declared channel on PG; subscriber queues registered
on SQLite), ``False`` on timeout. Cleared automatically on
disconnect call again after a reconnect to wait for the next
successful reopen.
Doesn't replace :meth:`start` — call ``start()`` first, then
``wait_until_ready()`` for the explicit sync point. Production
startup typically doesn't need this (the first real event tends
to arrive well after the listener is up); tests use it to close
the start-vs-notify race window.
"""
return self._listener_ready.wait(timeout=timeout)
def stop(self, timeout: float = 5.0) -> None:
"""Signal shutdown and join the worker threads.
Idempotent safe to call multiple times. Workers exit on the
next iteration of their poll loops; :meth:`stop` blocks up to
``timeout`` seconds per thread before giving up (the threads are
daemons so the process can exit regardless).
"""
with self._lifecycle_lock:
if not self._started:
return
self._stopping.set()
listener = self._listener_thread
dispatcher = self._dispatch_thread
# Sentinel wakes the dispatch loop out of queue.get().
with contextlib.suppress(queue.Full):
self._dispatch_queue.put_nowait(None)
if listener is not None:
listener.join(timeout=timeout)
if dispatcher is not None:
dispatcher.join(timeout=timeout)
with self._lifecycle_lock:
self._listener_thread = None
self._dispatch_thread = None
self._started = False
log.info("notify_dispatcher.stopped")
# ------------------------------------------------------------------
# Internal threading
# ------------------------------------------------------------------
def _listener_loop(self) -> None:
"""Drain the storage stream onto the dispatch queue, reconnecting on loss.
After any disconnect whether surfaced through the stream's
:class:`NotifyConnectionError` (post-open ``poll`` failure) or
through the generic exception path (``psycopg.connect`` /
initial ``LISTEN`` execute failures during reopen, which are
NOT wrapped by the stream) the loop sets a ``reconcile_pending``
flag, waits the backoff, then enqueues one synthetic ``reconcile``
notify per channel ONLY after the next stream successfully
reopens. Handlers see the synthetic notify and re-read the
relevant rows on the same code path they use for any real event,
closing the missed-notification window regardless of which
exception type caused the disconnect.
"""
backoff = _RECONNECT_BACKOFF_INITIAL
reconcile_pending = False
while not self._stopping.is_set():
try:
with self._storage.listen(self._channels) as stream:
log.debug(
"notify_dispatcher.stream_open",
channels=self._channels,
)
# Stream is open — reset backoff for the next outage
# and flush any pending reconcile so consumers see a
# wake-up against a now-live DB.
backoff = _RECONNECT_BACKOFF_INITIAL
if reconcile_pending:
self._synthesize_reconcile()
reconcile_pending = False
# Signal ``wait_until_ready`` callers that LISTEN is
# in place (PG) / subscriber queues are bound
# (SQLite). Must come AFTER the synthesize so any
# post-reconnect reconcile reaches handlers before
# the caller assumes "fresh notifies will deliver".
self._listener_ready.set()
while not self._stopping.is_set():
batch = stream.poll(_LISTENER_POLL_TIMEOUT)
for n in batch:
self._enqueue(n)
except NotifyConnectionError as exc:
if self._stopping.is_set():
return
self._listener_ready.clear()
log.warning(
"notify_dispatcher.connection_lost",
error=str(exc),
backoff_seconds=backoff,
)
reconcile_pending = True
if self._stopping.wait(backoff):
return
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
except Exception:
if self._stopping.is_set():
return
self._listener_ready.clear()
log.exception("notify_dispatcher.listener_unexpected_error")
reconcile_pending = True
if self._stopping.wait(backoff):
return
backoff = min(backoff * 2.0, _RECONNECT_BACKOFF_MAX)
log.debug("notify_dispatcher.listener_exiting")
def _synthesize_reconcile(self) -> None:
"""Push one synthetic ``reconcile`` notify per channel on reconnect.
Reconcile-on-wake is the same logic handlers run for any real
notification, so a single synthetic event per channel covers
any notifications missed during the connection-loss window.
"""
for ch in self._channels:
self._enqueue(Notify(channel=ch, payload="reconcile", pid=0))
def _enqueue(self, notify: Notify) -> None:
"""Put a notify on the dispatch queue, dropping oldest on overflow."""
try:
self._dispatch_queue.put_nowait(notify)
except queue.Full:
# Drop oldest to make room — a slow handler shouldn't be able
# to silently block the listener thread. Log once per power
# of two so a sustained backpressure problem shows up
# in logs without flooding.
self._drop_count += 1
if self._drop_count & (self._drop_count - 1) == 0:
log.warning(
"notify_dispatcher.dispatch_queue_full_dropping_oldest",
drops_total=self._drop_count,
channel=notify.channel,
)
with contextlib.suppress(queue.Empty):
self._dispatch_queue.get_nowait()
with contextlib.suppress(queue.Full):
self._dispatch_queue.put_nowait(notify)
def _dispatch_loop(self) -> None:
"""Pull notifies off the queue and invoke handlers per channel.
Notifies queued on the same channel coalesce per dispatch batch:
after blocking ``get()`` returns one notify, the loop drains
whatever else is already queued and collapses to one
``per-channel`` notify before invoking handlers. The payload is
signal-only by design (handlers reconcile by re-reading the
underlying rows), so N same-channel notifies have the same
observable effect as one coalescing turns an N-node deploy
burst into a single ``_discover_nodes`` per channel instead of N.
Each handler runs under exception suppression so one buggy
consumer can't take down the dispatch thread.
"""
while not self._stopping.is_set():
try:
first = self._dispatch_queue.get(timeout=_LISTENER_POLL_TIMEOUT)
except queue.Empty:
continue
if first is None:
# Sentinel from :meth:`stop`.
return
# Coalesce by channel: keep the most recent payload per
# channel from this drain batch. Drops a stop sentinel
# silently — the next loop iteration will see _stopping set
# and exit anyway, so we don't need to re-queue the sentinel.
per_channel: dict[str, Notify] = {first.channel: first}
stop_seen = False
while True:
try:
nxt = self._dispatch_queue.get_nowait()
except queue.Empty:
break
if nxt is None:
stop_seen = True
continue
per_channel[nxt.channel] = nxt
for notify in per_channel.values():
with self._handlers_lock:
handlers = list(self._handlers.get(notify.channel, ()))
for handler in handlers:
t0 = time.monotonic()
try:
handler(notify)
except Exception:
log.exception(
"notify_dispatcher.handler_failed",
channel=notify.channel,
)
else:
elapsed_ms = (time.monotonic() - t0) * 1000.0
if elapsed_ms > 100.0:
log.debug(
"notify_dispatcher.handler_slow",
channel=notify.channel,
elapsed_ms=round(elapsed_ms, 1),
)
if stop_seen:
return
log.debug("notify_dispatcher.dispatch_exiting")
+6
View File
@@ -319,6 +319,12 @@ class TaskScheduler:
user_id=task.get("created_by", ""),
skill=task.get("skill", ""),
notify_targets=task.get("notify_targets", "[]"),
# Mark the resulting ChatSession as non-interactive-for-
# consent so OAuth-MCP errors get persisted to
# ``mcp_pending_consent`` for later dashboard surfacing,
# rather than relying on an in-flight SSE redirect the
# absent user can't complete.
client_type="scheduled",
)
ws_id = resp.ws_id
except Exception:
+384 -18
View File
@@ -41,6 +41,7 @@ from starlette.staticfiles import StaticFiles
from turnstone.api.console_spec import build_console_spec
from turnstone.api.docs import make_docs_handler, make_openapi_handler
from turnstone.console.collector import ClusterCollector
from turnstone.console.coordinator_alias import resolve_coordinator_alias
from turnstone.console.coordinator_client import load_task_envelope
from turnstone.console.metrics import ConsoleMetrics
from turnstone.console.router import ConsoleRouter
@@ -1651,6 +1652,27 @@ async def mcp_oauth_revoke_connection(request: Request) -> Response:
return await handle_mcp_oauth_revoke_connection(request)
async def mcp_oauth_list_pending(request: Request) -> Response:
"""GET /v1/api/mcp/oauth/pending — list deferred-consent records (Phase 9)."""
from turnstone.core.mcp_oauth import handle_mcp_oauth_list_pending
return await handle_mcp_oauth_list_pending(request)
async def mcp_oauth_clear_pending(request: Request) -> Response:
"""DELETE /v1/api/mcp/oauth/pending/{server_name} — dismiss a deferred-consent record."""
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_pending
return await handle_mcp_oauth_clear_pending(request)
async def mcp_oauth_clear_all_pending(request: Request) -> Response:
"""DELETE /v1/api/mcp/oauth/pending — bulk-dismiss deferred-consent records."""
from turnstone.core.mcp_oauth import handle_mcp_oauth_clear_all_pending
return await handle_mcp_oauth_clear_all_pending(request)
# ---------------------------------------------------------------------------
# Route handlers — available models (lightweight, no admin permission)
# ---------------------------------------------------------------------------
@@ -1671,20 +1693,55 @@ async def list_available_models(request: Request) -> JSONResponse:
# Include effective defaults for clients (web UI, channel gateway).
default_alias = ""
channel_default_alias = ""
coordinator_default_alias = ""
judge_default_alias = ""
cs = getattr(request.app.state, "config_store", None)
if cs is not None:
default_alias = cs.get("model.default_alias") or ""
channel_default_alias = cs.get("channels.default_model_alias") or ""
judge_default_alias = (cs.get("judge.model") or "").strip()
enabled_aliases = {r["alias"] for r in rows}
if default_alias and default_alias not in enabled_aliases:
default_alias = ""
if channel_default_alias and channel_default_alias not in enabled_aliases:
channel_default_alias = ""
# Coordinator default walks the standard three-tier chain (see
# :func:`turnstone.console.coordinator_alias.resolve_coordinator_alias`).
# The placeholder restricts every tier to enabled DB rows so the home
# composer doesn't advertise a model the workstream-creation picker
# can't actually offer — the session factory uses the same chain
# without that filter so explicit operator pins surface as 503 at
# ``registry.resolve`` instead of being silently swapped out.
coord_registry = getattr(request.app.state, "coord_registry", None)
if cs is not None and coord_registry is not None:
coordinator_default_alias = resolve_coordinator_alias(
explicit=cs.get("coordinator.model_alias"),
config_store=cs,
registry=coord_registry,
alias_filter=lambda a: a in enabled_aliases,
)
elif coord_registry is not None:
# ConfigStore failed lifespan but coord_registry is still bound —
# fall through to ``registry.default`` (filtered) so the home
# composer isn't blank. Mirrors the helper's tier 3 with the
# placeholder's enabled-rows filter applied.
registry_default = getattr(coord_registry, "default", "") or ""
if registry_default in enabled_aliases:
coordinator_default_alias = registry_default
# Judge falls back to the resolved coordinator alias when
# ``judge.model`` is empty *or* not a registered alias — judge.model
# is alias-only (matches IntentJudge.__init__), so an unknown value is
# operator misconfiguration that the judge itself silently inherits
# the session model on.
if not judge_default_alias or judge_default_alias not in enabled_aliases:
judge_default_alias = coordinator_default_alias
return JSONResponse(
{
"models": models,
"default_alias": default_alias,
"channel_default_alias": channel_default_alias,
"coordinator_default_alias": coordinator_default_alias,
"judge_default_alias": judge_default_alias,
}
)
@@ -2600,10 +2657,67 @@ async def proxy_shared_static(request: Request) -> Response:
return JSONResponse({"error": "Node unreachable"}, status_code=502)
# Auth endpoints the console handles locally instead of forwarding to
# the upstream node. Single source of truth for the dispatch table and
# the path set used by both the 405 short-circuit and the
# test parametrize, so a new entry can't drift between code and tests.
#
# Values are handler NAMES (strings) rather than function references.
# ``proxy_api`` resolves them via ``globals()`` at call time so test
# ``patch("turnstone.console.server.auth_login")`` is observed; a dict
# of refs would capture the original function at module load.
_PROXY_AUTH_LOCAL_HANDLERS: dict[tuple[str, str], str] = {
("POST", "auth/login"): "auth_login",
("POST", "auth/logout"): "auth_logout",
("POST", "auth/setup"): "auth_setup",
("POST", "auth/refresh"): "auth_refresh",
("GET", "auth/status"): "auth_status",
("GET", "auth/whoami"): "auth_whoami",
("GET", "auth/oidc/authorize"): "oidc_authorize",
("GET", "auth/oidc/callback"): "oidc_callback",
}
_PROXY_AUTH_LOCAL_PATHS: frozenset[str] = frozenset(path for _, path in _PROXY_AUTH_LOCAL_HANDLERS)
async def proxy_api(request: Request) -> Response:
"""Proxy API requests to target node. Detects SSE vs regular."""
"""Proxy API requests to target node, with two exceptions handled in-process:
1. ``auth/*`` endpoints in ``_PROXY_AUTH_LOCAL_HANDLERS`` are
dispatched to the console's own auth handlers so JWTs carry
``JWT_AUD_CONSOLE`` and Set-Cookie lands on the console origin.
Forwarding upstream would mint ``JWT_AUD_SERVER`` tokens that the
console's ``AuthMiddleware`` rejects on the next proxied call,
locking the user out of the proxied UI and ``_proxy_post``
drops Set-Cookie when forwarding anyway. ``refresh`` and
``whoami`` are intentionally NOT in ``PUBLIC_PATHS`` (caller must
still hold a valid cookie); local dispatch is about cookie-origin
and audience, not public access.
2. SSE endpoints (per-ws + global events) stream via ``_proxy_sse``.
Everything else is forwarded to ``server_url`` via ``_proxy_post`` /
``_proxy_get``.
"""
node_id = request.path_params["node_id"]
path = request.path_params["path"]
handler_name = _PROXY_AUTH_LOCAL_HANDLERS.get((request.method, path))
if handler_name is not None:
# Resolve via ``globals()`` so ``patch("...auth_login")`` in
# tests is observed. A direct function-ref dict would have
# captured the original at module load.
handler = globals()[handler_name]
return await handler(request) # type: ignore[no-any-return]
# Path matches a local-dispatch auth endpoint but the method does not:
# short-circuit with 405 so the request can't fall through to
# ``_proxy_post`` / ``_proxy_get`` and reach the upstream
# authenticated as the console's service token
# (``_proxy_auth_headers`` falls back to the service identity when
# there's no user context). Harmless today because every upstream
# auth route 405s on the wrong method too, but kept tight so a
# future upstream patch can't widen the surface by accident.
if path in _PROXY_AUTH_LOCAL_PATHS:
return JSONResponse({"error": "Method not allowed"}, status_code=405)
server_url = _get_server_url(request, node_id)
if not server_url:
return JSONResponse({"error": "Node not found"}, status_code=404)
@@ -4104,6 +4218,7 @@ def _coord_idle_cleanup_thread(
mgr: SessionManager,
timeout_sec: float,
stop_event: threading.Event | None = None,
min_sweep_interval: float = 5.0,
) -> None:
"""Periodically reap idle + DB-orphan coordinator workstreams.
@@ -4114,7 +4229,7 @@ def _coord_idle_cleanup_thread(
and which aren't currently loaded. The latter pass catches coords left
behind by prior console process incarnations.
Runs an initial sweep BEFORE the first sleep so cold-start orphans are
Runs an initial sweep BEFORE the first wait so cold-start orphans are
reaped immediately rather than waiting one ``check_every`` interval (~30
min on default 2h timeout). This intentionally diverges from the regular
server pattern, which has no initial sweep the regular server runs
@@ -4122,26 +4237,90 @@ def _coord_idle_cleanup_thread(
is a small fixed-size cache where orphans dominate the row count after
a cold boot.
Wait shape: subscribes a callback to ``mgr._state_subscribers`` that
sets a ``tick_now`` event; the loop blocks on ``tick_now.wait(check_every)``
so any workstream state-change wakes the sweeper without waiting a
full check interval, AND the timeout still fires the periodic sweep
even when no activity happens (catching the DB-orphan-only case).
Net: blocked most of the time instead of repeating storage scans.
``min_sweep_interval`` is the hard floor between successive
``close_idle`` calls (default 5 s) without it, sustained
state-change activity (each turn typically fires
thinking/running/attention/idle on the coord SessionManager) would
cause every ``tick_now.set`` mid-sweep to leave the next ``wait``
returning immediately, and the loop would tight-spin ``close_idle``
at the rate of its own DB latency (~20-50 calls/sec). The floor
bounds DB-call traffic at ``1 / min_sweep_interval`` per second
under any external activity while still letting a quiet system
fire on every state-change wake-up. Tests inject 0.0 to keep the
suite fast.
Default 5 s is a 6x improvement on the pre-refactor fixed 30 s
cadence while bounding DB-call traffic at ~0.2 calls/sec under
sustained activity an order of magnitude below ``close_idle``'s
DB-latency budget, but tight enough that idle-row reaping still
feels prompt to a human watching the sidebar. Tunable post-merge
if profiling shows close_idle latency dominates the cadence.
``stop_event`` is for tests when set, the thread exits cleanly after
the next loop check. Production callers pass ``None`` (the daemon is
process-lifetime).
"""
check_every = min(300.0, timeout_sec / 4)
# Initial sweep — runs once before entering the sleep loop.
tick_now = threading.Event()
def _on_state_change(_ws_id: str, _state: Any) -> None:
# Any workstream state-change resets the idle clock for that
# ws AND may make a different ws newly-eligible (close-idle
# pass 2 evaluates DB rows by timestamp). Cheap signal, full
# re-evaluation deferred to the next loop iteration.
tick_now.set()
mgr.subscribe_to_state(_on_state_change)
try:
mgr.close_idle(timeout_sec)
except Exception:
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
while True:
if stop_event is not None and stop_event.is_set():
return
time.sleep(check_every)
if stop_event is not None and stop_event.is_set():
return
# Initial sweep — runs once before entering the wait loop.
# ``tick_now`` is intentionally not cleared here: any
# state-change event that arrives between subscribe and the
# first ``wait`` should fire close_idle immediately, not be
# discarded.
try:
mgr.close_idle(timeout_sec)
except Exception:
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
log.debug("console.coord_idle_cleanup_initial_failed", exc_info=True)
last_sweep_at = time.monotonic()
while True:
if stop_event is not None and stop_event.is_set():
return
tick_now.wait(check_every)
if stop_event is not None and stop_event.is_set():
return
# Clear BEFORE the cadence floor so any state-change event
# arriving during the cooldown (or during the close_idle
# below) leaves ``tick_now`` set — the next loop iteration
# then re-enters ``wait`` already-set and re-evaluates
# promptly. close_idle is idempotent so a spurious extra
# tick is just one redundant scan.
tick_now.clear()
# Cadence floor — see docstring for the tight-spin
# hazard rationale. Cooldown uses ``stop_event.wait``
# (not ``time.sleep``) so the test stop hook still
# terminates promptly during the cooldown window.
since_last = time.monotonic() - last_sweep_at
if since_last < min_sweep_interval:
gap = min_sweep_interval - since_last
if stop_event is not None:
if stop_event.wait(gap):
return
else:
time.sleep(gap)
try:
mgr.close_idle(timeout_sec)
except Exception:
log.debug("console.coord_idle_cleanup_failed", exc_info=True)
last_sweep_at = time.monotonic()
finally:
mgr.unsubscribe_from_state(_on_state_change)
# Guards concurrent attempts to bootstrap the coord subsystem from the
@@ -4203,12 +4382,22 @@ def _bootstrap_coord_subsystem(
def _token_factory() -> str:
return tm.token
# ``coord_adapter`` is bound later in this same
# ``_bootstrap_coord_subsystem`` call, after the adapter and
# manager are constructed but before any session is created
# — so this factory is *defined* before the adapter exists but
# only ever *called* after it does. The free-variable lookup
# at call time resolves to the adapter built in this same
# bootstrap pass, giving the client a handle to the in-process
# wakeup bus the dispatch sink notifies on every child
# state-change event.
return CoordinatorClient(
console_base_url=console_bind_url,
storage=storage,
token_factory=_token_factory,
coord_ws_id=ws_id,
user_id=user_id,
child_event_bus=coord_adapter.child_event_bus,
)
# Pre-compute config-derived integers BEFORE any thread starts so a
@@ -4734,6 +4923,12 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
await close_oidc_state(app.state)
app.state.collector.stop()
# Stop the dispatcher after the collector — collector.stop() drops its
# subscription, so the dispatcher's dispatch thread won't fire into a
# half-torn-down collector during shutdown.
notify_dispatcher = getattr(app.state, "notify_dispatcher", None)
if notify_dispatcher is not None:
notify_dispatcher.stop()
audit_exec_shutdown = getattr(app.state, "audit_executor", None)
if audit_exec_shutdown is not None:
_set_audit_executor(None)
@@ -7884,6 +8079,7 @@ _MODEL_AFFECTING_SETTING_KEYS: frozenset[str] = frozenset(
"coordinator.model_alias",
"coordinator.reasoning_effort",
"judge.model",
"channels.default_model_alias",
}
)
@@ -8643,10 +8839,19 @@ def _mask_mcp_secrets(server: dict[str, Any], reveal: bool = False) -> dict[str,
def _mcp_server_to_detail(
server: dict[str, Any],
node_statuses: dict[str, dict[str, Any]] | None = None,
consented_users_count: int | None = None,
) -> dict[str, Any]:
"""Convert a storage dict to a McpServerDetail-shaped dict."""
"""Convert a storage dict to a McpServerDetail-shaped dict.
*consented_users_count* is the Phase 9 admin pill data distinct
non-expired tokens issued for this ``(server_name)``. Omitted
(``None``) when the row's ``auth_type`` is not ``oauth_user``, so
static / none rows don't carry an irrelevant ``0``.
"""
d = dict(server)
d["status"] = node_statuses or {}
if consented_users_count is not None:
d["consented_users_count"] = consented_users_count
return d
@@ -8697,8 +8902,31 @@ async def admin_list_mcp_servers(request: Request) -> JSONResponse:
reveal = str(request.query_params.get("reveal", "")).lower() in ("true", "1")
servers = storage.list_mcp_servers()
# Collect live status from all nodes
node_statuses = await _collect_mcp_status(request)
# Phase 9: bulk-aggregate consented-users-count across all oauth_user
# rows in a single GROUP BY query (rather than N per-row sync DB
# round-trips inside this async handler). Run in parallel with the
# cross-node HTTP status fan-out below — neither has a data
# dependency on the other, so awaiting them sequentially would
# stack the DB latency on top of the fan-out latency. Skipped
# entirely when no row is oauth_user so static-only installs
# exercise zero new storage queries.
has_oauth_user = any(s.get("auth_type") == "oauth_user" for s in servers)
status_task: asyncio.Task[dict[str, dict[str, dict[str, Any]]]] = asyncio.create_task(
_collect_mcp_status(request)
)
count_task: asyncio.Task[dict[str, int]] | None = (
asyncio.create_task(asyncio.to_thread(storage.count_mcp_consented_users_grouped_by_server))
if has_oauth_user
else None
)
node_statuses = await status_task
consent_counts: dict[str, int] = {}
if count_task is not None:
try:
consent_counts = await count_task
except Exception:
log.debug("admin.mcp_consented_users_bulk_count_failed", exc_info=True)
db_names: set[str] = set()
result = []
@@ -8710,8 +8938,14 @@ async def admin_list_mcp_servers(request: Request) -> JSONResponse:
status = node_servers.get(s["name"])
if status:
per_node[node_id] = status
# Phase 9: surface the consented-users-count pill for
# oauth_user rows. Aggregate was pre-computed above with a
# single bulk GROUP BY query; we just look up here.
consent_count: int | None = None
if s.get("auth_type") == "oauth_user":
consent_count = consent_counts.get(s["name"], 0)
s = _mask_mcp_secrets(s, reveal)
result.append(_mcp_server_to_detail(s, per_node))
result.append(_mcp_server_to_detail(s, per_node, consent_count))
# Merge config-sourced servers visible on nodes but not in DB
config_names: set[str] = set()
@@ -9352,6 +9586,92 @@ async def admin_mcp_reconnect_one(request: Request) -> JSONResponse:
return await _admin_mcp_action(request, "reconnect")
async def admin_mcp_bulk_revoke(request: Request) -> JSONResponse:
"""POST /v1/api/admin/mcp-servers/{name}/bulk-revoke — clear every user's token (Phase 9).
Admin-side counterpart to the per-user
``DELETE /v1/api/mcp/oauth/connections/{server_name}`` revoke that
shipped in Phase 8. Used to drop orphaned tokens after an
``auth_type`` transition (oauth_user static) or after rotating
the configured OAuth client.
Authoritative local delete via
:meth:`StorageBackend.delete_mcp_oauth_rows_by_server_name`
purges both ``mcp_user_tokens`` and ``mcp_oauth_pending`` rows for
the named server. Upstream RFC 7009 revoke is intentionally NOT
attempted in bulk (would require per-row decrypt + N upstream HTTP
calls); operators who need upstream cleanup should use the per-
user revoke endpoint or let tokens expire naturally. The audit
detail records ``upstream_revoke_outcome="bulk_admin_no_upstream"``
so the deferral is visible.
Pool eviction is NOT performed by this handler. Per-user revoke
has a per-(user, server) eviction primitive
(``MCPClientManager.evict_user_session``); bulk-revoke would need a
per-server iteration over consented users that no current primitive
supports. Stale in-flight sessions surface as a per-user 401 on
the next dispatch, which refreshes through the now-empty token row
and emits ``mcp_consent_required`` the documented v1 fallback.
See :func:`turnstone.core.mcp_client._dispatch_pool` retry path.
"""
from turnstone.core.audit import record_audit
from turnstone.core.auth import require_permission
from turnstone.core.web_helpers import require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
err = require_permission(request, "admin.mcp")
if err:
return err
name = request.path_params.get("name", "").strip()
if not name or "__" in name:
return JSONResponse({"error": "invalid server name"}, status_code=400)
existing = storage.get_mcp_server_by_name(name)
if existing is None:
return JSONResponse({"error": "No such server"}, status_code=404)
if existing.get("auth_type") != "oauth_user":
return JSONResponse(
{"error": "bulk-revoke is only valid for auth_type=oauth_user servers"},
status_code=400,
)
target_id = existing.get("server_id", name)
consented_before = 0
try:
consented_before = storage.count_mcp_consented_users_by_server(name)
except Exception:
log.debug("admin.mcp_bulk_revoke_pre_count_failed server=%s", name, exc_info=True)
deleted = storage.delete_mcp_oauth_rows_by_server_name(name)
audit_uid, ip = _audit_context(request)
record_audit(
storage,
audit_uid,
"mcp_server.oauth.bulk_revoked",
"mcp_server",
target_id,
{
"name": name,
"rows_deleted": deleted,
"consented_users_before": consented_before,
"upstream_revoke_outcome": "bulk_admin_no_upstream",
},
ip,
)
return JSONResponse(
{
"status": "ok",
"rows_deleted": deleted,
"consented_users_before": consented_before,
}
)
async def admin_import_mcp_config(request: Request) -> JSONResponse:
"""POST /v1/api/admin/mcp-servers/import — import from pasted JSON config."""
import uuid
@@ -9904,6 +10224,9 @@ async def admin_create_model_definition(request: Request) -> JSONResponse:
if not reasoning_effort:
reasoning_effort = None
surface_persisted_reasoning = bool(body.get("surface_persisted_reasoning", True))
replay_reasoning_to_model = bool(body.get("replay_reasoning_to_model", False))
storage.create_model_definition(
definition_id=definition_id,
alias=alias,
@@ -9918,6 +10241,8 @@ async def admin_create_model_definition(request: Request) -> JSONResponse:
temperature=temperature,
max_tokens=max_tokens,
reasoning_effort=reasoning_effort,
surface_persisted_reasoning=surface_persisted_reasoning,
replay_reasoning_to_model=replay_reasoning_to_model,
)
record_audit(
@@ -10079,6 +10404,10 @@ async def admin_update_model_definition(request: Request) -> JSONResponse:
)
else:
updates["reasoning_effort"] = re_val
if "surface_persisted_reasoning" in body:
updates["surface_persisted_reasoning"] = bool(body["surface_persisted_reasoning"])
if "replay_reasoning_to_model" in body:
updates["replay_reasoning_to_model"] = bool(body["replay_reasoning_to_model"])
if updates:
storage.update_model_definition(definition_id, **updates)
@@ -11836,6 +12165,7 @@ def create_app(
console_url: str = "",
router: ConsoleRouter | None = None,
console_metrics: ConsoleMetrics | None = None,
notify_dispatcher: Any = None,
) -> Starlette:
"""Build the Starlette ASGI application for the console dashboard."""
_spec = build_console_spec()
@@ -12062,6 +12392,17 @@ def create_app(
mcp_oauth_revoke_connection,
methods=["DELETE"],
),
Route("/api/mcp/oauth/pending", mcp_oauth_list_pending),
Route(
"/api/mcp/oauth/pending",
mcp_oauth_clear_all_pending,
methods=["DELETE"],
),
Route(
"/api/mcp/oauth/pending/{server_name}",
mcp_oauth_clear_pending,
methods=["DELETE"],
),
Route("/api/admin/users", admin_list_users),
Route("/api/admin/users", admin_create_user, methods=["POST"]),
Route("/api/admin/users/{user_id}", admin_delete_user, methods=["DELETE"]),
@@ -12247,6 +12588,11 @@ def create_app(
admin_mcp_reconnect_one,
methods=["POST"],
),
Route(
"/api/admin/mcp-servers/{name}/bulk-revoke",
admin_mcp_bulk_revoke,
methods=["POST"],
),
Route(
"/api/admin/mcp-servers/{server_id}",
admin_get_mcp_server,
@@ -12462,6 +12808,7 @@ def create_app(
lifespan=_lifespan,
)
app.state.collector = collector
app.state.notify_dispatcher = notify_dispatcher
app.state.jwt_secret = jwt_secret
app.state.auth_storage = auth_storage
app.state.proxy_token_mgr = proxy_token_mgr
@@ -12557,7 +12904,7 @@ def main() -> None:
from turnstone.core.config import add_config_arg, apply_config
add_config_arg(parser)
apply_config(parser, ["console", "auth"])
apply_config(parser, ["console", "auth", "database"])
args = parser.parse_args()
from turnstone.core.log import configure_logging_from_args
@@ -12576,6 +12923,13 @@ def main() -> None:
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
db_url = os.environ.get("TURNSTONE_DB_URL", "")
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
# Optional dedicated LISTEN URL — config.toml ``[database] listen_url``
# (lifted onto args by ``apply_config``) wins over env, and an empty
# value falls through to the main DB URL inside the storage layer.
# Only used by the ``NotifyDispatcher``; ignored on SQLite.
db_listen_url = getattr(args, "db_listen_url", None) or os.environ.get(
"TURNSTONE_DB_LISTEN_URL", ""
)
auth_storage = init_storage(
db_backend,
path=db_path,
@@ -12584,6 +12938,7 @@ def main() -> None:
sslrootcert=os.environ.get("TURNSTONE_DB_SSLROOTCERT", ""),
sslcert=os.environ.get("TURNSTONE_DB_SSLCERT", ""),
sslkey=os.environ.get("TURNSTONE_DB_SSLKEY", ""),
listen_url=db_listen_url,
)
except Exception:
log.info("Console storage not available — admin API disabled, JWT-only auth")
@@ -12613,11 +12968,21 @@ def main() -> None:
router = ConsoleRouter(storage=auth_storage)
console_metrics = ConsoleMetrics()
# NotifyDispatcher multiplexes the dedicated LISTEN connection for all
# console-side consumers. Currently one channel: ``services`` for
# reactive node discovery. Followup PRs (ConfigStore live reload,
# scheduler immediate dispatch) add additional channels here.
from turnstone.console.notify_dispatcher import NotifyDispatcher
notify_dispatcher = NotifyDispatcher(auth_storage, channels=["services"])
notify_dispatcher.start()
collector = ClusterCollector(
storage=auth_storage,
token_manager=collector_token_mgr,
router=router,
console_metrics=console_metrics,
notify_dispatcher=notify_dispatcher,
)
collector.start()
@@ -12709,6 +13074,7 @@ def main() -> None:
console_url=console_url,
router=router,
console_metrics=console_metrics,
notify_dispatcher=notify_dispatcher,
)
log.info("Console starting on %s", console_url)
+14 -12
View File
@@ -21,6 +21,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING
from turnstone.console.coordinator_alias import resolve_coordinator_alias
from turnstone.core.log import get_logger
from turnstone.core.session import ChatSession
from turnstone.core.workstream import WorkstreamKind
@@ -96,18 +97,19 @@ def build_console_session_factory(
f"console session factory only supports kind=COORDINATOR, got {kind!r}"
)
# Resolve coordinator.model_alias from settings if caller didn't
# override. Unset ``coordinator.model_alias`` falls back to the
# model registry's default alias — operators get a working
# coordinator on a freshly-provisioned console without an extra
# manual setting. Resolve to the CONCRETE alias name
# (``registry.default``) rather than passing None downstream:
# ``ChatSession.__init__`` reads ``registry.get_provider(alias)``
# to pick the right provider class, and passing None makes it
# fall through to a generic OpenAI-compat provider — which
# mismatches when the default is Anthropic/Google-backed.
explicit_alias = model_alias or (config_store.get("coordinator.model_alias") or "").strip()
effective_alias = explicit_alias or registry.default
# Resolve to the CONCRETE alias name rather than passing None
# downstream: ``ChatSession.__init__`` reads
# ``registry.get_provider(alias)`` to pick the right provider
# class, and passing None makes it fall through to a generic
# OpenAI-compat provider — which mismatches when the default is
# Anthropic/Google-backed. See
# :func:`turnstone.console.coordinator_alias.resolve_coordinator_alias`
# for the three-tier chain shared with the placeholder API.
effective_alias = resolve_coordinator_alias(
explicit=model_alias,
config_store=config_store,
registry=registry,
)
r_client, r_model, r_cfg = registry.resolve(effective_alias)
+224 -4
View File
@@ -1142,12 +1142,54 @@ function _populateScheduleSelect(selectId, url, labelKey, valueKey, opts) {
sel.appendChild(opt);
});
if (opts && opts.selected) sel.value = opts.selected;
// Caller hook for placeholder annotation / other post-load tweaks.
// Used by the schedule modals to rewrite the bare "Default model"
// placeholder with the resolved alias so the label matches the
// home composer (see app.js _populateHomeModelDropdowns).
if (opts && typeof opts.afterPopulate === "function") {
try {
opts.afterPopulate(sel, data, items);
} catch (_e) {
/* hook errors must not break the dropdown */
}
}
})
.catch(function () {
/* dropdown stays with placeholder or temporary option */
});
}
// Update the schedule-model placeholder option (first <option>) to
// "Default — alias (model)" using /v1/api/models's resolved
// default_alias, mirroring the home composer. Schedules don't carry
// a coordinator/judge split, so they consume the workstream-creation
// default rather than coordinator_default_alias / judge_default_alias.
// Em-dash separator (rather than nested parens) keeps the alias's
// "(model)" suffix legible.
function _decorateScheduleModelPlaceholder(sel, data) {
if (!sel || sel.options.length === 0) return;
var alias = (data && data.default_alias) || "";
if (!alias) return;
var match = null;
var models = (data && data.models) || [];
for (var i = 0; i < models.length; i++) {
if (models[i].alias === alias) {
match = models[i];
break;
}
}
var label;
if (match) {
label =
match.alias === match.model
? match.alias
: match.alias + " (" + match.model + ")";
} else {
label = alias;
}
sel.options[0].textContent = "Default — " + label;
}
// Channel platforms shown in admin notify-target rows. Mirror server-side
// channel adapters; expand here when a new adapter ships (Discord / Slack
// today, MS Teams / etc. later).
@@ -1333,6 +1375,7 @@ function showCreateScheduleModal() {
display: function (m) {
return m.alias === m.model ? m.alias : m.alias + " (" + m.model + ")";
},
afterPopulate: _decorateScheduleModelPlaceholder,
});
// Populate skill dropdown
_populateScheduleSelect(
@@ -1487,6 +1530,7 @@ function showEditScheduleModal(taskId) {
display: function (m) {
return m.alias === m.model ? m.alias : m.alias + " (" + m.model + ")";
},
afterPopulate: _decorateScheduleModelPlaceholder,
});
// Populate skill dropdown with current value pre-selected
_populateScheduleSelect(
@@ -2637,7 +2681,7 @@ function loadSettings() {
// Merge values + schema. Skip role-assignment settings owned by
// the Models → Roles sub-tab (judge.* settings still live on the
// Judge tab; the four model-tab roles render only there).
// Judge tab; the model-tab roles render only there).
var merged = {};
var roleKeys = {
"coordinator.model_alias": 1,
@@ -2646,6 +2690,7 @@ function loadSettings() {
"model.plan_effort": 1,
"model.task_alias": 1,
"model.task_effort": 1,
"channels.default_model_alias": 1,
};
for (var j = 0; j < valuesArr.length; j++) {
var v = valuesArr[j];
@@ -3269,6 +3314,11 @@ function _renderMcpServers(items) {
var totalTools = 0,
totalRes = 0,
totalPrompts = 0;
// Phase 9: aggregate the most-recent refresh entry across nodes
// so the admin pill reflects "the freshest known state" rather
// than picking an arbitrary node.
var newestRefreshAt = null;
var newestRefreshOutcome = null;
for (var j = 0; j < nodeIds.length; j++) {
var ns = statusEntries[nodeIds[j]];
if (ns.connected) {
@@ -3281,6 +3331,13 @@ function _renderMcpServers(items) {
anyError = true;
if (!firstError) firstError = ns.error;
}
if (
typeof ns.last_refresh_at === "number" &&
(newestRefreshAt === null || ns.last_refresh_at > newestRefreshAt)
) {
newestRefreshAt = ns.last_refresh_at;
newestRefreshOutcome = ns.last_refresh_outcome || null;
}
}
var dotClass = "mcp-status-dot disabled";
@@ -3306,6 +3363,42 @@ function _renderMcpServers(items) {
statusText = "idle";
}
// Phase 9: refresh pill shows the short-relative age (e.g. "12m") with
// outcome-tinted color (ok vs err) and the full ISO timestamp + outcome
// in the tooltip. Pill is omitted (and the cell stays unchanged from
// its pre-Phase-9 shape) when no node has yet recorded a refresh
// outcome for this server.
var refreshPill = "";
if (newestRefreshAt !== null) {
var ageSeconds = Math.max(
0,
Math.floor(Date.now() / 1000 - newestRefreshAt),
);
var ageShort;
if (ageSeconds < 60) ageShort = ageSeconds + "s";
else if (ageSeconds < 3600) ageShort = Math.floor(ageSeconds / 60) + "m";
else if (ageSeconds < 86400)
ageShort = Math.floor(ageSeconds / 3600) + "h";
else ageShort = Math.floor(ageSeconds / 86400) + "d";
var outcomeText = newestRefreshOutcome || "unknown";
var pillCls =
outcomeText === "ok" ? "mcp-refresh-pill-ok" : "mcp-refresh-pill-err";
var pillTitle =
"Last refresh " +
new Date(newestRefreshAt * 1000).toISOString() +
" (" +
outcomeText +
")";
refreshPill =
' <span class="mcp-refresh-pill ' +
pillCls +
'" title="' +
escapeHtml(pillTitle) +
'">' +
escapeHtml(ageShort) +
"</span>";
}
var transportCls =
s.transport === "stdio" ? "mcp-transport-stdio" : "mcp-transport-http";
var toolsVal = anyConnected
@@ -3340,6 +3433,24 @@ function _renderMcpServers(items) {
'<button class="admin-btn-action" data-mcp-oauth-connect="' +
escapeHtml(s.name) +
'">connect</button>';
// Phase 9: surface the consented-users count + bulk-revoke
// affordance only when at least one user has consented.
var consentCount =
typeof s.consented_users_count === "number"
? s.consented_users_count
: 0;
if (consentCount > 0) {
actionBtns +=
'<button class="admin-btn-danger" data-mcp-bulk-revoke="' +
escapeHtml(s.name) +
'" data-mcp-consent-count="' +
consentCount +
'" title="Drop all ' +
consentCount +
' user consents for this server">bulk-revoke (' +
consentCount +
")</button>";
}
}
var actions = isConfig
? actionBtns
@@ -3384,6 +3495,7 @@ function _renderMcpServers(items) {
dotClass +
'" aria-hidden="true"></span>' +
escapeHtml(statusText) +
refreshPill +
"</span>" +
'<span class="admin-col admin-col-mactions">' +
actions +
@@ -3462,6 +3574,42 @@ function _renderMcpServers(items) {
window.open(url, "_blank", "noopener");
});
});
el.querySelectorAll("[data-mcp-bulk-revoke]").forEach(function (btn) {
btn.addEventListener("click", function () {
var name = this.getAttribute("data-mcp-bulk-revoke");
var count = this.getAttribute("data-mcp-consent-count") || "?";
showConfirmModal(
"Bulk-revoke MCP consents",
"Drop all " +
count +
' user consents for server "' +
name +
'"? Users will need to re-consent on next use. Upstream revoke is not attempted in bulk; tokens at the authorization server will expire naturally.',
"Bulk-revoke",
function () {
authFetch(
"/v1/api/admin/mcp-servers/" +
encodeURIComponent(name) +
"/bulk-revoke",
{ method: "POST" },
)
.then(function (r) {
if (!r.ok) throw new Error();
return r.json();
})
.then(function (j) {
showToast(
"Bulk-revoked " + (j.rows_deleted || 0) + " row(s) for " + name,
);
loadAdminMcp();
})
.catch(function () {
showToast("Failed to bulk-revoke " + name);
});
},
);
});
});
el.querySelectorAll("[data-mcp-delete]").forEach(function (btn) {
btn.addEventListener("click", function () {
var sid = this.getAttribute("data-mcp-delete");
@@ -4639,6 +4787,14 @@ var _modelCreateTrigger = null;
// setting render a second selector inline. Adding a new role (e.g.
// ``perception.audio.model``) is purely additive: drop a row here once
// the SettingDef lands in turnstone/core/settings_registry.py.
// ``fallbackKind`` controls how the empty/blank option in the alias
// dropdown is labelled. Coordinator and Judge fall back to a single
// well-defined alias (model.default_alias / coordinator alias) so we
// surface that concrete model in the placeholder. Plan/Task agents
// cascade through ``[model].plan_model → [model].agent_model →
// session model`` per turnstone/core/settings_registry.py — there's
// no single "default" to advertise, so the blank reads "(inherit)"
// to match the vocabulary of the reasoning-effort dropdowns.
var MODEL_ROLES = [
{
label: "Coordinator",
@@ -4646,12 +4802,14 @@ var MODEL_ROLES = [
"Console-hosted coordinator sessions that drive child workstreams.",
aliasKey: "coordinator.model_alias",
effortKey: "coordinator.reasoning_effort",
fallbackKind: "default",
},
{
label: "Judge",
description:
"Intent-validation judge that scores tool calls before approval.",
aliasKey: "judge.model",
fallbackKind: "default",
},
{
label: "Plan agent",
@@ -4659,6 +4817,7 @@ var MODEL_ROLES = [
"plan_agent sub-agent — produces high-level plans before task dispatch.",
aliasKey: "model.plan_alias",
effortKey: "model.plan_effort",
fallbackKind: "inherit",
},
{
label: "Task agent",
@@ -4666,6 +4825,14 @@ var MODEL_ROLES = [
"task_agent sub-agent — runs autonomous subtasks dispatched by the parent.",
aliasKey: "model.task_alias",
effortKey: "model.task_effort",
fallbackKind: "inherit",
},
{
label: "Channel adapter",
description:
"Workstreams created by channel adapters (Discord, Slack) when no model is specified at creation time.",
aliasKey: "channels.default_model_alias",
fallbackKind: "default",
},
];
@@ -4848,11 +5015,40 @@ function _renderModelRoles(container, values, schema) {
"aria-label",
role.label + " model (empty = default)",
);
// Format the blank/inherit option in the same "alias (model)" shape
// as the other rows so the dropdown reads consistently — without
// this the empty row was bare "(default — flatspark)" while every
// other row carried a "(/models/...)" suffix. Plan/Task agent
// fall back through a multi-step chain (config.toml → agent_model
// → session) that has no single concrete "default", so they get
// a plain "(inherit)" instead of the misleading
// "(default — <coordinator-alias>)".
var blank = document.createElement("option");
blank.value = "";
blank.textContent = _modelDefaultAlias
? "(default — " + _modelDefaultAlias + ")"
: "(default)";
if (role.fallbackKind === "inherit") {
blank.textContent = "(inherit)";
} else {
var defaultDef = null;
if (_modelDefaultAlias) {
for (var dm = 0; dm < enabledAliases.length; dm++) {
if (enabledAliases[dm].alias === _modelDefaultAlias) {
defaultDef = enabledAliases[dm];
break;
}
}
}
if (defaultDef) {
var defLabel =
defaultDef.alias === defaultDef.model
? defaultDef.alias
: defaultDef.alias + " (" + defaultDef.model + ")";
blank.textContent = "(default — " + defLabel + ")";
} else if (_modelDefaultAlias) {
blank.textContent = "(default — " + _modelDefaultAlias + ")";
} else {
blank.textContent = "(default)";
}
}
aliasSel.appendChild(blank);
var currentAlias = aliasInfo.value || "";
var matched = false;
@@ -5008,6 +5204,11 @@ function _renderModels(items) {
if (m.max_tokens != null) overrides.push("max_tok=" + m.max_tokens);
if (m.reasoning_effort != null)
overrides.push("effort=" + m.reasoning_effort);
// Reasoning persistence flags surface only when non-default
// (persist=False is the operator opt-out; replay=True is the
// operator opt-in). Default values are silent.
if (m.surface_persisted_reasoning === false) overrides.push("surface=off");
if (m.replay_reasoning_to_model === true) overrides.push("replay=on");
if (overrides.length) {
var ovrSpan = document.createElement("span");
ovrSpan.className = "model-overrides-hint";
@@ -5196,6 +5397,8 @@ function showCreateModelModal() {
el.style.borderColor = "";
});
document.getElementById("model-enabled").checked = true;
document.getElementById("model-surface-persisted-reasoning").checked = true;
document.getElementById("model-replay-reasoning").checked = false;
document.getElementById("model-detect-result").style.display = "none";
document.getElementById("model-detect-btn").disabled = false;
document.getElementById("model-detect-btn").textContent = "Detect";
@@ -5278,6 +5481,13 @@ function showEditModelModal(definitionId) {
document.getElementById("model-capabilities").value =
capsText === "{}" ? "" : capsText;
document.getElementById("model-enabled").checked = m.enabled !== false;
// Reasoning persistence flags — defaults match the dataclass
// defaults (persist=true, replay=false) when the API returns
// them as undefined (legacy / pre-052 row).
document.getElementById("model-surface-persisted-reasoning").checked =
m.surface_persisted_reasoning !== false;
document.getElementById("model-replay-reasoning").checked =
m.replay_reasoning_to_model === true;
_applyProviderDefaults();
})
.catch(function () {
@@ -5419,6 +5629,16 @@ function submitCreateModel() {
form.reasoning_effort = null;
}
// Reasoning persistence flags — always serialize so a flip from
// default takes effect on PUT (the server's update path keys off
// "field present in body").
form.surface_persisted_reasoning = document.getElementById(
"model-surface-persisted-reasoning",
).checked;
form.replay_reasoning_to_model = document.getElementById(
"model-replay-reasoning",
).checked;
var apiKey = document.getElementById("model-api-key").value;
if (apiKey) form.api_key = apiKey;
+44 -5
View File
@@ -1761,11 +1761,11 @@ function _mountHomeCoordComposer() {
id: "judge_model",
label: "Judge Model",
type: "select",
// Neutral label — the actual default is ConfigStore
// ``judge.model`` when set, IntentJudge's agent-model
// fallback when not. "Default judge model" doesn't
// mislead either way.
choices: [{ value: "", text: "Default judge model" }],
// Initial placeholder; _populateHomeModelDropdowns rewrites this
// to "Default model (<alias>)" once /v1/api/models reports the
// resolved judge alias (judge.model when set, otherwise the
// session model — see IntentJudge.__init__).
choices: [{ value: "", text: "Default model" }],
},
],
},
@@ -1801,6 +1801,21 @@ function _populateHomeSkillDropdown() {
});
}
// Format a resolved alias with its model suffix the same way as the
// dropdown rows ("alias (model)", or just "alias" when they coincide).
// Returns "" when alias is empty or unknown so callers can fall back
// to a neutral placeholder.
function _resolveModelLabel(alias, models) {
if (!alias) return "";
for (var i = 0; i < (models || []).length; i++) {
var m = models[i];
if (m.alias === alias) {
return m.alias === m.model ? m.alias : m.alias + " (" + m.model + ")";
}
}
return "";
}
// Populate Model + Judge Model dropdowns from /v1/api/models — same
// list the interactive new-ws modal uses. Empty/default option stays
// at the top so submitting without a choice falls back to the
@@ -1819,6 +1834,30 @@ function _populateHomeModelDropdowns() {
});
_homeCoordComposer.setOptionChoices("model", choices);
_homeCoordComposer.setOptionChoices("judge_model", choices);
// Both placeholders use the same "Default — alias (model)"
// template — the field-row labels (MODEL / JUDGE MODEL) already
// carry the role context, so an asymmetric "Default judge model"
// reads awkwardly alongside the plain "Default model" line above
// it. Em-dash separator (rather than nested parens) keeps the
// alias's "(model)" suffix legible and matches the
// ``(default — alias (model))`` pattern used in the admin Roles
// tab.
var coordDefault = _resolveModelLabel(
data.coordinator_default_alias || "",
data.models || [],
);
var judgeDefault = _resolveModelLabel(
data.judge_default_alias || "",
data.models || [],
);
_homeCoordComposer.setOptionPlaceholder(
"model",
coordDefault ? "Default — " + coordDefault : "Default model",
);
_homeCoordComposer.setOptionPlaceholder(
"judge_model",
judgeDefault ? "Default — " + judgeDefault : "Default model",
);
})
.catch(function () {
/* defaults still work even without the dropdown populated */
@@ -579,6 +579,13 @@
if (callId && toolRows.has(callId)) {
const entry = toolRows.get(callId);
_appendResultToRow(entry.row, output, isError, opts);
// The batch may have been --running (live tool_info auto path,
// approval_resolved approved path, or replay-time orphan).
// Drop --running once every row in the batch has a result so
// the kicker text + visual style flip back to the post-execution
// state. Per-row check (not a counter) keeps the logic
// resilient to out-of-order replay + late SSE deliveries.
_unsetBatchRunningIfAllResults(entry.batch);
// Result blocks grow scrollHeight; without this the user pinned
// at the bottom loses their pin when the row inflates. appendMsg
// already routes through _scheduleScroll on the legacy path; this
@@ -1166,6 +1173,37 @@
return status;
}
function _setBatchRunning(batch) {
if (!batch) return;
batch.classList.add("coord-tool-batch--running");
const kicker = batch.querySelector(".coord-tool-batch-kicker");
if (kicker) {
const rowCount = batch.querySelectorAll(".coord-tool-row").length;
kicker.textContent =
rowCount >= 2 ? "Running · Parallel " + rowCount : "Running";
}
}
function _unsetBatchRunningIfAllResults(batch) {
// Remove ``--running`` once every row in the batch has rendered a
// result block. Caller invokes after each tool_result; the test
// is "did THIS result complete the batch?" — cheap DOM walk over
// the same handful of rows we already track.
if (!batch) return;
if (!batch.classList.contains("coord-tool-batch--running")) return;
const rows = batch.querySelectorAll(".coord-tool-row");
for (const row of rows) {
if (!row.querySelector(".coord-tool-row-result")) return;
}
batch.classList.remove("coord-tool-batch--running");
const kicker = batch.querySelector(".coord-tool-batch-kicker");
if (kicker && !batch.classList.contains("coord-tool-batch--pending")) {
const rowCount = rows.length;
kicker.textContent =
rowCount >= 2 ? "Parallel · " + rowCount + " tools" : "Tool";
}
}
function _morphBatchResolved(batch, opts) {
if (!batch) return;
batch.classList.remove("coord-tool-batch--pending");
@@ -1283,9 +1321,15 @@
_announceAssertive(_approvalAriaLabel(items));
} else if (
opts.auto &&
existing.classList.contains("coord-tool-batch--running")
existing.classList.contains("coord-tool-batch--running") &&
!existing.classList.contains("coord-tool-batch--auto")
) {
existing.classList.remove("coord-tool-batch--running");
// SSE tool_info clarifies an existing --running batch as
// auto-approved. Keep --running (the tool is still in
// flight; tool_result will remove it) and add --auto so the
// batch reflects BOTH "auto-approved" + "running" — historical
// behaviour swapped --running out, which lost the running
// indicator the moment tool_info clarified the approval state.
existing.classList.add("coord-tool-batch--auto");
} else if (opts.pending) {
// Already pending — keep the action row, just refresh
@@ -1328,7 +1372,6 @@
);
if (opts.pending) batch.classList.add("coord-tool-batch--pending");
else if (opts.auto) batch.classList.add("coord-tool-batch--auto");
else if (opts.running) batch.classList.add("coord-tool-batch--running");
else if (opts.resolved) {
batch.classList.add(
opts.resolved.approved
@@ -1336,6 +1379,12 @@
: "coord-tool-batch--denied",
);
}
// ``running`` is additive — coexists with ``auto`` (auto-approved
// and currently in flight) or stands alone (replay-time orphan
// before SSE clarifies the approval state). Removed by
// :func:`_unsetBatchRunningIfAllResults` when every row in the
// batch has a tool_result.
if (opts.running) batch.classList.add("coord-tool-batch--running");
const head = document.createElement("div");
head.className = "coord-tool-batch-head";
@@ -1960,6 +2009,47 @@
case "reasoning":
appendReasoningToken(ev.text || "");
break;
case "in_progress_snapshot":
// One-shot replay of the in-progress turn's reasoning + content
// when this client connects mid-stream (page refresh while the
// model is generating). Idempotent on EventSource auto-reconnect:
// skip overwrite when the current buffer is already at-or-past
// the snapshot length, so a stale replay can't reset the live-
// streamed view back to a shorter prefix.
if (ev.reasoning && ev.reasoning.length > currentReasoningBuf.length) {
if (!currentReasoningEl) {
currentReasoningEl = appendMsg("reasoning", "", {
label: "reasoning",
});
messagesEl.setAttribute("aria-live", "off");
}
currentReasoningBuf = ev.reasoning;
var rbody = currentReasoningEl.querySelector(".msg-body");
if (rbody) rbody.textContent = currentReasoningBuf;
_scheduleScroll();
}
if (ev.content && ev.content.length > currentAssistantBuf.length) {
if (!currentAssistantEl) {
currentAssistantEl = appendMsg("assistant", "", {
label: "assistant",
});
messagesEl.setAttribute("aria-live", "off");
}
currentAssistantBuf = ev.content;
var abody = currentAssistantEl.querySelector(".msg-body");
if (abody && typeof streamingRender === "function") {
try {
streamingRender(abody, currentAssistantBuf);
} catch (e) {
console.warn("coordinator streamingRender failed", e);
abody.textContent = currentAssistantBuf;
}
} else if (abody) {
abody.textContent = currentAssistantBuf;
}
_scheduleScroll();
}
break;
case "stream_end":
finishAssistantStream();
break;
@@ -2007,11 +2097,19 @@
const wasAlways =
ev.always === true ||
(ev.always === undefined && target.dataset.requestedAlways === "1");
const approved = ev.approved !== false;
_morphBatchResolved(target, {
approved: ev.approved !== false,
approved,
always: wasAlways,
feedback: ev.feedback || null,
});
// Approved batches start running the moment the user clicks
// approve — mirror the auto path so the live RUNNING
// indicator shows during execution (not just on refresh).
// Denied batches don't run at all, so no --running.
if (approved) {
_setBatchRunning(target);
}
}
break;
}
@@ -2202,8 +2300,12 @@
// for both kinds, matching the interactive payload name. All
// items in a single ``tool_info`` envelope share a dispatch
// turn, so render them as one batch construct (parallel when
// ≥2, solo otherwise) rather than N separate bubbles.
appendToolBatch(ev.items || [], { auto: true });
// ≥2, solo otherwise) rather than N separate bubbles. ``auto``
// marks the approval-state class; ``running`` marks "in flight"
// — the tool starts executing the moment auto-approval lands,
// and the batch should show the same RUNNING indicator the
// replay path renders for an unresolved committed turn.
appendToolBatch(ev.items || [], { auto: true, running: true });
break;
// Child-workstream fan-out routed through the coordinator's own
// SSE stream. CoordinatorManager filters the cluster event bus
@@ -4093,6 +4195,20 @@
appendUserMessageWithAttachments(text, [], { label: "user" });
});
} else if (role === "assistant") {
// Reasoning bubble (Phase 1 reasoning persistence) — render
// BEFORE the content card so the visual order matches the
// live SSE flow (reasoning_delta arrives before content_delta
// for thinking-enabled models). Mirrors the live ":1524" /
// snapshot ":2021" call sites — same appendMsg("reasoning")
// helper, just driven from history-render rather than the
// SSE handler. Only present when the active model's
// surface_persisted_reasoning flag is true and the message round-tripped
// a thinking lane.
if (typeof m.reasoning === "string" && m.reasoning.length) {
const rEl = appendMsg("reasoning", "", { label: "reasoning" });
const rBody = rEl && rEl.querySelector(".msg-body");
if (rBody) rBody.textContent = m.reasoning;
}
// Render content BEFORE the tool batch so DOM order matches
// chronological order (the model emits text first, then
// dispatches tools). Whitespace-only content (e.g. "\n\n"
@@ -10,7 +10,7 @@
<link rel="stylesheet" href="/shared/base.css">
<link rel="stylesheet" href="/shared/ui-base.css">
<link rel="stylesheet" href="/shared/chat.css">
<link rel="stylesheet" href="/shared/katex-0.16.45/katex.min.css">
<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">
<link rel="stylesheet" href="/static/style.css">
<link rel="stylesheet" href="/static/coordinator/coordinator.css">
<style>
@@ -634,7 +634,7 @@
<script src="/shared/composer_attachments.js"></script>
<script src="/shared/composer_queue.js"></script>
<script src="/shared/status_bar.js"></script>
<script src="/shared/katex-0.16.45/katex.min.js"></script>
<script src="/shared/katex-0.16.47/katex.min.js"></script>
<script src="/shared/hljs-11.11.1/highlight.min.js"></script>
<script src="/shared/renderer.js"></script>
<script src="/static/coordinator/coordinator.js"></script>
+108 -47
View File
@@ -143,46 +143,91 @@ function _renderGovRoles(items) {
}
}
// All permission names for the checkbox UI
var _ALL_PERMISSIONS = [
"read",
"write",
"approve",
"admin.users",
"admin.roles",
"admin.orgs",
"admin.policies",
"admin.skills",
"admin.audit",
"admin.usage",
"admin.schedules",
"admin.watches",
"admin.judge",
"admin.memories",
"admin.settings",
"admin.mcp",
"tools.approve",
"workstreams.create",
"workstreams.close",
// Permission inventory grouped by namespace so the role modal can
// render each section under its own heading. Sectioning prevents the
// row-flow grid from slicing a namespace mid-column (e.g. half of
// ``admin.*`` ending up in column 1, the rest in column 2) and lets
// readers who don't yet know the permission taxonomy scan by
// concept. Each section's permissions render as a 2-column grid;
// the ``Scopes`` and ``Workstreams & Tools`` sections are short
// enough to fit one row, ``Admin`` carries the bulk.
var _PERMISSION_SECTIONS = [
{
label: "Scopes",
permissions: ["read", "write", "approve"],
},
{
label: "Admin",
permissions: [
"admin.users",
"admin.roles",
"admin.orgs",
"admin.policies",
"admin.skills",
"admin.audit",
"admin.usage",
"admin.schedules",
"admin.watches",
"admin.judge",
"admin.memories",
"admin.settings",
"admin.mcp",
],
},
{
label: "Workstreams & Tools",
permissions: ["workstreams.create", "workstreams.close", "tools.approve"],
},
];
function _buildPermCheckboxes(prefix, selected) {
var html = '<div class="perm-grid">';
for (var i = 0; i < _ALL_PERMISSIONS.length; i++) {
var p = _ALL_PERMISSIONS[i];
var checked = selected && selected.indexOf(p) >= 0 ? " checked" : "";
html +=
'<label class="perm-checkbox"><input type="checkbox" value="' +
p +
'" name="' +
prefix +
'-perm"' +
checked +
"> " +
escapeHtml(p) +
"</label>";
// Flat list — kept for any caller that wants the full permission
// inventory without caring about sectioning.
var _ALL_PERMISSIONS = (function () {
var flat = [];
for (var i = 0; i < _PERMISSION_SECTIONS.length; i++) {
flat = flat.concat(_PERMISSION_SECTIONS[i].permissions);
}
return flat;
})();
function _buildPermCheckboxes(prefix, selected) {
// Emits the toggle-switch component used elsewhere in the admin
// modals so each permission reads as a deliberate on/off rather
// than a generic checkbox. Sections are wrapped in a
// ``.perm-section`` block with a caps-styled heading so the
// typographic system inside the role modal stays consistent (the
// surrounding label cadence is also caps + 0.08em letter-spacing).
// The underlying ``<input type="checkbox" name="{prefix}-perm">``
// shape is preserved so ``_collectPermCheckboxes`` still picks
// them up regardless of section.
var html = "";
for (var s = 0; s < _PERMISSION_SECTIONS.length; s++) {
var section = _PERMISSION_SECTIONS[s];
html +=
'<div class="perm-section">' +
'<div class="perm-section-label">' +
escapeHtml(section.label) +
"</div>" +
'<div class="perm-grid">';
for (var i = 0; i < section.permissions.length; i++) {
var p = section.permissions[i];
var checked = selected && selected.indexOf(p) >= 0 ? " checked" : "";
html +=
'<label class="toggle-switch perm-toggle">' +
'<input type="checkbox" value="' +
p +
'" name="' +
prefix +
'-perm"' +
checked +
">" +
'<span class="toggle-track" aria-hidden="true"></span>' +
'<span class="toggle-label">' +
escapeHtml(p) +
"</span></label>";
}
html += "</div></div>";
}
html += "</div>";
return html;
}
@@ -346,20 +391,27 @@ function showUserRolesModal(userId) {
var assigned = {};
for (var i = 0; i < userRoles.length; i++)
assigned[userRoles[i].role_id] = true;
var html = "";
// Role-assignment rows reuse the toggle-switch component for
// consistency with the rest of the admin UX. Role display names
// are human-readable text, so no monospace override is needed.
var html = '<div class="user-roles-list">';
for (var j = 0; j < allRoles.length; j++) {
var r = allRoles[j];
var checked = assigned[r.role_id] ? " checked" : "";
html +=
'<label class="perm-checkbox"><input type="checkbox" value="' +
'<label class="toggle-switch user-role-toggle">' +
'<input type="checkbox" value="' +
escapeHtml(r.role_id) +
'" name="ur-role"' +
checked +
"> " +
">" +
'<span class="toggle-track" aria-hidden="true"></span>' +
'<span class="toggle-label">' +
escapeHtml(r.display_name) +
"</label>";
"</span></label>";
}
container.innerHTML = html;
html += "</div>";
container.innerHTML = html; // values escaped via escapeHtml above
})
.catch(function () {
container.innerHTML =
@@ -3199,18 +3251,27 @@ function renderJudgeSettings() {
var isDefault = s.source === "default";
if (s.type === "bool") {
// Toggle switch — same component used by the admin modals.
// ``onchange`` reads the box's new ``.checked`` and writes via
// saveJudgeSetting. The ``.toggle-label`` is a static "Enabled"
// because the slider position is the truth — flipping the
// caption text on save round-tripped through reload, so the
// slider moved instantly while the caption lagged 50-300ms and
// looked broken. ``aria-label`` carries the setting name so
// screen readers get the row context inline.
inputHtml =
'<label class="toggle-label" style="display:flex;align-items:center;gap:8px;cursor:pointer">' +
'<label class="toggle-switch toggle--flush">' +
'<input type="checkbox" data-key="' +
s.key +
'" aria-label="' +
escapeHtml(shortKey) +
'" ' +
(currentVal ? "checked" : "") +
" onchange=\"saveJudgeSetting('" +
s.key +
'\',this.checked)" style="width:16px;height:16px">' +
'<span style="font-size:12px">' +
(currentVal ? "Enabled" : "Disabled") +
"</span></label>";
"',this.checked)\">" +
'<span class="toggle-track" aria-hidden="true"></span>' +
'<span class="toggle-label">Enabled</span></label>';
} else if (s.type === "float") {
inputHtml =
'<div style="display:flex;gap:8px;align-items:center">' +
+136 -100
View File
@@ -2507,10 +2507,11 @@
rows="3"
placeholder="What should the workstream do?"
></textarea>
<label class="admin-checkbox"
><input id="cs-autoapprove" type="checkbox" /> Auto-approve tool
calls</label
>
<label class="toggle-switch">
<input id="cs-autoapprove" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Auto-approve tool calls</span>
</label>
<label
>Notify on completion
<span class="label-hint">optional</span></label
@@ -2609,13 +2610,16 @@
</select>
<label for="es-message">Initial message</label>
<textarea id="es-message" rows="3"></textarea>
<label class="admin-checkbox"
><input id="es-autoapprove" type="checkbox" /> Auto-approve tool
calls</label
>
<label class="admin-checkbox"
><input id="es-enabled" type="checkbox" /> Enabled</label
>
<label class="toggle-switch">
<input id="es-autoapprove" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Auto-approve tool calls</span>
</label>
<label class="toggle-switch">
<input id="es-enabled" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Enabled</span>
</label>
<label
>Notify on completion
<span class="label-hint">optional</span></label
@@ -2859,9 +2863,11 @@
</select>
<label for="ep-priority">Priority</label>
<input id="ep-priority" type="number" value="0" min="0" max="9999" />
<label class="admin-checkbox"
><input id="ep-enabled" type="checkbox" checked /> Enabled</label
>
<label class="toggle-switch">
<input id="ep-enabled" type="checkbox" checked />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Enabled</span>
</label>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideEditPolicyModal()">
Cancel
@@ -2963,9 +2969,11 @@
<textarea id="epp-content" rows="10" spellcheck="false"></textarea>
<label for="epp-priority">Priority</label>
<input id="epp-priority" type="number" value="0" min="0" max="9999" />
<label class="admin-checkbox"
><input id="epp-enabled" type="checkbox" checked /> Enabled</label
>
<label class="toggle-switch">
<input id="epp-enabled" type="checkbox" checked />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Enabled</span>
</label>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideEditPromptPolicyModal()">
Cancel
@@ -3083,10 +3091,11 @@
</option>
<option value="search">Search — BM25 discoverable</option>
</select>
<label class="admin-checkbox"
><input id="ctm-default" type="checkbox" /> Apply to new
workstreams by default</label
>
<label class="toggle-switch">
<input id="ctm-default" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Apply to new workstreams by default</span>
</label>
</div>
</div>
<div class="skill-spec-col skill-spec-col-content">
@@ -3189,10 +3198,11 @@
/>
</div>
</div>
<label class="admin-checkbox"
><input id="csk-auto-approve" type="checkbox" /> Auto-approve all
tools</label
>
<label class="toggle-switch">
<input id="csk-auto-approve" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Auto-approve all tools</span>
</label>
<label for="csk-allowed-tools"
>Allowed Tools
<span class="label-hint"
@@ -3222,9 +3232,11 @@
style="display: block; margin-top: 3px"
>JSON array. Each: channel_type + channel_id or user_id</span
>
<label class="admin-checkbox"
><input id="csk-enabled" type="checkbox" checked /> Enabled</label
>
<label class="toggle-switch">
<input id="csk-enabled" type="checkbox" checked />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Enabled</span>
</label>
</details>
<details class="admin-details">
<summary>
@@ -3395,10 +3407,11 @@
</option>
<option value="search">Search — BM25 discoverable</option>
</select>
<label class="admin-checkbox"
><input id="etm-default" type="checkbox" /> Apply to new
workstreams by default</label
>
<label class="toggle-switch">
<input id="etm-default" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Apply to new workstreams by default</span>
</label>
</div>
</div>
<div class="skill-spec-col skill-spec-col-content">
@@ -3480,10 +3493,11 @@
/>
</div>
</div>
<label class="admin-checkbox"
><input id="esk-auto-approve" type="checkbox" /> Auto-approve all
tools</label
>
<label class="toggle-switch">
<input id="esk-auto-approve" type="checkbox" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Auto-approve all tools</span>
</label>
<label for="esk-allowed-tools"
>Allowed Tools
<span class="label-hint"
@@ -3513,9 +3527,11 @@
style="display: block; margin-top: 3px"
>JSON array. Each: channel_type + channel_id or user_id</span
>
<label class="admin-checkbox"
><input id="esk-enabled" type="checkbox" checked /> Enabled</label
>
<label class="toggle-switch">
<input id="esk-enabled" type="checkbox" checked />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Enabled</span>
</label>
</details>
<div id="etm-scan-section" style="display: none" class="admin-field">
<span class="admin-field-heading" id="etm-scan-heading"
@@ -3699,43 +3715,48 @@
<legend style="font-size: 12px; padding: 0 6px">
Multitenant Authorization
</legend>
<label
style="display: block; margin: 4px 0; font-weight: 400; font-size: 13px"
>
<input
type="radio"
name="mcp-auth-type"
id="mcp-auth-none"
value="none"
onchange="toggleMcpAuthFields()"
style="margin-right: 6px"
/>No authorization
</label>
<label
style="display: block; margin: 4px 0; font-weight: 400; font-size: 13px"
>
<input
type="radio"
name="mcp-auth-type"
id="mcp-auth-static"
value="static"
onchange="toggleMcpAuthFields()"
checked
style="margin-right: 6px"
/>Static headers (single shared identity)
</label>
<label
style="display: block; margin: 4px 0; font-weight: 400; font-size: 13px"
>
<input
type="radio"
name="mcp-auth-type"
id="mcp-auth-oauth"
value="oauth_user"
onchange="toggleMcpAuthFields()"
style="margin-right: 6px"
/>Per-user OAuth 2.1 (recommended)
</label>
<div class="segmented-control" role="radiogroup" aria-label="Multitenant Authorization">
<label class="segmented-option">
<input
type="radio"
name="mcp-auth-type"
id="mcp-auth-none"
value="none"
onchange="toggleMcpAuthFields()"
/>
<span class="segmented-indicator" aria-hidden="true"></span>
<span class="segmented-text">No authorization</span>
</label>
<label class="segmented-option">
<input
type="radio"
name="mcp-auth-type"
id="mcp-auth-static"
value="static"
onchange="toggleMcpAuthFields()"
checked
/>
<span class="segmented-indicator" aria-hidden="true"></span>
<span class="segmented-text"
>Static headers
<span class="segmented-hint">single shared identity</span></span
>
</label>
<label class="segmented-option">
<input
type="radio"
name="mcp-auth-type"
id="mcp-auth-oauth"
value="oauth_user"
onchange="toggleMcpAuthFields()"
/>
<span class="segmented-indicator" aria-hidden="true"></span>
<span class="segmented-text"
>Per-user OAuth 2.1
<span class="segmented-hint">recommended</span></span
>
</label>
</div>
<div id="mcp-oauth-fields" style="display: none; margin-top: 8px">
<label for="mcp-oauth-as-url"
>Authorization Server URL
@@ -3796,22 +3817,17 @@
/>
</div>
</fieldset>
<div style="display: flex; gap: 20px; margin-top: 14px">
<label style="margin: 0; font-size: 12px; color: var(--fg-dim)"
><input
type="checkbox"
id="mcp-auto-approve"
style="margin-right: 5px"
/>Auto-approve tools</label
>
<label style="margin: 0; font-size: 12px; color: var(--fg-dim)"
><input
type="checkbox"
id="mcp-enabled"
checked
style="margin-right: 5px"
/>Enabled</label
>
<div class="toggle-stack">
<label class="toggle-switch">
<input type="checkbox" id="mcp-auto-approve" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Auto-approve tools</span>
</label>
<label class="toggle-switch">
<input type="checkbox" id="mcp-enabled" checked />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Enabled</span>
</label>
</div>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideCreateMcpModal()">
@@ -3936,6 +3952,14 @@
<h2 id="model-create-title">Add Model</h2>
<div id="model-create-error" role="alert" aria-live="assertive" aria-atomic="true"></div>
<input type="hidden" id="model-edit-id" value="" />
<label
class="toggle-switch toggle--flush"
title="Disable to hide this alias from every model dropdown (workstreams, schedules, channel adapters, role assignments) without removing the definition."
>
<input type="checkbox" id="model-enabled" checked />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Active</span>
</label>
<label for="model-alias">Alias</label>
<input
type="text"
@@ -4116,15 +4140,27 @@
placeholder='{"supports_vision": true}'
style="font-family: var(--font-mono); font-size: 11px"
></textarea>
<div style="display: flex; gap: 20px; margin-top: 14px">
<label style="margin: 0; font-size: 12px; color: var(--fg-dim)"
><input
type="checkbox"
id="model-enabled"
checked
style="margin-right: 5px"
/>Enabled</label
<div class="toggle-stack">
<label
class="toggle-switch"
title="Surface stored reasoning text on /history responses (UI bubble on page reload). Storage of reasoning bytes is unaffected by this flag — they ride in provider_data regardless."
>
<input
type="checkbox"
id="model-surface-persisted-reasoning"
checked
/>
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Surface persisted reasoning</span>
</label>
<label
class="toggle-switch"
title="Replay stored reasoning blocks back to the model on subsequent provider calls. Capability-dependent; off by default for cost/spec compliance."
>
<input type="checkbox" id="model-replay-reasoning" />
<span class="toggle-track" aria-hidden="true"></span>
<span class="toggle-label">Replay reasoning to model</span>
</label>
</div>
<div id="model-detect-area" style="margin-top: 14px">
<button
+343 -27
View File
@@ -1572,23 +1572,284 @@
padding-left: 12px;
}
/* Checkbox labels inside admin modals */
.admin-modal label.admin-checkbox {
/* Toggle switch modern replacement for boolean checkboxes inside
* admin modals. The native <input type="checkbox"> stays in the
* markup (visually hidden but keyboard-focusable) so existing JS
* that reads `.checked` keeps working; the .toggle-track + ::before
* pseudo render the slider, .toggle-label carries the caption.
*
* Usage:
* <label class="toggle-switch">
* <input type="checkbox" id="..." />
* <span class="toggle-track" aria-hidden="true"></span>
* <span class="toggle-label">Enabled</span>
* </label>
*/
/* Selector is doubled with .admin-modal so we win the specificity
* battle against ``.admin-modal label`` (0,1,1) without that, the
* parent rule's display:block + text-transform:uppercase + margins
* cascade and we lose the inline-flex layout.
*
* Default margin-top: 14px matches the .admin-modal label cadence
* for toggles that sit directly between regular labelled rows
* (schedule/policy/skill modals). When a toggle is inside an
* explicit ``.toggle-stack`` flex container, the stack resets the
* margin so the parent's gap controls spacing on its own. */
.admin-modal label.toggle-switch,
label.toggle-switch {
display: inline-flex;
align-items: center;
gap: 10px;
cursor: pointer;
user-select: none;
margin: 14px 0 0 0;
padding: 0;
text-transform: none;
letter-spacing: 0;
font-size: 12px;
font-weight: 500;
color: var(--fg);
}
.admin-modal .toggle-stack > label.toggle-switch,
.toggle-stack > label.toggle-switch {
margin: 0;
}
/* Modifier for toggles that should sit flush against the surrounding
* rhythm rather than carry the default 14px top margin used when a
* toggle is the first row of a modal (under the h2) or when it lives
* in a dynamically-rendered row that already supplies its own
* spacing (e.g. judge bool settings). */
.admin-modal label.toggle-switch.toggle--flush,
label.toggle-switch.toggle--flush {
margin-top: 0;
}
.admin-modal .toggle-stack,
.toggle-stack {
display: flex;
flex-direction: column;
gap: 10px;
margin-top: 16px;
}
/* Hidden-input + track rules also need the .admin-modal prefix to
* outrank ``.admin-modal input:not([type="hidden"])`` (same
* specificity 0,2,1; that one comes later in source so without the
* bump it wins and forces width:100% on the hidden input, popping it
* back into the layout). */
.admin-modal .toggle-switch input[type="checkbox"],
.toggle-switch input[type="checkbox"] {
/* Visually hidden but still focusable + click-targetable via the
* <label> wrap. Avoids display:none, which would strip the input
* from the keyboard tab order. */
position: absolute;
width: 1px;
height: 1px;
margin: -1px;
padding: 0;
border: 0;
overflow: hidden;
clip: rect(0 0 0 0);
white-space: nowrap;
}
.admin-modal .toggle-switch .toggle-track,
.toggle-switch .toggle-track {
position: relative;
flex: 0 0 auto;
/* 40×22 meets WCAG 2.5.5 (Level AAA) target size when combined with
* the label's hit area, and reads as a deliberate switch on touch
* targets without dominating the form rhythm. */
width: 40px;
height: 22px;
/* Off state: depressed inset on the modal background so the track
* silhouette stays visible at 1.5+ contrast ratio. The earlier
* ``var(--border-strong)`` solid fill was ~1.18:1 against the modal
* surface visible to most readers, invisible to anyone on a
* glare-y screen or at ``prefers-contrast: more``. */
background: var(--bg);
box-shadow: inset 0 0 0 1px var(--border-strong);
border-radius: 11px;
transition:
background 0.18s ease,
box-shadow 0.18s ease;
}
.admin-modal .toggle-switch .toggle-track::before,
.toggle-switch .toggle-track::before {
content: "";
position: absolute;
top: 3px;
left: 3px;
width: 16px;
height: 16px;
background: var(--fg);
border-radius: 50%;
transition: transform 0.18s ease;
}
.admin-modal .toggle-switch input:checked + .toggle-track,
.toggle-switch input:checked + .toggle-track {
background: var(--accent);
box-shadow: none;
}
.admin-modal .toggle-switch input:checked + .toggle-track::before,
.toggle-switch input:checked + .toggle-track::before {
transform: translateX(18px);
background: var(--bg);
}
.admin-modal .toggle-switch input:focus-visible + .toggle-track,
.toggle-switch input:focus-visible + .toggle-track {
box-shadow: 0 0 0 3px var(--accent-dim);
}
.admin-modal .toggle-switch input:disabled + .toggle-track,
.toggle-switch input:disabled + .toggle-track {
opacity: 0.4;
}
.toggle-switch:has(input:disabled) {
cursor: not-allowed;
opacity: 0.7;
}
.toggle-switch .toggle-label {
display: inline-block;
/* The label text rides at the modal's normal label cadence caps
* + letter-spacing so it sits next to the surrounding form rows
* without shifting the visual rhythm. */
font-family: var(--font-ui);
font-size: 10px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.08em;
color: var(--fg-dim);
}
/* Hair divider for separating conceptually-grouped toggles inside a
* stack used between the lone "Enabled" toggle and the paired
* Reasoning toggles in the Add Model modal so the grouping reads
* without needing a subheading or indent. Margin: 0 because the
* .toggle-stack flex container already supplies a 10px gap on each
* side; adding a margin on top would visually separate the divider
* twice. */
.admin-modal hr.toggle-group-divider,
hr.toggle-group-divider {
border: 0;
border-top: 1px solid var(--border);
margin: 0;
width: 100%;
}
/* Segmented option list vertical card group for radio choices that
* benefit from a strong selected-state highlight. The native
* ``<input type="radio">`` is visually hidden but stays focusable;
* the ``.segmented-indicator`` pseudo-circle and the row's
* background carry the selected state.
*
* Usage:
* <div class="segmented-control" role="radiogroup">
* <label class="segmented-option">
* <input type="radio" name="..." value="..." />
* <span class="segmented-indicator" aria-hidden="true"></span>
* <span class="segmented-text">Label
* <span class="segmented-hint">(hint)</span>
* </span>
* </label>
* ...
* </div>
*/
.admin-modal .segmented-control,
.segmented-control {
display: flex;
flex-direction: column;
border: 1px solid var(--border-strong);
border-radius: var(--radius-sm);
overflow: hidden;
background: var(--bg);
margin-top: 8px;
}
.admin-modal .segmented-option,
.segmented-option {
display: flex;
align-items: center;
gap: 8px;
font-size: 12px;
gap: 10px;
padding: 11px 14px;
cursor: pointer;
margin: 0;
font-family: var(--font-ui);
font-size: 13px;
font-weight: 500;
text-transform: none;
letter-spacing: 0;
color: var(--fg);
cursor: pointer;
margin-top: 14px;
color: var(--fg-dim);
border-top: 1px solid var(--border);
transition:
background 0.15s ease,
color 0.15s ease;
}
.admin-modal label.admin-checkbox input[type="checkbox"],
.admin-modal label.admin-checkbox input[type="radio"] {
width: auto;
margin: 0;
.segmented-option:first-of-type {
border-top: none;
}
.segmented-option input[type="radio"] {
/* Visually hidden but focusable + click-targetable via the label
* wrap. Keyboard arrows still cycle within the radiogroup. */
position: absolute;
width: 1px;
height: 1px;
margin: -1px;
padding: 0;
border: 0;
overflow: hidden;
clip: rect(0 0 0 0);
}
.segmented-option .segmented-indicator {
position: relative;
flex: 0 0 auto;
width: 16px;
height: 16px;
border-radius: 50%;
background: var(--bg);
box-shadow: inset 0 0 0 1px var(--border-strong);
transition:
background 0.15s ease,
box-shadow 0.15s ease;
}
.segmented-option .segmented-indicator::after {
content: "";
position: absolute;
top: 4px;
left: 4px;
width: 6px;
height: 6px;
border-radius: 50%;
background: transparent;
transition: background 0.15s ease;
}
.segmented-option:hover {
background: var(--bg-highlight);
color: var(--fg);
}
.segmented-option:has(input:checked) {
background: var(--accent-dim);
color: var(--fg);
}
.segmented-option:has(input:checked) .segmented-indicator {
box-shadow: inset 0 0 0 1px var(--accent);
}
.segmented-option:has(input:checked) .segmented-indicator::after {
background: var(--accent);
}
.segmented-option:has(input:focus-visible) {
/* Bright accent (not --accent-dim) so the ring stays visible even
* on the currently-selected row, which already paints --accent-dim
* as its background. */
box-shadow: inset 0 0 0 2px var(--accent);
}
.segmented-option .segmented-text {
flex: 1 1 auto;
min-width: 0;
}
.segmented-option .segmented-hint {
font-size: 11px;
font-weight: 400;
color: var(--fg-dim);
margin-left: 4px;
}
.segmented-option:has(input:checked) .segmented-hint {
color: var(--accent);
}
/* Admin modals */
@@ -1692,9 +1953,6 @@
border-color: var(--border);
color: var(--fg-dim);
}
.admin-modal label.admin-checkbox input:disabled {
opacity: 0.4;
}
.admin-modal input::placeholder,
.admin-modal textarea::placeholder {
color: var(--fg-dim);
@@ -2755,28 +3013,64 @@ textarea.skill-content-area {
padding: 0;
margin-bottom: 4px;
}
/* Permission section grouping keeps each namespace contiguous so
* the row-flow grid below doesn't slice ``admin.*`` mid-column. The
* caps-styled ``.perm-section-label`` re-anchors the toggles to the
* modal's typographic system (matches ``.admin-modal label``: 10px
* caps, 0.08em letter-spacing, fg-dim). */
.perm-section {
margin-top: 12px;
}
.perm-section:first-of-type {
margin-top: 0;
}
.perm-section-label {
font-family: var(--font-ui);
font-size: 10px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.08em;
color: var(--fg-dim);
margin-bottom: 4px;
}
.perm-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 4px 16px;
padding: 8px 0;
gap: 8px 16px;
padding: 4px 0 0;
}
.perm-checkbox {
display: flex;
align-items: center;
gap: 6px;
font-size: 11px;
/* Permission rows inherit the toggle-switch component but render the
* permission name in monospace + lower case (it's an identifier, not
* a heading) overrides the .toggle-label's caps/letter-spacing
* cadence used elsewhere in admin modals. */
.admin-modal .perm-grid label.toggle-switch.perm-toggle,
.perm-grid label.toggle-switch.perm-toggle {
margin: 0;
}
.perm-grid .toggle-switch.perm-toggle .toggle-label {
font-family: var(--font-mono);
color: var(--fg);
padding: 3px 0;
cursor: pointer;
font-size: 11px;
font-weight: 500;
text-transform: none;
letter-spacing: normal;
color: var(--fg);
}
.perm-checkbox input[type="checkbox"] {
width: auto;
/* User-role assignment list (Users tab Manage roles modal). Same
* toggle-switch component as elsewhere, but the labels are
* human-readable role display names so we keep ui-font + caps-on so
* the stack reads like a settings list. Only override the layout
* margin (the modal already provides outer padding). */
.admin-modal .user-roles-list,
.user-roles-list {
display: flex;
flex-direction: column;
gap: 6px;
margin-top: 8px;
}
.admin-modal .user-roles-list label.toggle-switch.user-role-toggle,
.user-roles-list label.toggle-switch.user-role-toggle {
margin: 0;
accent-color: var(--accent);
}
/* ==========================================================================
@@ -3308,6 +3602,28 @@ textarea.skill-content-area {
opacity: 0.4;
}
/* Phase 9 last-refresh pill in the MCP status cell. Compact age +
outcome indicator inline with the existing status text. Uses the
semantic theme tokens (--bg-highlight, --fg-dim, --warn) defined in
shared_static/base.css so the pill follows dark/light theme swaps. */
.mcp-refresh-pill {
display: inline-block;
margin-left: 6px;
padding: 0 4px;
border-radius: var(--radius-sm);
font-size: 0.8em;
font-variant-numeric: tabular-nums;
opacity: 0.85;
}
.mcp-refresh-pill-ok {
background: var(--bg-highlight);
color: var(--fg-dim);
}
.mcp-refresh-pill-err {
background: color-mix(in srgb, var(--warn) 15%, transparent);
color: var(--warn);
}
.mcp-detail-modal::before {
background: linear-gradient(
90deg,
+15 -1
View File
@@ -513,7 +513,21 @@ def is_public_path(path: str) -> bool:
normalized = _strip_version_prefix(path)
if normalized in PUBLIC_PATHS:
return True
return any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES)
if any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES):
return True
# Console proxy: a public proxied path is still public. Without this
# the login/status/setup endpoints are unreachable from inside a
# ``/node/{id}/...`` proxied page once the cookie expires — the
# AuthMiddleware 401s the login POST before any handler runs and the
# user is locked out of the proxied UI.
if normalized.startswith("/node/"):
proxied = _extract_proxied_path(normalized)
if proxied is not None:
if proxied in PUBLIC_PATHS:
return True
if any(proxied.startswith(prefix) for prefix in PUBLIC_PREFIXES):
return True
return False
def required_scope(method: str, path: str) -> str:
+121
View File
@@ -0,0 +1,121 @@
"""Per-workstream wakeup primitive for in-process child state-change subscribers.
Retires the polling pattern in ``CoordinatorClient.wait_for_workstream``,
where the coord LLM's wait tool issued a storage snapshot every 0.5 s
regardless of whether anything had changed. The dispatch path
(:meth:`turnstone.console.coordinator_adapter.CoordinatorAdapter._dispatch_child_event`)
now calls :meth:`ChildEventBus.notify` after each translated child event;
waiters block on a per-call :class:`threading.Event` returned by
:meth:`register_waiter` and re-read storage only when an event fires or
the heartbeat cap expires.
Bus is in-process only. Cross-process / cross-node child events are
already merged into ``_dispatch_child_event`` via the cluster collector's
SSE multiplex before the bus sees them there is no locality branching
in the bus itself.
Design constraints:
- Waiter primitive is :class:`threading.Event` because the wait tool runs
on the coordinator's sync worker thread, not an asyncio loop.
- Concurrent ``register`` / ``unregister`` / ``notify`` is safe a
single ``threading.Lock`` guards the dict. ``Event.set`` itself is
thread-safe and is called outside the lock so a slow waker can't block
registration.
- ``notify`` with no subscribers is a no-op (the steady state most
state-change events fire while no wait tool is active).
- A waiter watching multiple ws_ids fires once on any of them; the
caller's ``_snapshot_all`` re-read resolves which one changed.
- Empty per-ws_id buckets are popped on unregister so long-lived buses
don't accumulate dead keys after wait churn.
"""
from __future__ import annotations
import threading
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Iterable
class ChildEventBus:
"""Fan ``notify(child_ws_id)`` to every :class:`threading.Event`
registered against that ws_id.
Use :meth:`register_waiter` once per wait call to obtain an Event,
then call :meth:`unregister_waiter` in a ``finally`` so a crash mid-
wait doesn't leak the registration. The dispatch side calls
:meth:`notify` on every translated child state-change event; an
empty bucket is a cheap dict lookup + immediate return.
"""
def __init__(self) -> None:
self._waiters: dict[str, set[threading.Event]] = {}
self._lock = threading.Lock()
def register_waiter(self, child_ws_ids: Iterable[str]) -> threading.Event:
"""Return a fresh Event registered against every listed ws_id.
A wait on ``[A, B, C]`` returns a single Event that fires when
*any* of A/B/C changes. The caller's snapshot re-read resolves
which one. Empty / falsy ids are silently skipped callers that
clean their input upstream (e.g. ``wait_for_workstream``'s
dedup + cap) don't need to filter again here.
"""
event = threading.Event()
with self._lock:
for wid in child_ws_ids:
if not wid:
continue
self._waiters.setdefault(wid, set()).add(event)
return event
def unregister_waiter(
self,
child_ws_ids: Iterable[str],
event: threading.Event,
) -> None:
"""Remove ``event`` from each listed ws_id's waiter set.
Idempotent already-removed Events silently no-op. Pops empty
sets so a long-lived bus doesn't accumulate dead keys after
many waits have come and gone. Must be called from the same
``finally`` that paired with :meth:`register_waiter` so a
crash mid-wait doesn't leak the registration past one wait's
lifetime.
"""
with self._lock:
for wid in child_ws_ids:
if not wid:
continue
bucket = self._waiters.get(wid)
if bucket is None:
continue
bucket.discard(event)
if not bucket:
self._waiters.pop(wid, None)
def notify(self, child_ws_id: str) -> None:
"""Wake every Event registered for ``child_ws_id``.
Called from the coord dispatch sink after each translated child
event. Snapshot the bucket under the lock, then call
``Event.set`` outside the lock so a slow waker doesn't block
``register`` / ``unregister`` / further ``notify``. ``Event.set``
is thread-safe and idempotent re-firing a still-set Event is
a no-op.
Empty / falsy ws_ids are silently dropped; the same hot path
runs for every dispatched event regardless of whether anyone's
waiting, so the empty-bucket case must stay cheap.
"""
if not child_ws_id:
return
with self._lock:
bucket = self._waiters.get(child_ws_id)
if not bucket:
return
events = list(bucket)
for event in events:
event.set()
+23
View File
@@ -54,6 +54,27 @@ def set_config_path(path: str) -> None:
_cache = None # invalidate cache so next load_config() re-reads
def _warn_if_world_readable(cfg_path: Path) -> None:
"""Warn once if config.toml is group- or world-readable.
DB passwords, OIDC client secrets, and TLS key paths live in this
file operators usually want it at 0600. POSIX-only; no-ops where
``stat()`` modes are meaningless (Windows).
"""
try:
mode = cfg_path.stat().st_mode & 0o777
except OSError:
return
if mode & 0o077:
log.warning(
"%s is mode %04o (group/world-readable); secrets live here — "
"run `chmod 0600 %s` to restrict access",
cfg_path,
mode,
cfg_path,
)
def load_config(section: str | None = None) -> dict[str, Any]:
"""Load config.toml and return the full dict or a specific section.
@@ -66,6 +87,7 @@ def load_config(section: str | None = None) -> dict[str, Any]:
cfg_path = _resolve_config_path()
if cfg_path.is_file():
try:
_warn_if_world_readable(cfg_path)
_cache = tomllib.loads(cfg_path.read_text(encoding="utf-8"))
except Exception as exc:
log.warning("Failed to parse %s: %s", cfg_path, exc)
@@ -143,6 +165,7 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
"sslrootcert": "db_sslrootcert",
"sslcert": "db_sslcert",
"sslkey": "db_sslkey",
"listen_url": "db_listen_url",
},
"judge": {
"enabled": "judge_enabled",
+156 -1
View File
@@ -19,7 +19,7 @@ either an async caller (via ``asyncio.to_thread``) or a sync hook.
from __future__ import annotations
import json
from typing import Any
from typing import TYPE_CHECKING, Any
from turnstone.core.log import get_logger
from turnstone.core.tool_advisory import (
@@ -268,6 +268,161 @@ def extract_advisories_from_tool_envelope(
return _entity_decode_wrapper_tags(inner), advisories
if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.core.providers._protocol import LLMProvider
def _make_provider_factory(module_path: str, class_name: str) -> Callable[[], LLMProvider]:
"""Build a thread-unsafe lazy-init factory for a provider singleton.
Each block-type entry in ``_BLOCK_TYPE_PROVIDER_FACTORY`` closes
over its own (module_path, class_name) pair. Adding a fourth
provider is a single tuple in the dict, not a new 9-line getter.
Uses ``nonlocal`` instead of ``functools.lru_cache`` so the cache
state stays inside this closure (lru_cache would attach state to
the inner function object, which is correct but adds a per-call
hash lookup on a bound key for what's effectively a single-slot
cache).
"""
instance: LLMProvider | None = None
def factory() -> LLMProvider:
nonlocal instance
if instance is None:
import importlib
module = importlib.import_module(module_path)
instance = getattr(module, class_name)()
return instance
return factory
# Block-type → provider factory. Routing is structural — block shape
# is non-overlapping across providers by API design. Recognised
# block types today:
#
# * ``"thinking"`` — Anthropic native (Phase 1). Walks the
# ``thinking`` field on each block.
# * ``"redacted_thinking"`` — Anthropic native (Phase 1). Anthropic's
# safety system rewrites a thinking block into a sealed
# ``redacted_thinking`` block; the Anthropic docs note these can
# appear before, after, or interleaved with regular ``thinking``
# blocks. Same factory: AnthropicProvider's extractor walks the
# full block list and filters to ``type == "thinking"``, so the
# redacted blocks are correctly skipped while the surrounding
# real thinking text still surfaces.
# * ``"reasoning"`` — OpenAI Responses native (Phase 3). Walks
# ``summary[*].text`` (always present) and ``content[*].text``
# (present when ``include=["reasoning.encrypted_content"]`` is
# requested AND the response carries raw reasoning text).
# * ``"reasoning_text"`` — synthetic, stamped by
# ``ChatSession._maybe_synth_reasoning_block`` for Chat Completions
# paths (vLLM, llama.cpp, Gemini-compat) where reasoning surfaces
# only as ``reasoning_delta`` chunks with no native block shape.
_anthropic_factory = _make_provider_factory(
"turnstone.core.providers._anthropic", "AnthropicProvider"
)
_BLOCK_TYPE_PROVIDER_FACTORY: dict[str, Callable[[], LLMProvider]] = {
"thinking": _anthropic_factory,
"redacted_thinking": _anthropic_factory,
"reasoning": _make_provider_factory(
"turnstone.core.providers._openai_responses", "OpenAIResponsesProvider"
),
"reasoning_text": _make_provider_factory(
"turnstone.core.providers._openai_chat", "OpenAIChatCompletionsProvider"
),
}
def extract_reasoning_text_from_provider_content(provider_content: Any) -> str:
"""Dispatch reasoning extraction by scanning for a recognised block type.
Walks ``provider_content`` looking for the first block whose
``type`` is in :data:`_BLOCK_TYPE_PROVIDER_FACTORY`, then dispatches
the WHOLE list to that provider's ``extract_reasoning_text``. Each
provider's extractor already filters internally by its own block
type (Anthropic walks ``thinking``, OpenAI Responses walks
``reasoning``, OpenAI Chat walks ``reasoning_text``), so passing
the full list is correct interleaved foreign blocks are ignored.
Returns ``""`` for empty / missing / non-list input or when no
recognised reasoning-bearing block type appears anywhere in the
list.
Why scan instead of just inspecting ``provider_content[0]``: the
OpenAI Responses streaming layer captures EVERY ``output_item.done``
event into ``provider_blocks`` (``_openai_responses.py:415-420``),
not just reasoning items. In practice the order is usually
``[reasoning, message, ...]`` but the API doesn't guarantee that —
a hypothetical ``[message, reasoning]`` ordering would silently
drop the reasoning under an index-only check. Same robustness
point for Anthropic's hypothetical mixed-order outputs.
Pure transform safe from any thread. Both history surfaces
(interactive ``_build_history`` and lifted ``make_history_handler``)
call this directly. See ``_BLOCK_TYPE_PROVIDER_FACTORY`` above
for the recognised block types and the providers that own them.
"""
if not isinstance(provider_content, list) or not provider_content:
return ""
for block in provider_content:
if not isinstance(block, dict):
continue
block_type = block.get("type")
if not isinstance(block_type, str):
continue
factory = _BLOCK_TYPE_PROVIDER_FACTORY.get(block_type)
if factory is not None:
return factory().extract_reasoning_text(provider_content)
return ""
def extract_reasoning_for_history(
messages: list[dict[str, Any]],
surface_persisted_reasoning_flag: bool,
) -> None:
"""Surface stored reasoning text on each assistant message; strip the
raw provider content from the wire payload.
For the ``make_history_handler`` REST path where the response
payload IS the messages list returned from ``storage.load_messages``
both extraction source and stamp destination are the same dict.
Walks *messages* in place: for every assistant message, dispatches
via :func:`extract_reasoning_text_from_provider_content` and stamps
``msg["reasoning"]`` when *surface_persisted_reasoning_flag* is True and the
dispatcher returned non-empty text. Strips ``_provider_content``
unconditionally the field is internal and never read by either UI.
The interactive ``_build_history`` surface DOES NOT call this
helper; it builds new entry dicts from scratch and calls
:func:`extract_reasoning_text_from_provider_content` directly per
assistant message, stamping ``entry["reasoning"]`` inline. The two
surfaces converge on the same dispatcher; only the mutation shape
differs.
Pure transform. Safe to call from ``asyncio.to_thread``.
"""
for msg in messages:
if msg.get("role") != "assistant":
continue
provider_content = msg.get("_provider_content")
# Always strip the internal lane before the wire payload leaves
# the helper, even when surface_persisted_reasoning_flag is False or the
# field is empty/missing. The strip is the contract; reasoning
# surfacing is conditional on top of it.
if "_provider_content" in msg:
del msg["_provider_content"]
if not surface_persisted_reasoning_flag:
continue
text = extract_reasoning_text_from_provider_content(provider_content)
if text:
msg["reasoning"] = text
def decorate_history_messages(
messages: list[dict[str, Any]],
verdicts_by_call_id: dict[str, dict[str, Any]],
+20 -13
View File
@@ -910,7 +910,17 @@ class IntentJudge:
self._context_window = context_window
self._rule_registry = rule_registry
# Resolve judge model via ModelRegistry alias, falling back to session
# Resolve judge model via ModelRegistry alias, otherwise self-
# consistency on the session model. ``judge.model`` is alias-only
# — same contract as ``coordinator.model_alias`` /
# ``model.plan_alias`` / ``model.task_alias``. A non-alias value
# used to be accepted as a raw model id pinned onto the session
# provider, but that path silently broke whenever the session
# provider didn't speak that model id (e.g. coordinator on
# Anthropic, ``judge.model = "gpt-5-mini"`` → every judge call
# returned ``llm_fallback``). Operators register an alias
# instead; an unknown value here logs a warning and inherits the
# session model.
resolved = False
if config.model and model_registry is not None:
try:
@@ -928,18 +938,15 @@ class IntentJudge:
except Exception:
log.debug("Model alias resolution failed for %r, falling back", config.model)
if not resolved and config.model:
# Model name override with session provider
self._provider = session_provider
self._client_factory_args = self._extract_client_config(
session_client,
session_provider.provider_name,
)
self._model = config.model
caps = self._provider.get_capabilities(self._model)
self._judge_context_window = caps.context_window
elif not resolved:
# Self-consistency: same model as session
if not resolved:
if config.model:
log.warning(
"judge.model=%r is not a registered alias — falling back to "
"session model %r. Register the model in the Models tab and "
"set judge.model to its alias.",
config.model,
session_model,
)
self._provider = session_provider
self._client_factory_args = self._extract_client_config(
session_client,

Some files were not shown because too many files have changed in this diff Show More