Wake path:
- Denial metacog nudge moves to the tool channel so it drains with the
denied tool batch instead of the next user-message seam.
- wake_workstream_if_pending: shared wake gate for watch fires on
already-idle workstreams (no IDLE transition for the watcher to
observe), wired as wake_fn at every set_watch_runner site via the
shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT
— after eviction+restore an id-keyed manager lookup would miss).
- session_worker exit backstop re-runs the wake gate the moment worker
ownership clears: IDLE fans out on the worker thread, so
transition-time wakes always landed on the reuse path and no-op'd
(the coordinator idle_children strand).
- deliver_wake_nudge_from_queue contains GenerationCancelled — it is
the wake worker's run() closure and only Exception is caught
downstream.
Watch delivery:
- Terminal fires that cannot reach their workstream are HELD and
redelivered on min(interval, 60s) without re-running the command,
bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own
max_polls across cycles; the poll charge commits durably at hold
time so restarts stay budget-bounded.
- Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES
cap, presence-only re-check under the lock, detection-only stall
alerts (reclaiming a wedged admission would trade capped degradation
for total poll-pool collapse).
- Permanent-vs-transient restore taxonomy: corrupt persona stamp and
genuinely-missing history (confirmed by a raising storage probe —
the resume loader swallows read blips into []) deactivate the watch
immediately; everything else holds and retries.
- Cancel-race defense: delivery paths re-check is_watch_active before
stashing/dispatching, cancel paths write the row BEFORE
forget_terminal_dispatched, the HTTP cancel endpoint clears runner
state, and a per-tick sweep bounds the residual stash-after-clear
interleaving to one check_interval.
- Abandon/exhaustion commits are write-then-clear so storage that can
read but not write retries the row write instead of re-running the
command every cycle; the fresh-fire unrestorable path stashes before
its deactivation write for the same reason.
Registry follows identity:
- The dispatch registry is keyed by _ws_id at registration time; every
rebind now moves it: non-fork resume() and /new go through
_follow_watch_registration (new key live before the old is removed,
never stealing a registration another live session holds), removals
are owner-checked so tearing down a watch-restore shell or a
resumed-away session cannot unregister a live pane, the restore
shell yields to a registration that appears mid-restore, CLI
--resume registers after the successful resume, and both the open
path and the detail-GET lazy rehydrate wire the registration.
Teardown gating and backpressure honesty:
- cleanup_session_ui marks ws._closed FIRST under ws._lock — every
teardown path (close, close_idle, evict, delete, discard) funnels
through it — and session_worker.send re-checks under the same lock,
so a wake can never spawn a worker on a torn-down workstream.
- Create responses carry initial_message_status when the initial
message could not be delivered (queue_full / refused_closed) instead
of reading as success; staged attachments survive for the retry;
/send surfaces a closed workstream as 404 rather than queue_full.
Docs/spec: OpenAPI artifacts regenerated; api-reference documents the
new create-response field; TS SDK type extended.
Tests: ~30 new pins (cancel races, budget durability across restarts,
owner-checked registry moves, teardown gating, stall alerts,
backpressure surfaces, wait_until final re-check); wide subsystem
sweep green (2353 passed).
Review feedback (PR #800): the URL lane hard-capped fetched bodies at
10 MB before kind resolution, making the 32 MiB pdf cap unreachable for
URL targets while path targets honored it. The flat pre-check is gone;
the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values())
- mirroring the path lane's stat pre-check - and the per-kind caps after
resolution stay authoritative.
Also drops a redundant function-local asyncio import in test_console.py.
fetch_with_ssrf_guard now streams the response under a max_bytes budget
(default 32 MiB, counted on decoded bytes so gzip cannot expand past it)
instead of buffering blind - an unbounded body previously filled memory
before any caller-side size cap could run. Redirect-hop bodies are no
longer read at all, and the realized response drops stale wire-framing
headers (content-encoding/content-length/transfer-encoding) that no
longer describe the decoded content it carries.
Preview blob ids are salted out of the model-visible attachment
namespace (sha256("preview:" + body)): uploads use bare sha256(body)
and save_attachment freezes kind at first insert, so a byte-identical
preview/upload pair would otherwise share a row - whichever landed
second inherited the other's kind, silently hiding an upload from model
context or materializing preview bytes into a tool turn.
turnstone's primary audience self-hosts it beside other lab services —
a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev
node on the local network is the operator using their own network, not
an attack. The hard SSRF refusal made those targets unreachable.
New runtime setting tools.allow_private_network (settings registry,
default off, rendered in console Settings → Tools; hot — read per tool
call, no restart). When enabled, a call NAMING a private address
becomes approvable: the approval prompt tags it "(private network)" so
the operator approves it as what it is, and the human gate stays.
The redirect side-door stays closed either way: a PUBLIC target that
302s into private address space is refused regardless of the opt-in —
that address never appeared on the approval card, so it is never
fetched. Only a chain whose approved origin was itself private skips
hop screening (its redirects are the operator's own network).
Refusals now teach the knob (mirrors the oidc opt-in hint): the error
names tools.allow_private_network and where to enable it. Surfaces
without a ConfigStore (bare CLI, eval) stay strict — there is no admin
surface to have opted in on.
Four follow-ups to the preview pane:
- Probe-mode preflight: the pane preflights src-loaded kinds with
GET ?probe=1 (204, real hardening headers, no body) instead of HEAD —
the console reverse proxy forwards HEAD as a full GET, so the old
preflight dragged the whole blob across the node→console hop twice.
Ownership gate + renderable-type check still run on probes.
- Legacy-charset text: table/text/markdown now transcode to UTF-8 at
store time (declared charset → UTF-8 → cp1252-replace ladder), same
model the web kind already used. The ladder applies only when the
text kind was DECLARED (MIME/extension/override); the bare no-hint
fallback stays strict UTF-8 and NUL bytes still hard-reject, so
binary rejection is unchanged.
- Remote assets default OFF: previewed pages are now served under
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src
data:; font-src data:" — they render with inline styling but cannot
contact their origin site (no viewer IP/traffic disclosure). A
per-pane "Load remote images & styles" checkbox (web previews only,
sticky, not persisted) reloads with ?assets=1 for the permissive
bare-sandbox mode.
- Markdown vendor parity: preview markdown now runs renderer.js's
postRenderMarkdown (hljs token coloring + lazy mermaid diagrams)
like the conversation pane, with preview-scoped code-block/KaTeX
chrome (the conversation theme is .msg.assistant-scoped).
Tests: probe/assets HTTP + policy coverage, charset ladder units +
stored-bytes round-trip, JS static guards for the probe form, the
default-off toggle, and the post-pass; headless-chrome harness grew to
41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code
render). Full suite green.
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.
Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
URL, a file path, or attachment:<id> to bytes; classifies into
web/pdf/image/table/text/markdown (magic bytes > MIME hint >
extension > UTF-8 fallback, legacy-charset pages transcoded); caps
size per kind; persists content-addressed with kind="preview" —
refcounted and GC'd with the workstream, skipped by trajectory
reconstruction so preview bytes can never materialize onto the wire.
URL targets gate like web_fetch (network egress); paths/attachments
run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
SSRF-screens every hop BEFORE requesting it (follow_redirects=True
checked nothing between hops); adopted by both open_preview and
web_fetch. URL userinfo is stripped before the descriptor or the
stored bytes see it; <base href> is injected doctype-safely so
relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
ONE shape on every boundary: the live tool_result SSE event, the
conversations.meta column, and the /history projection. Cancelled
batches commit an already-announced preview (blob + meta) instead of
stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
gate as /content) serves the STORED type with per-MIME hardening:
bare CSP sandbox for text/html (renderable, scriptless, opaque
origin), no CSP for application/pdf (Chromium's viewer refuses
sandboxed contexts), full default-src 'none' otherwise; filenames
fold to latin-1-safe ASCII. The console /node proxy now forwards
CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
the query (they were read and discarded on every load).
Frontend
- New "preview" pane type registered in the shared shell (server +
console): openPaneBeside placement, per-kind renderers — fully
sandboxed iframe for pages, browser PDF viewer, sortable tables
(CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
text — plus back/forward history with arrow keys, reload persistence
via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
preview chip (the reopen + replay affordance); live results auto-open
the pane only while the originating pane holds focus.
Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).
The discovered-endpoint wrapper converted every OAuthSSRFError to a
bare OIDCError, so a private-resolving endpoint or trusted host got the
non-public message without the allow_private_network remediation even
though the same knob fixes it. Hoist the hint into a module constant
and append it in both wrappers.
Also name "unspecified" in the refused-even-with-opt-in message so
0.0.0.0/:: rejections read unambiguously.
The SSRF guard on OIDC endpoint URLs hard-refused any hostname
resolving to a non-public address, which made it impossible to use a
self-hosted IdP (Keycloak, Authentik, Dex) on an internal network —
even though the login-flow issuer is operator-configured, i.e. trusted
input.
Add [oidc] allow_private_network in config.toml (or
TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the
issuer and its discovered endpoints may resolve to private-range,
unique-local, CGNAT, and loopback addresses. Link-local, multicast,
unspecified, and reserved ranges stay refused regardless — cloud
metadata services live on link-local and no legitimate IdP does. The
HTTPS requirement and same-origin endpoint checks are unchanged.
The private-address refusal now raises OAuthSSRFPrivateAddressError,
and the OIDC wrapper appends the remediation hint to the error message
so the failure is self-service. mcp_oauth call sites — where endpoint
URLs come from untrusted remote-server metadata — do not get the knob
and keep the strict public-address rule.
RFC 3986 schemes are case-insensitive, so POSTGRESQL+PSYCOPG2:// or
HTTPS://user:pass@host in tool output leaked the password past the
case-sensitive scheme alternation. Compile with IGNORECASE on both
sides of the FE/backend mirror; the structural userinfo requirement
is unchanged. Uppercase-scheme cases added to both test suites.
Connection-string redaction (the output_guard pattern and its frontend
mirror) only enumerated bare dialects plus +psycopg, so SQLAlchemy
dialect+driver URLs — postgresql+psycopg2://, postgresql+asyncpg://,
mysql+pymysql:// — leaked the password through every redaction surface.
The scheme now takes an optional +suffix instead of enumerating drivers.
redactCredentials() also gains a single early-exit prefilter scan ahead
of its sixteen replace passes, for plain-log tool output on card render.
The prefilter is documented and pinned as a superset of the pattern
set's required substrings, so a miss is provably a no-op: new smoke
cases assert bare sk-/AKIA/Bearer credentials with no '=', quote or '@'
anywhere in the text still redact, alongside the fast-path no-op and
the driver-scheme URLs on both sides of the mirror.
- Proven: name supervisory control's controllability and nonblocking
conditions as the ancestors of gate-early-on-irreversibles and the
always-enabled escalation required behind a learned veto; cite TCSEC
covert-channel analysis (NCSC-TG-030) for the verdict channel.
- Asserted: add the narrow-only rule's influence-side twin (verdict
payloads to the plant selected, never generated).
- New caveat paragraph: a denial is free only in the authority lattice;
in the dynamics it is an input (selection + targeted-liveness
channels), so a learned veto needs a nonblocking escape it cannot
disable, verdict payloads are selected rather than generated with the
symbols/tokens/language thresholds bounding the alphabet, and the
strongest form dissolves the verdict into scheduling over
deterministic checks.
Review: the wrapped ValueError is echoed in 503 bodies, and arbitrary
SDK exception text can embed filesystem paths. Echo the exception type
only; log the full exception with traceback at the raise site.
SDK client construction can fail on environment problems the config
never sees (e.g. httpx resolving a certifi CA path deleted by a venv
rebuild). Those escaped as bare exceptions and turned every workstream
open/create into an opaque 500; re-type them as ValueError in
ModelRegistry.get_client so routes answer 503 with the message and the
alias.
to_tsvector was computed inline over full row content, so one row
whose tsvector exceeds PostgreSQL's 1MB limit aborted every
search_history scan. Cap the FTS input at 250K chars (worst-case
tsvector expansion stays under the limit; giant rows remain findable
by their head). The ILIKE fallback also never ran on postgres: the
failed statement leaves the autobegun transaction aborted, so roll it
back before falling back.
Address review on #794. The prior gate deepcopied every agent tool, then compared, then discarded the copy on a no-op render; and its comment framed the equality case as 'no personas' when it also covers an idempotent re-render of unchanged aliases/personas. Compute the target model/persona descriptions from the current (read-only) schema first and only deepcopy when one differs — so a no-op render is genuinely allocation-free, not just fork-free. Behaviour is unchanged: identity preserved when nothing differs, stale text still cleared on reload.
_render_agent_tool_descriptions rebuilt self._tools and reassigned it on every session init, deep-copying task_agent even with no model aliases and no personas to inject — the single-model CLI case the docstring says is skipped. This regressed after the persona-discoverability change removed the early 'if self._registry is None: return' guard, breaking the session._tools is INTERACTIVE_TOOLS invariant (test_session_without_mcp).
Gate the reassignment on whether a description actually changed: keep the original tool object when the render is a no-op, fork self._tools only when something was injected. Restores the shared-constant invariant, makes repeated renders idempotent, and preserves clear-stale-on-reload (an emptied registry still takes the changed path).
Add .github/FUNDING.yml to enable the native GitHub Sponsor button, plus a Sponsor badge and a Support section in the README. Primary CTA is GitHub Sponsors (eous); PayPal (paypal.me/eousphoros) is offered as a one-off fallback.
Review feedback on #792: the "not found or disabled" branch
interpolated the raw input unquoted, so the whitespace-only and
trailing-space inputs the forgiving lookup explicitly handles rendered
invisibly in CLI output and logs. Use {name!r} like the other two
resolver errors already do.
Coordinators and interactive agents had no way to enumerate valid
persona names: task_agent / spawn_workstream / spawn_batch described
`persona=` but nothing listed what it accepts, and resolution was an
exact case-sensitive slug match - users reaching for the display name
or a case variant got an unexplained failure.
- Inject the live persona catalog (enabled, interactive-kind; children
and sub-agents are always interactive) into the `persona` parameter
description of task_agent / spawn_workstream / spawn_batch, riding
the same render path as the model-alias injection. Rebuilt from the
pristine TOOLS base every render, so repeated renders are idempotent
and archived personas drop out instead of lingering. Entries carry
name + default marker + <=96-char description; names-only past 25
personas. spawn_batch's persona property is nested per-child under
children.items.properties (located via _persona_property, null-safe
against name-colliding MCP tools). Storage-less sessions keep the
base text: the render runs at session construction, so it gates on
is_storage_initialized() rather than get_storage(), which would
auto-init SQLite as a side effect.
- resolve_persona_for_kind - the ONE shared rule behind the HTTP
create handler, CLI --persona, the coordinator spawn precheck, and
task_agent prep - is now forgiving: exact slug, then the lowercased
input (created names are regex-enforced lowercase slugs), then a
case-insensitive display-name match accepted only when unique among
the kind's enabled personas. Duplicates refuse loudly naming the
candidate slugs; a same-label persona of another kind neither blocks
nor wins (the label the caller saw came from a kind-filtered
surface); whitespace-only input never matches blank display names
(display_name defaults to ""). Every failure now enumerates the
kind's valid names, so a stale injected list or a typo self-corrects
on the next attempt.
- The canonical slug is stamped everywhere: task_agent prep rewrites
its arg from the resolved snapshot, _validate_child_persona returns
(canonical, error) and both spawn call sites adopt it - approval
chrome, the wire, and workstream_config never carry a forgiven
variant.
- Create-persona shelf: label hint under Name explaining agents and
the CLI launch the persona by this name (case-insensitive) and the
display name is only a list label. docs/personas.md gains a "How
agents discover personas" section and drops the stale claim that
task_agent has no persona parameter.
Tests: resolver unit suite (case/display/ambiguity/cross-kind/
whitespace/disabled/storage-failure) + guards for injection content
and ordering, idempotent re-render, archive drop, the 25-persona
prose cutoff, coordinator-kind exclusion, and canonical stamping
through spawn_workstream / spawn_batch / task_agent.
- _coordinator_tenant_check and _coord_attachment_owner resolved storage from
the global registry (get_workstream_row / for_request without a storage arg),
which can evaluate the project-tenancy decision against a different or
auto-initialised backend and fail OPEN on a missing project row. Use
request.app.state.auth_storage explicitly, matching cluster_ws_detail and
_resolve_coordinator_or_404; fail closed (404) when it is unset.
- reject_unassignable_scopes now derives its allowed-scope error message from
ASSIGNABLE_SCOPES so validation and the message can't drift.
Workstreams attached to a private project were visible -- including their
conversation content -- to holders of admin.cluster.inspect / admin.coordinator
(both default builtin-admin permissions), defeating the project's confidentiality
boundary. Enforce that a private project's resources are visible only to people
IN the project (owner, workstream creator, or an explicit member), even for admins.
Surfaces closed:
- WorkstreamProjectVisibility bypass narrowed to service scope only (node->console
machine plumbing, re-filtered per-user at the console edge). No human principal
bypasses; admin.cluster.inspect gates the inspect surfaces, not tenancy. This
flows to /dashboard, session listings, the attachment row-gate, cluster_workstreams,
cluster_node_detail, and cluster_snapshot/SSE.
- cluster_ws_detail 404-masks a workstream in a private project the caller can't
see; cluster_ws_live_bulk routes such ids to the denied list (no private-project
oracle).
- Coordinator operator verbs (history/export/detail/send/approve/set_title/open/
children/tasks/attachments) now enforce project tenancy: _coordinator_tenant_check
on coord_endpoint_config, the gate in _resolve_coordinator_or_404 (children/tasks),
the tenant_check now run in make_open_handler before rehydrate, and a
project-visibility check in _coord_attachment_owner. admin.coordinator gates the
surface cluster-wide, but a non-member is 404-masked. The tenant-check mirrors the
manager-first + coordinator-kind ladder so kind-isolation is preserved.
- service scope is no longer user-assignable: admin_create_token and both
turnstone-admin CLI mint paths reject it via reject_unassignable_scopes, so an
admin.users holder cannot self-mint a service token and restore the bypass. Service
scope is minted only by ServiceTokenManager / the JWT secret.
- The events/global node proxy (service-elevated cross-tenant firehose) is gated on
admin.cluster.inspect so a plain authenticated user cannot reach it through the
console proxy.
Updates the OpenAPI description, the row-gate/tenancy-filter docstrings, and adds
tests for every surface (visibility predicate + cluster detail/bulk + coordinator
history/export/children/open/attachments + events/global proxy + scope-mint
rejection); inverts the tests that pinned the old admin-bypass contract.
* fix(mcp): route static transport lifecycles through per-server owner tasks
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).
Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).
Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.
Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.
* fix(mcp): route pool transport lifecycles through per-entry owner tasks
Completes the owner-task migration started for the static path: the
oauth_user pool path had the same latent anyio cancel-scope exposure
(host-task-bound scopes entered by short-lived connect tasks; a scope
cancelled after its host finished re-delivers cancellation via
call_soon forever - the 100%-CPU zombie), previously covered only by
the disarm backstop.
Each (user, server) pool entry's transport + ClientSession cms are now
entered, parked, and exited by ONE long-lived owner task
(_pool_transport_owner). The caller keeps building client_kwargs (the
per-user bearer and, when an auth-capture carrier is active, the
httpx_client_factory response hook) so 401/WWW-Authenticate capture
semantics are unchanged. Teardown is the shared one-cancel close
protocol (_teardown_pool_entry: signal before first await, graceful
grace, at most ONE cancel), used by the connect stale-guard, idle/LRU
eviction, and shutdown (parallel signal-then-reap). Unrequested owner
death evicts the session but keeps the entry and its discovered
catalog, matching the existing evict-session-keep-entry semantics the
auth_401 retry relies on.
Discovery still runs in the connecting caller while the transport is
hosted by the owner, so a transport collapse mid-discovery (e.g. the
SDK tearing its task group down on an upstream 401) cancels the OWNER,
not the caller - a bare await on the response stream would hang until
the 30s phase timeout. _await_pool_discovery races each discovery
await against owner completion and converts owner death into a prompt
ConnectionError (the owner is never cancelled there; teardown owns its
lifecycle). Carrier-first failure classification preserves auth_401
semantics for captured 401s.
With no cross-task stack closes left, _safe_close_stack and
_safe_teardown_on_connect_failure are deleted (zero callers).
Tests: new tests/test_mcp_pool_owner.py pins the pool close protocol
(graceful event-before-await close, exactly-one-cancel escalation,
owner-death eviction retaining entry+catalog, caller-cancel-mid-connect
cm-exit guarantee, factory-present-iff-capture, and the
owner-death-during-discovery fast-fail). 1010 mcp tests and the full
8471-test suite green, including the historical cross-task-anyio
sentinel test_integration_pool_reuse_401_refresh_and_retry_succeeds;
ruff+mypy clean; zero destroyed-task warnings.
* fix(mcp): harden disarm-sweep loop guard and owner BaseException arm
Review follow-ups on the owner-task migration:
- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
instead of trusting callers: it returns without walking (and without
advancing the rate-limit clock) unless the currently running loop IS
self._loop. A suppressed close can fire before start() or after
shutdown(), where the walk would be wasted at best and a cross-thread
reach at worst.
- The transport owner's BaseException arm now re-raises non-Exception,
non-group escapees (KeyboardInterrupt, SystemExit) after delivering
them to the readiness future - failure delivery is the arm's job;
swallowing an interpreter-level exit was not.
* fix(mcp): extend owner-death discovery fast-fail to the static path
The static connect path had the same exposure the pool's discovery race
closed: discovery runs in the connecting caller while the transport is
hosted by the owner task, so a transport collapse mid-discovery cancels
the OWNER and the caller's bare await on the response stream hung until
the caller-side attempt timeout (~45s) instead of failing promptly.
_await_pool_discovery is renamed to _await_owner_discovery (it is now
path-neutral) and wired into _connect_one_locked's four discovery
awaits. The helper also converts a discovery future that completes
CANCELLED without the race's own reap (an SDK-internal cancellation
shape) into the same ConnectionError, instead of leaking a bare
CancelledError the caller would misread as its own cancellation.
The pool transport owner's BaseException arm gains the same refinement
the static owner received in review: interpreter-level exits
(KeyboardInterrupt, SystemExit) re-raise after delivery to the
readiness future instead of being swallowed.
Tests: static owner-death-during-discovery fast-fail (<1s vs the ~45s
hang), and a direct pin on the cancelled-discovery-future conversion.
* fix(mcp): replace owner BaseException arm with targeted catch + finally delivery
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.
Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.
* fix(mcp): mirror targeted-catch + finally delivery in the pool owner
Same shape the static owner received in review: the failure arm catches
only (BaseExceptionGroup, Exception), and waiter delivery for anything
else moves to a finally that resolves the readiness future with a clean
ConnectionError before the task unwinds - interpreter exits and
BaseException-derived library escapes propagate exactly once, uncaught,
and the waiter can never be left hanging.
* test(mcp): narrow the escape test's waiter catch to explicit types
* test(mcp): narrow discovery-race waiter catches to explicit types
* refactor(mcp): make reap/synchronization awaits explicit to analyzers
Full-absorb reaps (cancel-then-drain of a future whose outcome is
deliberately consumed) become `await asyncio.gather(x,
return_exceptions=True)` - one line, self-describing, and in the
owner-died discovery reap it is also a small semantic improvement: a
caller cancellation arriving during the reap now propagates instead of
being masked by the ConnectionError. Bare synchronization awaits and
selective suppress blocks in tests keep their raise-through semantics
via throwaway assignment. Applied uniformly across the owner-task
test files, including sites introduced by the static-path PR.
The owner's failure arm now catches only (BaseExceptionGroup, Exception);
waiter delivery for everything else moves to a finally that resolves the
readiness future with a clean transport-failure ConnectionError before
the task unwinds. Interpreter exits and BaseException-derived library
control-flow escapes propagate from the owner exactly once, uncaught -
and the waiter can never be left hanging on an unresolved future (the
initial _connect_all connect has no outer bound). For SystemExit /
KeyboardInterrupt asyncio additionally stops the loop right after, so
the delivery is load-bearing for the non-exit BaseException shapes and
free for the exits.
Pinned by a test driving a BaseException-derived escape through the
owner: the waiter resolves promptly with ConnectionError while the
escape propagates unswallowed.
Review follow-ups on the owner-task migration:
- _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement
instead of trusting callers: it returns without walking (and without
advancing the rate-limit clock) unless the currently running loop IS
self._loop. A suppressed close can fire before start() or after
shutdown(), where the walk would be wasted at best and a cross-thread
reach at worst.
- The transport owner's BaseException arm now re-raises non-Exception,
non-group escapees (KeyboardInterrupt, SystemExit) after delivering
them to the readiness future - failure delivery is the arm's job;
swallowing an interpreter-level exit was not.
A crash-looping MCP server drove the mcp-loop thread to a sustained,
climbing 100%+ CPU spin. Root cause: anyio cancel scopes are
host-task-bound, and the static path entered the SDK's transport /
ClientSession task-group scopes from short-lived connect tasks (every
health tick is a new task since #768). Once such a scope was cancelled
after its host task had finished - by anyio's task_done when a
transport child died with the server, or by ClientSession.__aexit__
during a cross-task teardown - CancelScope._deliver_cancellation could
never make progress (task.cancel() on a done task is a no-op) and
re-armed itself via call_soon every loop iteration, forever: ~900k
callbacks/s per zombie scope, one more per flap cycle (verified against
anyio 4.14.1; no upstream fix exists as of that release).
Fix: each static server's transport + session cms are now entered,
parked, and exited by ONE long-lived owner task
(_static_transport_owner), so scopes always have a live host and always
exit in the task that entered them. Teardown follows a one-cancel close
protocol (signal the close event before the first await, graceful
grace, then at most ONE cancel - never a second, which would abandon a
scope exit mid-flight). Connect timeouts now cancel only the waiting
caller; connect failures are delivered through a readiness future;
unrequested owner death (server died under a live session) evicts the
session immediately via a done-callback instead of waiting for the next
liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop
(_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not
yet migrated (the oauth_user pool keeps the old cross-task-close shape;
follow-up).
Also fixed: BaseExceptionGroup (BaseException-derived, as raised by
anyio task groups wrapping a stray CancelledError, e.g. an
accept-then-RST server) escaped `except Exception` in _connect_all and
killed it before the health/sweep loops were created - silently
disabling all autonomous recovery. Handled there and in the
health/sweep/eviction loops and the reconnect/refresh callers.
Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one
armed scope per cycle) to 0.3% flat with zero armed scopes; the RST
repro now leaves both background loops alive (previously both silently
dead). New tests: owner-lifecycle + close-protocol units (incl. an
exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression,
a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke
test (real FastMCP subprocess, skips on environment gaps) asserting
zero armed scopes, exactly one live owner, and a post-recovery tool
call. Full 8470-test suite green; ruff+mypy clean.
The bounded key prefix pattern (api_key=/secret_key= etc.) avoids
monkey/turkey false positives, but compound keys like secret_access_key
and aws_secret_access_key only matched on the access_key= suffix, leaking
the secret_ / aws_secret_ prefix. Added these as explicit alternations.
Also added bearer_token and secret_token to the token prefix list.
Bare alternatives (|token, |key) for standalone token= and key= assignments
were re-added. A negative lookbehind (?<![a-zA-Z0-9_]) prevents matching
word-suffixed identifiers like monkey=, turkey=, mytoken=, over_tokenized=.
Also applied the same protection to _RE_QUERY_CRED which had a bare |token
alternative without boundary protection.
- Replace unbounded [a-zA-Z0-9_]*key= prefix with specific credential
key suffix alternation to avoid false matches on monkey=, turkey=, etc.
Same for *token= (access_token=/auth_token= but not over_tokenized=).
- Hoist redactCredentials() out of per-line diff render loop in
buildConvCmd - 1 call on full text instead of N calls per line,
eliminating ~2800 regex passes worst-case.
- Remove bare 'key' from _RE_QUERY_CRED alternation (too aggressive).
- Add x-api-key / x_api_key to JSON secret key lists in both Python
and JS.
- Add re.IGNORECASE to configurable-mode credential_bearer pattern.
- Fix annotation dedup guard in _check_credentials (was checking flag
name against annotation prose list, always-true dead code).
Address review findings on the client-side credential redactor and mirror
each fix into the backend output guard so both surfaces censor identically:
- Detect and redact single-quoted JSON secrets such as
{'Authorization': 'Bearer ...'} (Python dict reprs / JS object literals),
which the double-quote-only pattern silently bypassed on both sides.
- Cover mongodb+srv://, rediss:// and amqps:// connection strings.
- Match the Bearer auth scheme case-insensitively (RFC 7235).
- Redact prefixed key/token assignments (api_key=, secret_key=,
access_token=) as a whole rather than chewing the tail into a garbled
"api_[REDACTED:api_key]", while still covering bare key=/token=. A word
boundary was rejected because it would drop coverage for <prefix>_key=.
- Remove the redundant |Authorization alternative (covered by /i) and swap
the manual value-slicing helper for a capture-group substitution.
The backend edits touch both detection sites and both redaction pipelines,
so single-quoted secrets are flagged (and therefore sanitized), not merely
rewritten. Adds JS runtime-smoke and backend unit coverage for every case.
New shared ES6+ module (redact_credentials.js) provides comprehensive
visual credential censorship matching the backend output guard patterns:
- PEM private key blocks, connection strings, Bearer tokens
- OpenAI / GitHub / AWS / Google API key formats
- Query-string and JSON-style credential values
- JSON secret keys (api_key, password, token, authorization, etc.)
- ENV secret lines (SECRET_KEY=, DATABASE_URL=, etc.)
Integrated into both frontend surfaces:
- interactive.js: replaces legacy minimal _redactApiKeys function
- conversation.js::buildConvResult (shared substrate, used by coordinator)
- coordinator.js::renderToolOutput fallback paths
Backend parity: added 'authorization' to the JSON secret regex in
output_guard.py so the output guard flags and redacts Authorization
headers in JSON tool output.
Tests: ported the runtime smoke test from the removed _redactApiKeys
to import the new module directly; added redact_credentials.js to the
var-free and const-reassign guard bundles.
`tool_args_preview` feeds `stream.tool_args_malformed` (WARNING) and
`wire.tool_args_legalized` (DEBUG), and tool arguments are model/user
controlled — they can carry secrets (a token in a bash command, a password in a
connection string) or raw CR/LF that break log lines. Route the preview through
`output_guard.redact_credentials` over the full value first (before the 120-char
cap, so a secret straddling the cut isn't half-shown past the pattern's reach),
then collapse every control char to a space, mirroring `audit._scrub_string`.
Addresses the PR review comments.
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.
Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.
Convert `lowering.py` to structlog so the new pass emits structured events.
The pane/workstream accelerators only worked on macOS. They were bound to
Ctrl, which on Windows/Linux IS the browser's own accelerator: Ctrl+T,
Ctrl+W and Ctrl+1-9 were swallowed by the browser (new tab / close tab /
switch tab) and never reached the page. macOS browsers own Cmd instead, so
Ctrl was free there and everything appeared to work.
On top of that the shortcuts were declared in three places that had drifted
apart — the "?" overlay, each app.js keydown handler, and the tab-menu
badges in shell.js. The console fell to convTabMenu's node-proxy fallback
lane, which dropped every shortcut badge (and Fork), so its tab menu showed
no accelerators and Ctrl+W there just closed the browser tab.
Choose the modifier per platform (Ctrl on macOS, Alt on Windows/Linux) and
make shell.js the single source of truth for the per-pane accelerators: a
stable accel registry drives both the platform-aware badge and one shared
keydown handler that invokes the ACTIVE pane's own menu item, so a badge
can't advertise a chord the handler ignores and each surface contributes
only what it supports (the console omits Fork; it has no fork surface yet).
Each surface's app.js keeps only its global accels (new / switch /
dashboard); the console regains switch + dashboard to match. Mod+W now
uniformly means Close pane (drop the tab, session keeps running), matching
its badge and the universal Ctrl+W convention — previously the standalone's
Ctrl+W stopped the session. The previously-dead "Refresh title / Ctrl+Shift+R"
is wired, and Ctrl+T / Ctrl+D yield to text editing while a field is focused
(macOS transpose / delete-forward).
* Fix send button stuck disabled by pruning orphaned approval cycles
When a DOM wipe (clear_ui / replay_truncated / replaceChildren)
detaches approval card elements while an approve_request event is
processed between the wipe and refetch-restore, the matching
approval_resolved may never arrive. The orphaned cycle entry in
approvalCycles keeps pendingApproval=true and the send button
disabled forever.
The fix adds a pruning pass at the top of _syncApprovalState():
cycles whose blockEls are all .isConnected === false are deleted
from the Map. This runs on every register/resolve/rebuild so
orphans are cleaned up promptly.
Also fixes an ordering bug in showInlineToolBlock discovered
during review: the block element was appended to the DOM after
_registerApprovalCycle, so the new isConnected prune would kill
the just-registered cycle before it took effect.
* Fix comment inaccuracy in showInlineToolBlock append-before-register guard
The comment said 'blockEls.every(el => el.isConnected)' but the
actual prune check is '!blockEls.some(el => el.isConnected)' -
no block elements are connected, not every element.
The wire-payload golden matrix had no anthropic-compatible coverage —
both AnthropicProvider rows are the native lane (compat=False), so the
distinct compat wire shape (reasoning control in
extra_body.chat_template_kwargs, never the native thinking param) was
unfrozen. Add the compat lane across all eight representative fixtures
with a manual-mode capability (the lane has no static table, so caps
ride in as a model definition would supply them) and reasoning_effort=
high: every golden now pins {enable_thinking: true, reasoning_effort:
high} in chat_template_kwargs, asserts the native thinking param is
absent, and preserves temperature (no forced 1.0). _capture gains an
optional caps override to support the no-static-table lane.
Copilot review caught the native Anthropic adaptive ladders (sonnet-5,
fable-5, opus-4.8/4.7/4.6) labeling the none position 'None — sends
adaptive' — raw mode vocabulary the plain-language rule exists to
prevent. But treating the 'adaptive' token identically to the local
lane's 'on' would still misframe it: on these models thinking is always
on and an effort level rides output_config on every graded position
(verified on the wire), so 'thinking stays on' is true everywhere and
not what distinguishes none. The none position uniquely means no effort
is pinned — the model self-regulates it — so it now reads 'None — model
sets effort'. The local adaptive lane's bare toggle keeps 'thinking
stays on' (no effort lever exists there).
Aliased knob positions were labeled after the lowest sibling sharing
their wire token — a toggle-only model rendered 'Max (= minimal)',
implying a minimal-effort downgrade the wire doesn't contain, and with
declared values 'High (= minimal)' while the wire carries high. Each
position now states its delivered level: exact matches stay plain
('Max'), snapped positions say 'Low — sends high', the adaptive none
position warns 'thinking stays on', budget detail stays in the
tooltip. Effort-param placeholder corrected to the real graded keys
(reasoning_effort / reasoning).
Local lanes dropped the knob's graded value unless the operator declared
reasoning_effort_values (and, on the template channel, an effort key) —
picking Max sent a bare thinking toggle and the effort select
degenerated into seven positions that all meant 'on'. The user's
setting now always rides:
- openai-compatible: the flat reasoning_effort param carries the knob
verbatim (effort_passthrough on the lane default); declared values
still snap ordinally, and a declared effort_param still claims the
template channel and suppresses the flat param.
- anthropic-compatible: the graded value rides chat_template_kwargs
alongside the toggle whenever reasoning control is engaged — under
the operator's effort_param, else the conventional fallback key
(reasoning_effort); templates that don't reference the kwarg ignore
it. thinking_mode=none still injects nothing.
- Commercial lanes untouched: empty declared values still mean 'no
effort control' (o1-mini) and the ordinal snap is unchanged.
Golden writer now pins ensure_ascii=False: the baselines' literal em
dashes came from a hand edit (03f82521) the default-escaping writer
could never reproduce — regens no longer churn unrelated lines.
Local-lane model ids are operator-chosen strings (vLLM
--served-model-name), so a prefix collision with a cloud model id
inherited that model's sampling and effort contract: a box named
o3-distill silently lost temperature support, and one named
gpt-5.5-my-finetune was sent gpt-5.5's snapped reasoning_effort values
it never declared. Both surfaces of the lane now return plain defaults
(OPENAI_COMPAT_DEFAULT in _openai_common): the chat class directly, and
the responses pin via a compat-mode OpenAIResponsesProvider mirroring
AnthropicProvider(compat=True). Everything beyond the defaults is
declared by the operator on the model definition, matching the
anthropic-compatible lane and lookup_model_capabilities' documented
'no static table for local models' contract. The commercial openai
lane (Responses-only) is untouched.
Pre-split tests that reached commercial rows through the chat-class
OpenAIProvider alias now source them from lookup_openai_capabilities;
their subject (registry rows + shared gating helpers) is unchanged.
The except path for a malformed capabilities column appended the row
without the key, so clients had to null-check a field the happy path
guarantees. Initialize each entry with an empty ladder and let the try
block overwrite it — the response schema is stable per row.
Patrick Buckley