mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-15 16:32:26 -06:00
e60c19befd5e31376bb606cd380c3564ac4e27df
820 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
e60c19befd |
fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds
Wake path: - Denial metacog nudge moves to the tool channel so it drains with the denied tool batch instead of the next user-message seam. - wake_workstream_if_pending: shared wake gate for watch fires on already-idle workstreams (no IDLE transition for the watcher to observe), wired as wake_fn at every set_watch_runner site via the shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT — after eviction+restore an id-keyed manager lookup would miss). - session_worker exit backstop re-runs the wake gate the moment worker ownership clears: IDLE fans out on the worker thread, so transition-time wakes always landed on the reuse path and no-op'd (the coordinator idle_children strand). - deliver_wake_nudge_from_queue contains GenerationCancelled — it is the wake worker's run() closure and only Exception is caught downstream. Watch delivery: - Terminal fires that cannot reach their workstream are HELD and redelivered on min(interval, 60s) without re-running the command, bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own max_polls across cycles; the poll charge commits durably at hold time so restarts stay budget-bounded. - Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES cap, presence-only re-check under the lock, detection-only stall alerts (reclaiming a wedged admission would trade capped degradation for total poll-pool collapse). - Permanent-vs-transient restore taxonomy: corrupt persona stamp and genuinely-missing history (confirmed by a raising storage probe — the resume loader swallows read blips into []) deactivate the watch immediately; everything else holds and retries. - Cancel-race defense: delivery paths re-check is_watch_active before stashing/dispatching, cancel paths write the row BEFORE forget_terminal_dispatched, the HTTP cancel endpoint clears runner state, and a per-tick sweep bounds the residual stash-after-clear interleaving to one check_interval. - Abandon/exhaustion commits are write-then-clear so storage that can read but not write retries the row write instead of re-running the command every cycle; the fresh-fire unrestorable path stashes before its deactivation write for the same reason. Registry follows identity: - The dispatch registry is keyed by _ws_id at registration time; every rebind now moves it: non-fork resume() and /new go through _follow_watch_registration (new key live before the old is removed, never stealing a registration another live session holds), removals are owner-checked so tearing down a watch-restore shell or a resumed-away session cannot unregister a live pane, the restore shell yields to a registration that appears mid-restore, CLI --resume registers after the successful resume, and both the open path and the detail-GET lazy rehydrate wire the registration. Teardown gating and backpressure honesty: - cleanup_session_ui marks ws._closed FIRST under ws._lock — every teardown path (close, close_idle, evict, delete, discard) funnels through it — and session_worker.send re-checks under the same lock, so a wake can never spawn a worker on a torn-down workstream. - Create responses carry initial_message_status when the initial message could not be delivered (queue_full / refused_closed) instead of reading as success; staged attachments survive for the retry; /send surfaces a closed workstream as 404 rather than queue_full. Docs/spec: OpenAPI artifacts regenerated; api-reference documents the new create-response field; TS SDK type extended. Tests: ~30 new pins (cancel races, budget durability across restarts, owner-checked registry moves, teardown gating, stall alerts, backpressure surfaces, wait_until final re-check); wide subsystem sweep green (2353 passed). |
||
|
Patrick Buckley
|
bbe92faca1 |
fix(preview): fetch ceiling tracks the widest kind cap, not a flat 10 MB
Review feedback (PR #800): the URL lane hard-capped fetched bodies at 10 MB before kind resolution, making the 32 MiB pdf cap unreachable for URL targets while path targets honored it. The flat pre-check is gone; the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values()) - mirroring the path lane's stat pre-check - and the per-kind caps after resolution stay authoritative. Also drops a redundant function-local asyncio import in test_console.py. |
||
|
Patrick Buckley
|
29a4bbf876 |
fix(preview,web): stream guarded fetches under a byte budget; salt preview blob ids
fetch_with_ssrf_guard now streams the response under a max_bytes budget
(default 32 MiB, counted on decoded bytes so gzip cannot expand past it)
instead of buffering blind - an unbounded body previously filled memory
before any caller-side size cap could run. Redirect-hop bodies are no
longer read at all, and the realized response drops stale wire-framing
headers (content-encoding/content-length/transfer-encoding) that no
longer describe the decoded content it carries.
Preview blob ids are salted out of the model-visible attachment
namespace (sha256("preview:" + body)): uploads use bare sha256(body)
and save_attachment freezes kind at first insert, so a byte-identical
preview/upload pair would otherwise share a row - whichever landed
second inherited the other's kind, silently hiding an upload from model
context or materializing preview bytes into a tool turn.
|
||
|
Patrick Buckley
|
09abc9d199 |
feat(tools): allow_private_network opt-in for private-address fetch/preview
turnstone's primary audience self-hosts it beside other lab services — a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev node on the local network is the operator using their own network, not an attack. The hard SSRF refusal made those targets unreachable. New runtime setting tools.allow_private_network (settings registry, default off, rendered in console Settings → Tools; hot — read per tool call, no restart). When enabled, a call NAMING a private address becomes approvable: the approval prompt tags it "(private network)" so the operator approves it as what it is, and the human gate stays. The redirect side-door stays closed either way: a PUBLIC target that 302s into private address space is refused regardless of the opt-in — that address never appeared on the approval card, so it is never fetched. Only a chain whose approved origin was itself private skips hop screening (its redirects are the operator's own network). Refusals now teach the knob (mirrors the oidc opt-in hint): the error names tools.allow_private_network and where to enable it. Surfaces without a ConfigStore (bare CLI, eval) stay strict — there is no admin surface to have opted in on. |
||
|
Patrick Buckley
|
1e2ab91ec2 |
feat(preview): probe preflight, legacy charsets, remote-assets opt-in, md vendor parity
Four follow-ups to the preview pane: - Probe-mode preflight: the pane preflights src-loaded kinds with GET ?probe=1 (204, real hardening headers, no body) instead of HEAD — the console reverse proxy forwards HEAD as a full GET, so the old preflight dragged the whole blob across the node→console hop twice. Ownership gate + renderable-type check still run on probes. - Legacy-charset text: table/text/markdown now transcode to UTF-8 at store time (declared charset → UTF-8 → cp1252-replace ladder), same model the web kind already used. The ladder applies only when the text kind was DECLARED (MIME/extension/override); the bare no-hint fallback stays strict UTF-8 and NUL bytes still hard-reject, so binary rejection is unchanged. - Remote assets default OFF: previewed pages are now served under "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:" — they render with inline styling but cannot contact their origin site (no viewer IP/traffic disclosure). A per-pane "Load remote images & styles" checkbox (web previews only, sticky, not persisted) reloads with ?assets=1 for the permissive bare-sandbox mode. - Markdown vendor parity: preview markdown now runs renderer.js's postRenderMarkdown (hljs token coloring + lazy mermaid diagrams) like the conversation pane, with preview-scoped code-block/KaTeX chrome (the conversation theme is .msg.assistant-scoped). Tests: probe/assets HTTP + policy coverage, charset ladder units + stored-bytes round-trip, JS static guards for the probe form, the default-off toggle, and the post-pass; headless-chrome harness grew to 41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code render). Full suite green. |
||
|
Patrick Buckley
|
e010124008 |
feat(preview): rich preview pane + open_preview tool
Tool results only ever rendered as plain text in the transcript. This
adds the model-driven rich-preview lane every comparable surface has,
in turnstone's developer-tool idiom: a preview pane that opens BESIDE
the conversation, keyboard-operable, sandboxed, never replacing the
transcript that spawned it.
Backend
- New built-in open_preview(target, kind?, title?): resolves an http(s)
URL, a file path, or attachment:<id> to bytes; classifies into
web/pdf/image/table/text/markdown (magic bytes > MIME hint >
extension > UTF-8 fallback, legacy-charset pages transcoded); caps
size per kind; persists content-addressed with kind="preview" —
refcounted and GC'd with the workstream, skipped by trajectory
reconstruction so preview bytes can never materialize onto the wire.
URL targets gate like web_fetch (network egress); paths/attachments
run unprompted like read_file.
- New core.web.fetch_with_ssrf_guard: manual redirect walk that
SSRF-screens every hop BEFORE requesting it (follow_redirects=True
checked nothing between hops); adopted by both open_preview and
web_fetch. URL userinfo is stripped before the descriptor or the
stored bytes see it; <base href> is injected doctype-safely so
relative assets resolve without quirks mode.
- The preview descriptor rides the tool turn's meta side channel with
ONE shape on every boundary: the live tool_result SSE event, the
conversations.meta column, and the /history projection. Cancelled
batches commit an already-announced preview (blob + meta) instead of
stranding the open pane on a permanent 404.
- New GET {ws}/attachments/{id}/preview (read scope, same ownership
gate as /content) serves the STORED type with per-MIME hardening:
bare CSP sandbox for text/html (renderable, scriptless, opaque
origin), no CSP for application/pdf (Chromium's viewer refuses
sandboxed contexts), full default-src 'none' otherwise; filenames
fold to latin-1-safe ASCII. The console /node proxy now forwards
CSP/nosniff/disposition/cache-control instead of dropping them.
- History loads exclude preview blobs from the bulk content fetch at
the query (they were read and discarded on every load).
Frontend
- New "preview" pane type registered in the shared shell (server +
console): openPaneBeside placement, per-kind renderers — fully
sandboxed iframe for pages, browser PDF viewer, sortable tables
(CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown,
text — plus back/forward history with arrow keys, reload persistence
via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap
between the live descriptor and the batch fold that commits its blob.
- Tool results carrying a descriptor render a credential-redacted
preview chip (the reopen + replay affordance); live results auto-open
the pane only while the originating pane holds focus.
Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool
prepare/exec (mocked fetch), serving route + proxy header pass-through,
storage exclusion on both backends, cancel-path commit, JS static
guards; a headless-Chrome harness drives the real module graph (32 DOM
assertions).
|
||
|
Patrick Buckley
|
4350248d8f |
fix(oidc): carry the opt-in hint on discovered-endpoint rejections
The discovered-endpoint wrapper converted every OAuthSSRFError to a bare OIDCError, so a private-resolving endpoint or trusted host got the non-public message without the allow_private_network remediation even though the same knob fixes it. Hoist the hint into a module constant and append it in both wrappers. Also name "unspecified" in the refused-even-with-opt-in message so 0.0.0.0/:: rejections read unambiguously. |
||
|
Patrick Buckley
|
9c74673fd4 |
feat(oidc): allow_private_network opt-in for self-hosted IdPs
The SSRF guard on OIDC endpoint URLs hard-refused any hostname resolving to a non-public address, which made it impossible to use a self-hosted IdP (Keycloak, Authentik, Dex) on an internal network — even though the login-flow issuer is operator-configured, i.e. trusted input. Add [oidc] allow_private_network in config.toml (or TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK), default off. When set, the issuer and its discovered endpoints may resolve to private-range, unique-local, CGNAT, and loopback addresses. Link-local, multicast, unspecified, and reserved ranges stay refused regardless — cloud metadata services live on link-local and no legitimate IdP does. The HTTPS requirement and same-origin endpoint checks are unchanged. The private-address refusal now raises OAuthSSRFPrivateAddressError, and the OIDC wrapper appends the remediation hint to the error message so the failure is self-service. mcp_oauth call sites — where endpoint URLs come from untrusted remote-server metadata — do not get the knob and keep the strict public-address rule. |
||
|
Patrick Buckley
|
19b1a04f17 |
fix(redaction): match connection-string schemes case-insensitively
RFC 3986 schemes are case-insensitive, so POSTGRESQL+PSYCOPG2:// or HTTPS://user:pass@host in tool output leaked the password past the case-sensitive scheme alternation. Compile with IGNORECASE on both sides of the FE/backend mirror; the structural userinfo requirement is unchanged. Uppercase-scheme cases added to both test suites. |
||
|
Patrick Buckley
|
ed2623ff44 |
fix(redaction): match SQLAlchemy driver schemes; add FE prefilter bailout
Connection-string redaction (the output_guard pattern and its frontend mirror) only enumerated bare dialects plus +psycopg, so SQLAlchemy dialect+driver URLs — postgresql+psycopg2://, postgresql+asyncpg://, mysql+pymysql:// — leaked the password through every redaction surface. The scheme now takes an optional +suffix instead of enumerating drivers. redactCredentials() also gains a single early-exit prefilter scan ahead of its sixteen replace passes, for plain-log tool output on card render. The prefilter is documented and pinned as a superset of the pattern set's required substrings, so a miss is provably a no-op: new smoke cases assert bare sk-/AKIA/Bearer credentials with no '=', quote or '@' anywhere in the text still redact, alongside the fast-path no-op and the driver-scheme URLs on both sides of the mirror. |
||
|
Patrick Buckley
|
a029849724 |
fix(models): keep raw exception text out of client-construction 503s
Review: the wrapped ValueError is echoed in 503 bodies, and arbitrary SDK exception text can embed filesystem paths. Echo the exception type only; log the full exception with traceback at the raise site. |
||
|
Patrick Buckley
|
9c2e809b26 |
fix(models): surface client-construction failures as factory misconfig
SDK client construction can fail on environment problems the config never sees (e.g. httpx resolving a certifi CA path deleted by a venv rebuild). Those escaped as bare exceptions and turned every workstream open/create into an opaque 500; re-type them as ValueError in ModelRegistry.get_client so routes answer 503 with the message and the alias. |
||
|
Patrick Buckley
|
51ed336989 |
fix(storage): survive oversized rows in postgres history search
to_tsvector was computed inline over full row content, so one row whose tsvector exceeds PostgreSQL's 1MB limit aborted every search_history scan. Cap the FTS input at 250K chars (worst-case tsvector expansion stays under the limit; giant rows remain findable by their head). The ILIKE fallback also never ran on postgres: the failed statement leaves the autobegun transaction aborted, so roll it back before falling back. |
||
|
Patrick Buckley
|
647939fe4d |
fix(personas): repr the input in the not-found resolver error
Review feedback on #792: the "not found or disabled" branch interpolated the raw input unquoted, so the whitespace-only and trailing-space inputs the forgiving lookup explicitly handles rendered invisibly in CLI output and logs. Use {name!r} like the other two resolver errors already do. |
||
|
Patrick Buckley
|
457b01737a |
feat(personas): agent discoverability + forgiving name resolution
Coordinators and interactive agents had no way to enumerate valid persona names: task_agent / spawn_workstream / spawn_batch described `persona=` but nothing listed what it accepts, and resolution was an exact case-sensitive slug match - users reaching for the display name or a case variant got an unexplained failure. - Inject the live persona catalog (enabled, interactive-kind; children and sub-agents are always interactive) into the `persona` parameter description of task_agent / spawn_workstream / spawn_batch, riding the same render path as the model-alias injection. Rebuilt from the pristine TOOLS base every render, so repeated renders are idempotent and archived personas drop out instead of lingering. Entries carry name + default marker + <=96-char description; names-only past 25 personas. spawn_batch's persona property is nested per-child under children.items.properties (located via _persona_property, null-safe against name-colliding MCP tools). Storage-less sessions keep the base text: the render runs at session construction, so it gates on is_storage_initialized() rather than get_storage(), which would auto-init SQLite as a side effect. - resolve_persona_for_kind - the ONE shared rule behind the HTTP create handler, CLI --persona, the coordinator spawn precheck, and task_agent prep - is now forgiving: exact slug, then the lowercased input (created names are regex-enforced lowercase slugs), then a case-insensitive display-name match accepted only when unique among the kind's enabled personas. Duplicates refuse loudly naming the candidate slugs; a same-label persona of another kind neither blocks nor wins (the label the caller saw came from a kind-filtered surface); whitespace-only input never matches blank display names (display_name defaults to ""). Every failure now enumerates the kind's valid names, so a stale injected list or a typo self-corrects on the next attempt. - The canonical slug is stamped everywhere: task_agent prep rewrites its arg from the resolved snapshot, _validate_child_persona returns (canonical, error) and both spawn call sites adopt it - approval chrome, the wire, and workstream_config never carry a forgiven variant. - Create-persona shelf: label hint under Name explaining agents and the CLI launch the persona by this name (case-insensitive) and the display name is only a list label. docs/personas.md gains a "How agents discover personas" section and drops the stale claim that task_agent has no persona parameter. Tests: resolver unit suite (case/display/ambiguity/cross-kind/ whitespace/disabled/storage-failure) + guards for injection content and ordering, idempotent re-render, archive drop, the 25-persona prose cutoff, coordinator-kind exclusion, and canonical stamping through spawn_workstream / spawn_batch / task_agent. |
||
|
Patrick Buckley
|
a40ff249ec |
fix: address review — request-scoped storage in coord tenancy checks
- _coordinator_tenant_check and _coord_attachment_owner resolved storage from the global registry (get_workstream_row / for_request without a storage arg), which can evaluate the project-tenancy decision against a different or auto-initialised backend and fail OPEN on a missing project row. Use request.app.state.auth_storage explicitly, matching cluster_ws_detail and _resolve_coordinator_or_404; fail closed (404) when it is unset. - reject_unassignable_scopes now derives its allowed-scope error message from ASSIGNABLE_SCOPES so validation and the message can't drift. |
||
|
Patrick Buckley
|
36419a9809 |
fix: scope private-project workstream visibility to members, not admins
Workstreams attached to a private project were visible -- including their conversation content -- to holders of admin.cluster.inspect / admin.coordinator (both default builtin-admin permissions), defeating the project's confidentiality boundary. Enforce that a private project's resources are visible only to people IN the project (owner, workstream creator, or an explicit member), even for admins. Surfaces closed: - WorkstreamProjectVisibility bypass narrowed to service scope only (node->console machine plumbing, re-filtered per-user at the console edge). No human principal bypasses; admin.cluster.inspect gates the inspect surfaces, not tenancy. This flows to /dashboard, session listings, the attachment row-gate, cluster_workstreams, cluster_node_detail, and cluster_snapshot/SSE. - cluster_ws_detail 404-masks a workstream in a private project the caller can't see; cluster_ws_live_bulk routes such ids to the denied list (no private-project oracle). - Coordinator operator verbs (history/export/detail/send/approve/set_title/open/ children/tasks/attachments) now enforce project tenancy: _coordinator_tenant_check on coord_endpoint_config, the gate in _resolve_coordinator_or_404 (children/tasks), the tenant_check now run in make_open_handler before rehydrate, and a project-visibility check in _coord_attachment_owner. admin.coordinator gates the surface cluster-wide, but a non-member is 404-masked. The tenant-check mirrors the manager-first + coordinator-kind ladder so kind-isolation is preserved. - service scope is no longer user-assignable: admin_create_token and both turnstone-admin CLI mint paths reject it via reject_unassignable_scopes, so an admin.users holder cannot self-mint a service token and restore the bypass. Service scope is minted only by ServiceTokenManager / the JWT secret. - The events/global node proxy (service-elevated cross-tenant firehose) is gated on admin.cluster.inspect so a plain authenticated user cannot reach it through the console proxy. Updates the OpenAPI description, the row-gate/tenancy-filter docstrings, and adds tests for every surface (visibility predicate + cluster detail/bulk + coordinator history/export/children/open/attachments + events/global proxy + scope-mint rejection); inverts the tests that pinned the old admin-bypass contract. |
||
|
Patrick Buckley
|
0c2c534c86 |
fix(mcp): route pool transport lifecycles through per-entry owner tasks (#788)
* fix(mcp): route static transport lifecycles through per-server owner tasks A crash-looping MCP server drove the mcp-loop thread to a sustained, climbing 100%+ CPU spin. Root cause: anyio cancel scopes are host-task-bound, and the static path entered the SDK's transport / ClientSession task-group scopes from short-lived connect tasks (every health tick is a new task since #768). Once such a scope was cancelled after its host task had finished - by anyio's task_done when a transport child died with the server, or by ClientSession.__aexit__ during a cross-task teardown - CancelScope._deliver_cancellation could never make progress (task.cancel() on a done task is a no-op) and re-armed itself via call_soon every loop iteration, forever: ~900k callbacks/s per zombie scope, one more per flap cycle (verified against anyio 4.14.1; no upstream fix exists as of that release). Fix: each static server's transport + session cms are now entered, parked, and exited by ONE long-lived owner task (_static_transport_owner), so scopes always have a live host and always exit in the task that entered them. Teardown follows a one-cancel close protocol (signal the close event before the first await, graceful grace, then at most ONE cancel - never a second, which would abandon a scope exit mid-flight). Connect timeouts now cancel only the waiting caller; connect failures are delivered through a readiness future; unrequested owner death (server died under a live session) evicts the session immediately via a done-callback instead of waiting for the next liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop (_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not yet migrated (the oauth_user pool keeps the old cross-task-close shape; follow-up). Also fixed: BaseExceptionGroup (BaseException-derived, as raised by anyio task groups wrapping a stray CancelledError, e.g. an accept-then-RST server) escaped `except Exception` in _connect_all and killed it before the health/sweep loops were created - silently disabling all autonomous recovery. Handled there and in the health/sweep/eviction loops and the reconnect/refresh callers. Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one armed scope per cycle) to 0.3% flat with zero armed scopes; the RST repro now leaves both background loops alive (previously both silently dead). New tests: owner-lifecycle + close-protocol units (incl. an exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression, a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke test (real FastMCP subprocess, skips on environment gaps) asserting zero armed scopes, exactly one live owner, and a post-recovery tool call. Full 8470-test suite green; ruff+mypy clean. * fix(mcp): route pool transport lifecycles through per-entry owner tasks Completes the owner-task migration started for the static path: the oauth_user pool path had the same latent anyio cancel-scope exposure (host-task-bound scopes entered by short-lived connect tasks; a scope cancelled after its host finished re-delivers cancellation via call_soon forever - the 100%-CPU zombie), previously covered only by the disarm backstop. Each (user, server) pool entry's transport + ClientSession cms are now entered, parked, and exited by ONE long-lived owner task (_pool_transport_owner). The caller keeps building client_kwargs (the per-user bearer and, when an auth-capture carrier is active, the httpx_client_factory response hook) so 401/WWW-Authenticate capture semantics are unchanged. Teardown is the shared one-cancel close protocol (_teardown_pool_entry: signal before first await, graceful grace, at most ONE cancel), used by the connect stale-guard, idle/LRU eviction, and shutdown (parallel signal-then-reap). Unrequested owner death evicts the session but keeps the entry and its discovered catalog, matching the existing evict-session-keep-entry semantics the auth_401 retry relies on. Discovery still runs in the connecting caller while the transport is hosted by the owner, so a transport collapse mid-discovery (e.g. the SDK tearing its task group down on an upstream 401) cancels the OWNER, not the caller - a bare await on the response stream would hang until the 30s phase timeout. _await_pool_discovery races each discovery await against owner completion and converts owner death into a prompt ConnectionError (the owner is never cancelled there; teardown owns its lifecycle). Carrier-first failure classification preserves auth_401 semantics for captured 401s. With no cross-task stack closes left, _safe_close_stack and _safe_teardown_on_connect_failure are deleted (zero callers). Tests: new tests/test_mcp_pool_owner.py pins the pool close protocol (graceful event-before-await close, exactly-one-cancel escalation, owner-death eviction retaining entry+catalog, caller-cancel-mid-connect cm-exit guarantee, factory-present-iff-capture, and the owner-death-during-discovery fast-fail). 1010 mcp tests and the full 8471-test suite green, including the historical cross-task-anyio sentinel test_integration_pool_reuse_401_refresh_and_retry_succeeds; ruff+mypy clean; zero destroyed-task warnings. * fix(mcp): harden disarm-sweep loop guard and owner BaseException arm Review follow-ups on the owner-task migration: - _maybe_disarm_orphaned_scopes now enforces its mcp-loop requirement instead of trusting callers: it returns without walking (and without advancing the rate-limit clock) unless the currently running loop IS self._loop. A suppressed close can fire before start() or after shutdown(), where the walk would be wasted at best and a cross-thread reach at worst. - The transport owner's BaseException arm now re-raises non-Exception, non-group escapees (KeyboardInterrupt, SystemExit) after delivering them to the readiness future - failure delivery is the arm's job; swallowing an interpreter-level exit was not. * fix(mcp): extend owner-death discovery fast-fail to the static path The static connect path had the same exposure the pool's discovery race closed: discovery runs in the connecting caller while the transport is hosted by the owner task, so a transport collapse mid-discovery cancels the OWNER and the caller's bare await on the response stream hung until the caller-side attempt timeout (~45s) instead of failing promptly. _await_pool_discovery is renamed to _await_owner_discovery (it is now path-neutral) and wired into _connect_one_locked's four discovery awaits. The helper also converts a discovery future that completes CANCELLED without the race's own reap (an SDK-internal cancellation shape) into the same ConnectionError, instead of leaking a bare CancelledError the caller would misread as its own cancellation. The pool transport owner's BaseException arm gains the same refinement the static owner received in review: interpreter-level exits (KeyboardInterrupt, SystemExit) re-raise after delivery to the readiness future instead of being swallowed. Tests: static owner-death-during-discovery fast-fail (<1s vs the ~45s hang), and a direct pin on the cancelled-discovery-future conversion. * fix(mcp): replace owner BaseException arm with targeted catch + finally delivery The owner's failure arm now catches only (BaseExceptionGroup, Exception); waiter delivery for everything else moves to a finally that resolves the readiness future with a clean transport-failure ConnectionError before the task unwinds. Interpreter exits and BaseException-derived library control-flow escapes propagate from the owner exactly once, uncaught - and the waiter can never be left hanging on an unresolved future (the initial _connect_all connect has no outer bound). For SystemExit / KeyboardInterrupt asyncio additionally stops the loop right after, so the delivery is load-bearing for the non-exit BaseException shapes and free for the exits. Pinned by a test driving a BaseException-derived escape through the owner: the waiter resolves promptly with ConnectionError while the escape propagates unswallowed. * fix(mcp): mirror targeted-catch + finally delivery in the pool owner Same shape the static owner received in review: the failure arm catches only (BaseExceptionGroup, Exception), and waiter delivery for anything else moves to a finally that resolves the readiness future with a clean ConnectionError before the task unwinds - interpreter exits and BaseException-derived library escapes propagate exactly once, uncaught, and the waiter can never be left hanging. * test(mcp): narrow the escape test's waiter catch to explicit types * test(mcp): narrow discovery-race waiter catches to explicit types * refactor(mcp): make reap/synchronization awaits explicit to analyzers Full-absorb reaps (cancel-then-drain of a future whose outcome is deliberately consumed) become `await asyncio.gather(x, return_exceptions=True)` - one line, self-describing, and in the owner-died discovery reap it is also a small semantic improvement: a caller cancellation arriving during the reap now propagates instead of being masked by the ConnectionError. Bare synchronization awaits and selective suppress blocks in tests keep their raise-through semantics via throwaway assignment. Applied uniformly across the owner-task test files, including sites introduced by the static-path PR. |
||
|
Patrick Buckley
|
7da731cbe1 | test(mcp): narrow the escape test's waiter catch to explicit types | ||
|
Patrick Buckley
|
8ed86ae7ab |
fix(mcp): replace owner BaseException arm with targeted catch + finally delivery
The owner's failure arm now catches only (BaseExceptionGroup, Exception); waiter delivery for everything else moves to a finally that resolves the readiness future with a clean transport-failure ConnectionError before the task unwinds. Interpreter exits and BaseException-derived library control-flow escapes propagate from the owner exactly once, uncaught - and the waiter can never be left hanging on an unresolved future (the initial _connect_all connect has no outer bound). For SystemExit / KeyboardInterrupt asyncio additionally stops the loop right after, so the delivery is load-bearing for the non-exit BaseException shapes and free for the exits. Pinned by a test driving a BaseException-derived escape through the owner: the waiter resolves promptly with ConnectionError while the escape propagates unswallowed. |
||
|
Patrick Buckley
|
62f62ae624 |
fix(mcp): route static transport lifecycles through per-server owner tasks
A crash-looping MCP server drove the mcp-loop thread to a sustained, climbing 100%+ CPU spin. Root cause: anyio cancel scopes are host-task-bound, and the static path entered the SDK's transport / ClientSession task-group scopes from short-lived connect tasks (every health tick is a new task since #768). Once such a scope was cancelled after its host task had finished - by anyio's task_done when a transport child died with the server, or by ClientSession.__aexit__ during a cross-task teardown - CancelScope._deliver_cancellation could never make progress (task.cancel() on a done task is a no-op) and re-armed itself via call_soon every loop iteration, forever: ~900k callbacks/s per zombie scope, one more per flap cycle (verified against anyio 4.14.1; no upstream fix exists as of that release). Fix: each static server's transport + session cms are now entered, parked, and exited by ONE long-lived owner task (_static_transport_owner), so scopes always have a live host and always exit in the task that entered them. Teardown follows a one-cancel close protocol (signal the close event before the first await, graceful grace, then at most ONE cancel - never a second, which would abandon a scope exit mid-flight). Connect timeouts now cancel only the waiting caller; connect failures are delivered through a readiness future; unrequested owner death (server died under a live session) evicts the session immediately via a done-callback instead of waiting for the next liveness ping. A rate-limited, mcp-loop-scoped gc-walk backstop (_maybe_disarm_orphaned_scopes) disarms any zombie minted by paths not yet migrated (the oauth_user pool keeps the old cross-task-close shape; follow-up). Also fixed: BaseExceptionGroup (BaseException-derived, as raised by anyio task groups wrapping a stray CancelledError, e.g. an accept-then-RST server) escaped `except Exception` in _connect_all and killed it before the health/sweep loops were created - silently disabling all autonomous recovery. Handled there and in the health/sweep/eviction loops and the reconnect/refresh callers. Verified: a live SIGKILL-flap repro went from 130%+ CPU (climbing, one armed scope per cycle) to 0.3% flat with zero armed scopes; the RST repro now leaves both background loops alive (previously both silently dead). New tests: owner-lifecycle + close-protocol units (incl. an exactly-one-cancel pin), a _connect_all BaseExceptionGroup regression, a discriminating disarm-sweep test, and a ~10s live SIGKILL-flap smoke test (real FastMCP subprocess, skips on environment gaps) asserting zero armed scopes, exactly one live owner, and a post-recovery tool call. Full 8470-test suite green; ruff+mypy clean. |
||
|
Patrick Buckley
|
31a1d5c3ee |
fix(redact): harden credential redaction and restore JS/backend parity
Address review findings on the client-side credential redactor and mirror
each fix into the backend output guard so both surfaces censor identically:
- Detect and redact single-quoted JSON secrets such as
{'Authorization': 'Bearer ...'} (Python dict reprs / JS object literals),
which the double-quote-only pattern silently bypassed on both sides.
- Cover mongodb+srv://, rediss:// and amqps:// connection strings.
- Match the Bearer auth scheme case-insensitively (RFC 7235).
- Redact prefixed key/token assignments (api_key=, secret_key=,
access_token=) as a whole rather than chewing the tail into a garbled
"api_[REDACTED:api_key]", while still covering bare key=/token=. A word
boundary was rejected because it would drop coverage for <prefix>_key=.
- Remove the redundant |Authorization alternative (covered by /i) and swap
the manual value-slicing helper for a capture-group substitution.
The backend edits touch both detection sites and both redaction pipelines,
so single-quoted secrets are flagged (and therefore sanitized), not merely
rewritten. Adds JS runtime-smoke and backend unit coverage for every case.
|
||
|
Patrick Buckley
|
93a7486cc2 |
Add client-side credential redaction for tool call cards
New shared ES6+ module (redact_credentials.js) provides comprehensive visual credential censorship matching the backend output guard patterns: - PEM private key blocks, connection strings, Bearer tokens - OpenAI / GitHub / AWS / Google API key formats - Query-string and JSON-style credential values - JSON secret keys (api_key, password, token, authorization, etc.) - ENV secret lines (SECRET_KEY=, DATABASE_URL=, etc.) Integrated into both frontend surfaces: - interactive.js: replaces legacy minimal _redactApiKeys function - conversation.js::buildConvResult (shared substrate, used by coordinator) - coordinator.js::renderToolOutput fallback paths Backend parity: added 'authorization' to the JSON secret regex in output_guard.py so the output guard flags and redacts Authorization headers in JSON tool output. Tests: ported the runtime smoke test from the removed _redactApiKeys to import the new module directly; added redact_credentials.js to the var-free and const-reassign guard bundles. |
||
|
Patrick Buckley
|
56624f9597 |
fix(core): scrub credentials and control chars from tool-args log preview
`tool_args_preview` feeds `stream.tool_args_malformed` (WARNING) and `wire.tool_args_legalized` (DEBUG), and tool arguments are model/user controlled — they can carry secrets (a token in a bash command, a password in a connection string) or raw CR/LF that break log lines. Route the preview through `output_guard.redact_credentials` over the full value first (before the 120-char cap, so a secret straddling the cut isn't half-shown past the pattern's reach), then collapse every control char to a space, mirroring `audit._scrub_string`. Addresses the PR review comments. |
||
|
Patrick Buckley
|
16a68ae6d6 |
fix(core): legalize malformed tool-call arguments before the wire
A tool call whose `arguments` is not a JSON-object string (an unterminated
string from a non-`length` truncation, or an empty `""` from a no-arg call)
was committed verbatim and replayed on every subsequent send. Strict renderers
that re-parse arguments at render time (vLLM's `deepseek_v4`
`_postprocess_messages` runs `json.loads` on them) reject the whole request
with HTTP 400, wedging the conversation. The only prior guard dropped partial
tool calls on `finish_reason == "length"`; a `stop`/`tool_calls` finish reason
carrying invalid JSON slipped through, and its synthetic "retry" result kept it
from being an orphan, so the orphan-repair pass never touched it.
Add `sanitize_tool_call_arguments`, a wire-neutral legalize pass in lowering
(fold, legalize, repair), normalizing any non-JSON-object `arguments` to `{}`
on the transient wire copy only. The canonical trajectory keeps the raw model
output, so a wedged session self-recovers on its next send. A
`wire_valid_arguments` predicate is shared with a non-destructive
`stream.tool_args_malformed` warning at the stream accumulator, which surfaces
the model-quality problem at production time.
Convert `lowering.py` to structlog so the new pass emits structured events.
|
||
|
Patrick Buckley
|
2d4cb6fea9 |
fix(ui): make pane hotkeys work off macOS and match across surfaces
The pane/workstream accelerators only worked on macOS. They were bound to Ctrl, which on Windows/Linux IS the browser's own accelerator: Ctrl+T, Ctrl+W and Ctrl+1-9 were swallowed by the browser (new tab / close tab / switch tab) and never reached the page. macOS browsers own Cmd instead, so Ctrl was free there and everything appeared to work. On top of that the shortcuts were declared in three places that had drifted apart — the "?" overlay, each app.js keydown handler, and the tab-menu badges in shell.js. The console fell to convTabMenu's node-proxy fallback lane, which dropped every shortcut badge (and Fork), so its tab menu showed no accelerators and Ctrl+W there just closed the browser tab. Choose the modifier per platform (Ctrl on macOS, Alt on Windows/Linux) and make shell.js the single source of truth for the per-pane accelerators: a stable accel registry drives both the platform-aware badge and one shared keydown handler that invokes the ACTIVE pane's own menu item, so a badge can't advertise a chord the handler ignores and each surface contributes only what it supports (the console omits Fork; it has no fork surface yet). Each surface's app.js keeps only its global accels (new / switch / dashboard); the console regains switch + dashboard to match. Mod+W now uniformly means Close pane (drop the tab, session keeps running), matching its badge and the universal Ctrl+W convention — previously the standalone's Ctrl+W stopped the session. The previously-dead "Refresh title / Ctrl+Shift+R" is wired, and Ctrl+T / Ctrl+D yield to text editing while a field is focused (macOS transpose / delete-forward). |
||
|
Patrick Buckley
|
3615f98c19 |
Fix send button stuck disabled by pruning orphaned approval cycles (#775)
* Fix send button stuck disabled by pruning orphaned approval cycles When a DOM wipe (clear_ui / replay_truncated / replaceChildren) detaches approval card elements while an approve_request event is processed between the wipe and refetch-restore, the matching approval_resolved may never arrive. The orphaned cycle entry in approvalCycles keeps pendingApproval=true and the send button disabled forever. The fix adds a pruning pass at the top of _syncApprovalState(): cycles whose blockEls are all .isConnected === false are deleted from the Map. This runs on every register/resolve/rebuild so orphans are cleaned up promptly. Also fixes an ordering bug in showInlineToolBlock discovered during review: the block element was appended to the DOM after _registerApprovalCycle, so the new isConnected prune would kill the just-registered cycle before it took effect. * Fix comment inaccuracy in showInlineToolBlock append-before-register guard The comment said 'blockEls.every(el => el.isConnected)' but the actual prune check is '!blockEls.some(el => el.isConnected)' - no block elements are connected, not every element. |
||
|
Patrick Buckley
|
6f8efaa44e |
test(golden): freeze the anthropic-compatible reasoning-effort wire
The wire-payload golden matrix had no anthropic-compatible coverage —
both AnthropicProvider rows are the native lane (compat=False), so the
distinct compat wire shape (reasoning control in
extra_body.chat_template_kwargs, never the native thinking param) was
unfrozen. Add the compat lane across all eight representative fixtures
with a manual-mode capability (the lane has no static table, so caps
ride in as a model definition would supply them) and reasoning_effort=
high: every golden now pins {enable_thinking: true, reasoning_effort:
high} in chat_template_kwargs, asserts the native thinking param is
absent, and preserves temperature (no forced 1.0). _capture gains an
optional caps override to support the no-static-table lane.
|
||
|
Patrick Buckley
|
530958e06b |
fix(providers): the session effort level always reaches the local-lane wire
Local lanes dropped the knob's graded value unless the operator declared
reasoning_effort_values (and, on the template channel, an effort key) —
picking Max sent a bare thinking toggle and the effort select
degenerated into seven positions that all meant 'on'. The user's
setting now always rides:
- openai-compatible: the flat reasoning_effort param carries the knob
verbatim (effort_passthrough on the lane default); declared values
still snap ordinally, and a declared effort_param still claims the
template channel and suppresses the flat param.
- anthropic-compatible: the graded value rides chat_template_kwargs
alongside the toggle whenever reasoning control is engaged — under
the operator's effort_param, else the conventional fallback key
(reasoning_effort); templates that don't reference the kwarg ignore
it. thinking_mode=none still injects nothing.
- Commercial lanes untouched: empty declared values still mean 'no
effort control' (o1-mini) and the ordinal snap is unchanged.
Golden writer now pins ensure_ascii=False: the baselines' literal em
dashes came from a hand edit (
|
||
|
Patrick Buckley
|
e136237b63 |
fix(providers): openai-compatible never consults the commercial table
Local-lane model ids are operator-chosen strings (vLLM --served-model-name), so a prefix collision with a cloud model id inherited that model's sampling and effort contract: a box named o3-distill silently lost temperature support, and one named gpt-5.5-my-finetune was sent gpt-5.5's snapped reasoning_effort values it never declared. Both surfaces of the lane now return plain defaults (OPENAI_COMPAT_DEFAULT in _openai_common): the chat class directly, and the responses pin via a compat-mode OpenAIResponsesProvider mirroring AnthropicProvider(compat=True). Everything beyond the defaults is declared by the operator on the model definition, matching the anthropic-compatible lane and lookup_model_capabilities' documented 'no static table for local models' contract. The commercial openai lane (Responses-only) is untouched. Pre-split tests that reached commercial rows through the chat-class OpenAIProvider alias now source them from lookup_openai_capabilities; their subject (registry rows + shared gating helpers) is unchanged. |
||
|
Patrick Buckley
|
41e9907803 |
fix(console): available-models rows always carry effort_ladder
The except path for a malformed capabilities column appended the row without the key, so clients had to null-check a field the happy path guarantees. Initialize each entry with an empty ladder and let the try block overwrite it — the response schema is stable per row. |
||
|
Patrick Buckley
|
7f0e0406b3 |
test(approvals): concurrency matrix + suite migration to the cycle model
New regression matrix for the release blockers: cross-approval independence, lost-wakeup at gate entry, FIFO selector-less resolution, resolve-all sweep, double-resolution no-op, cards/legacy view tracking, and the generation-exactness set — stale delivery rejection, Smart-Approvals origin check, purge keep_origin, the purge-to-register window eviction, late cross-generation "superseded" stamping, concurrent smart+human gates, and the pre-delivered-verdict fast path. Plus sub-agent judge wiring (agent_gate off the main slot, close() firing all generations) and endpoint tests for cycle pinning and the Approve+Always race guard. Gate threads run under one shared mock-patch harness — mock.patch start/stop of the same target from concurrent threads corrupts the patcher's restore stack — with a sweep-until-dead teardown so the conftest leak guard can't trip. Existing suites migrate off the singleton fields to cycle assertions and the pending_approval_details wire shape. |
||
|
Patrick Buckley
|
3607517814 |
fix(providers): registry effort truth — o-series/gpt-5.5/codex-max/sonnet-5; forward declared none
Capability-registry corrections verified against the official OpenAI reasoning guide, the Azure reasoning-models matrix (2026-06 revision), and the Anthropic models-overview/effort/migration pages (2026-07): OpenAI (vocabulary confirmed none/minimal/low/medium/high/xhigh — no "max" level exists; knob max rides the xhigh ceiling via the ordinal snap): - o1/o3/o3-mini/o3-pro/o4-mini declare low/medium/high (every o-series model except o1-mini) — without declared values the session knob was silently dropped for these models. o1-mini stays effort-free. - gpt-5.5 default corrected none -> medium (5.5 reasons by default, unlike 5.1-5.4). - gpt-5.1-codex-max gets an explicit row: it prefix-matched the gpt-5.1 row (no xhigh), capping the knob's xhigh at high on the one model xhigh was introduced for. Anthropic (effort-page matrix): - claude-sonnet-5 row added — it previously fell through to _ANTHROPIC_DEFAULT (manual budgets, 200k ctx, no effort), all wrong: adaptive-by-default thinking (manual budgets are a 400), sampling params rejected, 1M ctx / 128k out, effort low..max incl. xhigh. - claude-sonnet-4-6 gains its documented "max" effort level (knob xhigh now rides max, not high) and the stale 64k max_output becomes the documented 128k. - fable-5 / opus-4-8 / opus-4-7 / opus-4-6 / opus-4-5 rows verified correct as declared. Knob semantics completed: resolve_reasoning_effort now forwards the knob's "none" position verbatim when the model DECLARES an explicit none level (gpt-5.1+, grok-4.3) — omitting the param there leaves a reasoning-on server default (gpt-5.5: medium) in charge of a knob that promises off. Models without a declared none still omit, and none is never a snap target. Parity harness swaps its synthetic openai shape for the real gpt-5.5 registry row. |
||
|
Patrick Buckley
|
a0e04a8588 |
fix(providers): effort snapping is ordinal — round up, cap at the ceiling
The knob domain grew xhigh/max after the snapping fallbacks were written, which silently inverted their semantics: off-list meant "unrecognized string" then, but now usually means "above the model's ceiling", where falling back to the default tier is directionally wrong (grok-4.3 at knob max got low; values low/medium/high at knob xhigh got medium; Anthropic manual mode gave xhigh/max a 4096 budget while high got 16384). One rule everywhere now, via snap_reasoning_effort in _protocol: exact match wins; otherwise the smallest declared level ranking at or above the knob; above the ceiling, the ceiling. "none" is never a snap target, and default_reasoning_effort only catches values the ordinal snap cannot rank. - resolve_reasoning_effort (flat chat / responses / validated effort_param lanes) snaps ordinally: xhigh over (low, medium, high) now sends high; xhigh over DeepSeek-style (high, max) sends max — matching DeepSeek's official xhigh-to-max aliasing, so a declared values list now reproduces that contract instead of defeating it. - _map_reasoning_to_effort (native output_config) rounds up too: knob xhigh on Opus 4.6 (low, medium, high, max) rides max instead of silently dropping output_config. - EFFORT_BUDGET_MAP is monotone across the whole knob domain: minimal/low 1024 (API floor), medium 4096, high 16384, xhigh 32768, max 65536. Unknown strings still fall to the 4096 default. Google defaults are unaffected (ceiling and default coincide at high); wire goldens unchanged. Parity harness caught the budget clamp interacting with its own max_tokens during development — capture budget raised above the largest manual budget. |
||
|
Patrick Buckley
|
ffe8214cfe |
test(providers): ladder-to-wire effort parity harness across all lanes
Proves the effort-ladder projection against the real request path instead of against the mapping helpers it shares with it. For 22 (provider lane x capability shape) points — both Anthropic lanes, openai-compatible on both API surfaces, openai, google (default and template-override hybrid), xai (default and inert-override), and the DeepSeek/qwen template contracts — every knob position is driven through the actual provider create_streaming against a recording fake client, and two invariants are asserted per shape: 1. each ladder token decodes to an expected effort wire subset (toggle / template effort / flat param / thinking budget / output_config) that must equal the captured kwargs exactly; 2. two knob positions carry equal tokens iff they produce identical effort-relevant wire payloads — the grouping promise the UI annotations lean on. The RecordingClient SDK-seam stub moves from the wire-payload golden harness into tests/_wire_capture.py so both suites capture at the same seam. Verified the harness catches the bug class it was built for: re-adding xai to _CHAT_LANES fails xai-template-override-inert. |
||
|
Patrick Buckley
|
1f63f622c9 |
fix(providers): xai effort ladder is flat-only; share the suppression rule
Second external audit round on the ladder. Verified and fixed:
- xai was in _CHAT_LANES on the false premise that XAIProvider
subclasses the chat provider. It subclasses OpenAIResponsesProvider,
whose surface ignores extra_body entirely, so a thinking_mode /
effort_param override never changes an xai request — but the ladder
claimed a template toggle ("on+low") that does not exist on the
wire. xai now projects through the flat channel only, like openai.
- The flat-param suppression rule (a declared effort_param claims the
template channel) was encoded independently in
apply_temperature_and_effort and the ladder. Extracted into
flat_effort_suppressed() in _protocol so the request path and the
projection cannot drift.
- admin_effort_ladder logs the swallowed resolver exception before
returning 400 (a genuine bug would otherwise hide as a silent 400).
- list_available_models reads server_compat with .get() instead of
destructively popping it out of the parsed capabilities dict.
- models_changed SSE now re-annotates the skill launch-config effort
select after invalidating the models cache instead of leaving a
stale ladder until the next keystroke.
- Capabilities JSON textarea placeholder hints the two effort fields
that have no structured control (reasoning_effort_values,
default_reasoning_effort).
|
||
|
Patrick Buckley
|
f4701bf0f9 |
test(golden): re-baseline Google wire payloads for the effort knob
The Gemini effort fix (
|
||
|
Patrick Buckley
|
06cc184227 |
fix(console): address verified external-audit findings on the effort ladder
The one that mattered: /v1/api/models passed the capabilities column — a JSON STRING (sa.Text) — straight into effort_ladder_for_model, whose field filter calls .items() on it; the per-row guard swallowed the AttributeError, so effort_ladder was silently absent from every row and the sklc annotation could never fire. The endpoint now parses the JSON and splits the namespaced server_compat exactly like the model_registry loader, and a regression test seeds a string-capabilities row. Projection fidelity: effort_ladder_for_model threads api_surface (the responses surface ignores extra_body — flat-param-only ladder, matching create_provider's request-time divergence); google/xai route through the chat-lane projection they actually inherit (_finalize_extra_body + flat param); the native-Anthropic branch reflects that output_config gates on supports_effort alone, independent of thinking_mode; budget clamping to per-request max_tokens is documented as out of scope. Hardening and symmetry: admin endpoint 400s (not 500s) on non-dict JSON bodies and gained HTTP tests; the admin edit-load strips effort_param from the raw JSON only for the lanes whose save path re-adds it, so non-compat rows can't silently lose a stored key; the empty-model early return bumps the ladder sequence so stale in-flight responses can't re-annotate; alias labels only reference positions the target select actually offers (the skill shelf omits none/minimal); the sklc models cache no longer pins a rejected promise and is invalidated on the models_changed event; debounces unified at 500ms; merge_reasoning_template_kwargs now always returns a fresh dict for non-empty input; the shared budget constants are public. |
||
|
Patrick Buckley
|
59a527f2f2 |
feat(console): surface each model's effective effort ladder
Seven knob positions render as seven behaviors in the UI, but the real
ladder depends on the lane and the model: qwen3.6 has two (off/on),
DeepSeek-V4 three, Claude 4.6 five. Operators had no way to see which
positions alias — the confusion class behind silently-equal effort
levels.
providers/effort_ladder.py projects the knob domain through the same
mapping functions the providers use at request time (resolve_reasoning_
effort, reasoning_template_kwargs, the manual budget map — hoisted to a
shared constant so the projection can't drift), yielding
{value, effective} rows where equal tokens promise identical requests.
/v1/api/models rows now carry the ladder (guarded per row), and
POST /v1/api/admin/models/effort-ladder computes it for the admin
modal's unsaved edits.
The admin per-model effort select and the skill launch-config effort
select annotate aliased positions ("Max (= high)", "None (model
default)") with a sends-tooltip; annotations refresh as thinking-mode /
effort-param / capabilities fields change. The ladder describes what
Turnstone sends — server-side templates may alias further (DeepSeek-V4
folds low/medium into its default high tier).
|
||
|
Patrick Buckley
|
d7941c88be |
fix(providers): thread the session effort knob to Gemini
_GOOGLE_DEFAULT declared no reasoning_effort_values, so resolve_reasoning_effort returned None and the session effort knob was silently dropped for every Gemini model — the same bug class this branch fixed on the local lanes. Gemini's OpenAI-compat surface documents a flat reasoning_effort (2.5: thinking_budget mapping; 3.x: thinking_level), so declaring values lights up the inherited chat-completions path. Values are the safe cross-model set (minimal/low/medium/high): "none" is excluded because 2.5 Pro and the 3.x family reject disabling thinking — and the resolver never forwards the knob's none anyway (the param is omitted, server default applies). Off-list xhigh/max snap to the declared default high. Encoded from the official compatibility docs per the static-caps pattern; not live-verified. |
||
|
Patrick Buckley
|
2cf23b6fe2 |
fix(providers): address high-effort review of the reasoning-knob branch
Verified findings applied: - adaptive thinking_mode never knob-disables: the shared mapping now sends the toggle unconditionally true for adaptive (the native adaptive branch ignores the knob's none), while manual keeps the knob-driven contract. Restores the invariant the deleted chat-lane code upheld. - a set effort_param suppresses the flat top-level reasoning_effort on the chat lane: the template channel replaces it — double-sending could 400 on schema-strict servers and disagree with operator pins. - admin edit-save no longer drops a stored thinking_param when the thinking-mode dropdown is empty: the raw-JSON strip now only fires when a mode value actually round-trips through the dropdown. - effort_param persistence gated on the local-server lanes so a value lingering across a provider switch never lands on commercial rows. - three stale _compat_extra_params references renamed to merge_reasoning_template_kwargs. Documented dispositions (no code change): the knob-none-disables flip on upgrade is intentional and now carries an upgrade note; gateways fronting real Claude belong on provider=anthropic with a custom base_url (the compat lane is vLLM-schema-only); nonstandard thinking_mode strings staying inert is the intended allowlist contract. The real anthropic provider is unaffected throughout — official Claude models keep native thinking/output_config. |
||
|
Patrick Buckley
|
68b22adfa3 |
feat(providers): share the effort-knob→chat_template_kwargs mapping with the openai-compatible lane
Hoist the compat-lane injection into _protocol.merge_reasoning_template_kwargs (next to ModelCapabilities — one implementation for both local-server lanes) and retire OpenAIChatCompletionsProvider._apply_thinking_mode in its favor: _finalize_extra_body now receives the session effort knob, so thinking_mode manual/adaptive maps knob "none" to an explicit thinking_param false (previously the toggle was unconditionally true) and caps.effort_param carries the graded effort key on chat completions too. Operator server_compat pins still win; the Responses surface is untouched (native reasoning handles effort itself). The admin Models form grows an "Effort param" field that round-trips like thinking_param: lifted out of the raw capabilities JSON on edit-load, re-added on save, cleared by emptying the field. Verified live against qwen3.6-27b /v1/chat/completions: knob medium streams reasoning_content, knob none suppresses it. |
||
|
Patrick Buckley
|
9289693730 |
fix(providers): drive reasoning via chat_template_kwargs on the anthropic-compatible lane
The compat lane sent no reasoning control at all: vLLM's /v1/messages has no thinking request field, thinking_mode stayed "none", and the session effort knob was silently dropped. The reasoning levers live in the chat template, so fold them into extra_body chat_template_kwargs (_compat_extra_params): thinking_mode manual/adaptive maps the knob onto caps.thinking_param (effort "none" = off, mirroring the native manual-mode contract), and caps.effort_param (new ModelCapabilities field) carries a graded effort value for gpt-oss-style templates, validated against reasoning_effort_values when declared. Operator server_compat entries win on key collision; native thinking params, temperature forcing, and output_config never fire on compat. resolve_reasoning_effort moves from _openai_common to _protocol next to ModelCapabilities — importing it into _anthropic would otherwise cross provider families. The admin Models form now shows and round-trips the thinking-mode dropdown for this lane; the #661 hide was premised on thinking_mode being inert here, which this change inverts. Verified live against qwen3.6-27b on vLLM /v1/messages: knob medium streams a thinking block, knob none suppresses it, an operator pin beats the knob. |
||
|
metaclassing
|
deff44bcea |
Addendum to Entra ID's... proclivities (#772)
* OIDC entra capture * copilot being nitpicky --------- Co-authored-by: pow3rtool <root@pow3rtools> |
||
|
Patrick Buckley
|
217d3a3a9b |
feat(mcp): autonomous reconnect + liveness for static MCP servers (#768)
* feat(mcp): autonomous reconnect + liveness for static MCP servers Static (non-oauth_user) MCP servers had no autonomous reconnect. Every reconnect path was lazy — a tool dispatch (_cb_auto_reconnect), an operator refresh, or a config edit — and the MCP SDK's own reconnect is a bounded 2-attempt burst on the streamable-http GET stream only (verified: mcp 1.28.1), with no backoff and nothing for the other transports. So a static server that went down and came back while nobody was dispatching to it stayed disconnected until a dispatch or a manual reconnect. Worse, a session whose transport dies while idle survives as a non-None ClientSession with closed streams — nothing evicts it, so even a later dispatch may not notice until it fails. Add a static-server health loop on the mcp-loop (started in _connect_all; config ``static_health_check_seconds`` default 30, <= 0 disables): - Reconnect: a disconnected server (session is None) is reconnected on a capped, jittered, FOREVER backoff (full jitter, base 1s, cap 60s, no attempt limit) — a server that returns after a long outage reconnects within ~a minute, and a permanently-misconfigured one costs at most one attempt per cap. The health loop owns this clock; the circuit breaker stays the DISPATCH fail-fast gate (a tool call to a down server errors immediately rather than blocking on the retry), and the loop keeps breaker state in sync so an open breaker closes on reconnect. - Liveness: a connected server is pinged (send_ping) each cadence; a dead-but- idle one — which nothing else would notice — is evicted so the next tick reconnects it. This is the core of the "never reconnects" failure. Serialize _connect_one per server behind a per-name lock (split into a thin wrapper + _connect_one_locked): the health loop, a dispatch's _cb_auto_reconnect, and an operator refresh could otherwise interleave teardown/rebuild on the shared StaticServerState and corrupt it — a latent pre-existing race this also closes. The body is unchanged (only relocated), so the delicate anyio / wait_for connect logic is untouched. Out of scope (follow-up): silent GET-stream / notification death — the SDK stops the notification stream after 2 attempts while the request path stays alive, so send_ping is blind to it; the fix is bounded session recycling, which needs a static in-flight guard first (only PoolEntryState tracks in_flight today). Tests: backoff bounds (capped / jittered / forever, no overflow), reconnect success resets backoff + closes breaker, reconnect failure retries forever, in-flight skip, per-name serialization (no overlap), ping keeps healthy / evicts dead / evicts on timeout, tick skips oauth_user, connect_all start + disable gating, clean cancel. * fix(mcp): harden static-server health loop (review findings) A max-effort review found 13 concurrency/correctness defects, all from the loop mutating shared StaticServerState without the interlocks the pool path carries. Fix all 13: - In-flight interlock: add StaticServerState.in_flight (parity with PoolEntryState); _static_session_op increments/decrements around the static call_tool/read_resource/get_prompt session ops; the ping skips and never evicts a busy server, so a long tool call can't be torn down mid-flight. - Dead-transport gating: the ping evicts + trips the breaker only on _is_dead_transport(exc); an McpError, httpx.PoolTimeout, or plain ping timeout is "slow, not dead" and only reschedules (matching the dispatch path). - Session-identity: only evict the exact session that was pinged. - asyncio.timeout (invariant-18) not wait_for for the ping; 5s->30s; the timeout-scoped cancel is distinguished from an external shutdown cancel (which still propagates) via .expired(). - Bounded reconnect: wrap _connect_one in asyncio.timeout so a server that handshakes then stalls list_tools can't wedge the loop or hold the per-name lock forever (connect internals untouched). - Concurrent tick under asyncio.gather with a freshly-read clock for the sleep. - Cross-path coordination: reconnect_sync/remove_server_sync take the per-name lock across teardown+rebuild (calling _connect_one_locked directly); _cb_auto_reconnect reuses a health-established session instead of racing a redundant reconnect and no longer trips the breaker on lock contention. - Backoff hygiene on recovery; skip '__' names; on health reconnect clear only the open-circuit deadline (not the failure count) so a connect-ok/calls-fail server still escalates to a trip. Adds 14 tests and adjusts those that assumed the old behavior; suite 195->209. * fix(mcp): unify static-server reconnect coordination (round-3 review) A third review round + live testing found 8 issues on the health loop, five sharing one root: reconnect logic was fragmented across five drivers, each handling the lock / session-reuse / in_flight / config-recheck / breaker / clock differently and incompletely. Introduce one primitive and route every lazy/autonomous driver through it. _ensure_static_connected(name, cfg) — the single lazy (re)connect path, all under the per-name lock: config re-check (+ lock-identity re-check, closing the remove->re-add race) so a removed server is never resurrected; reuse-if-live so a queued/concurrent driver never tears down and rebuilds a live session (the observed reconnect storm); in_flight guard so a reconnect can't tear down a session with a call still in flight on the evicted stack; bounded connect; and the circuit breaker owned in one place (clear the open-circuit deadline on success per finding-13, record one failure on real connect failure). Returns the session on success/reuse, None on a deliberate skip, raises on real failure. Routed through it: the health loop, a dispatch's _cb_auto_reconnect, and _refresh_all; operator reconnect_sync stays a deliberate force-rebuild. Also: fresh-clock deadlines (the stale tick-start clock was landing deadlines in the past and collapsing the backoff into an every-tick retry storm); loop-death fix (the tick no longer re-raises a CancelledError found in the gather results — per-server fallout, not shutdown; the loop returns only when Task.cancelling() marks a genuine shutdown); dispatch breaker records no failure on a sync-boundary reconnect timeout (lock contention is not a server failure; real outcomes recorded once, inside the primitive). Cleanups: extract _teardown_static_session (was copy-pasted 3x); share _capped_exponential between the breaker cooldown and the reconnect backoff. Adds 17 tests; test_mcp_client 204->226. * fix(mcp): coherent timeout hierarchy + round-4 review fixes A fourth review round on the unified reconnect coordination found 5 correctness regressions + 1 cleanup, five sharing one root: the inner reconnect attempt bound (45s) was LONGER than every caller wait (dispatch 30s, remove 15s, reconnect 30s), so a caller cancelling mid-attempt delivered a bare CancelledError that slipped past the primitive's `except Exception`. - Coherent timeout hierarchy: add _STATIC_RECONNECT_CALLER_TIMEOUT_S (> the inner attempt bound) for the dispatch + operator waits, so the inner asyncio.timeout always fires first — a clean TimeoutError the primitive converts, cleans up, and records on the breaker — instead of a caller cancelling a live attempt. Fixes [0] (half-discovered session left installed, served with a stale catalog) and [1] (breaker never trips via dispatch). - Primitive cancel-safe (belt-and-suspenders): its handler is now `except BaseException`, so even a bare CancelledError drops the partial session and records the failed attempt before re-raising. - Operator waits: reconnect_sync / remove_server_sync default timeouts raised above the reconnect bound; remove_server_sync now CANCELS the pending _remove on timeout (so it can't later pop a re-added entry and corrupt state) and reports failure instead of a false 'removed' ([2], [4]). - in_flight defer is gated on defer_if_busy: autonomous callers (health loop, _refresh_all) defer, but a DISPATCH reconnects rather than hard-fail a reachable server with an in-flight sibling ([3]). - Cleanup: extract _schedule_next_ping (was a copy-pasted triplet in 3 branches) [5]. Adds 6 tests; the lock-contention test now shadows the caller-timeout constant so it runs in ~1s instead of the full wait. * fix(mcp): address PR review feedback on static reconnect - _ensure_static_connected: skip breaker record on CancelledError (cancel proves nothing about the server; aligns docstring with impl) - reconnect_sync: add asyncio.timeout wrapper so discovery-phase stalls get a clean TimeoutError inside the lock - reconnect_sync: change except Exception to except BaseException so CancelledError from future.cancel() triggers catalog cleanup - reconnect_sync: null state.session on failure so a tool-less session isn't mistaken for a live one - _static_reconnect_one: use fresh monotonic clock for backoff gate instead of stale tick-start snapshot; remove dead now param - _static_health_tick: correct docstring (0.5s clamp prevents busy-spin, not 'no sleep through short backoff') - Test: new test for reconnect_sync timeout + catalog cleanup - Test: update cancelled-attempt test for new CancelledError semantics - Test: narrow except BaseException to except Exception + type hints - Test: fix typo 'Understone' -> 'Turnstone' - Test: remove unused stale_now variable; update mock signatures * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Stefano Maffeis
|
bcf509a440 |
Drop dead last_ws_id/last_tool_call_id columns from mcp_pending_consent
Migration 054 added these columns but they were never populated (the sole writer hardcodes None) and never read by any query, dashboard, or API. Drop them so the schema matches reality. Closes #769 Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Patrick Buckley
|
acc262c405 |
feat(mcp): proactively keep consented OAuth (OBO) tokens fresh for unattended work
Per-user OAuth (auth_type=oauth_user) token refresh is entirely lazy: a token is refreshed only when a tool is dispatched or a session binds the acting user, and a dead refresh token is discovered only when a dispatch fails. That assumes a human is driving the session, which breaks for autonomous / scheduled work acting on behalf of an absent user — the token may be expired (latency), in a transient-failure cooldown (unavailable), or the grant may be dead with nobody present to re-consent. The only periodic MCP-loop task, idle eviction, actively tears OBO connections down; nothing keeps tokens warm. Add a background token-freshness sweep that keeps every consented oauth_user grant hot WITHOUT keeping connections warm and WITHOUT mutating consent state on a timer. Sweep (_user_token_sweep_loop, default 240s): - Enumerates consented (user, server) grants from the token store and runs the canonical refresh path for each. Strictly oauth_user-scoped: gates on _oauth_user_server_names and drives off mcp_user_tokens rows, so a static / no-auth server — which has neither — is structurally invisible (no DB scan, no authorization-server round-trip, no MCP-server call). Never connects to the MCP server; connections stay lazy. - Observe-only: passes revoke_on_failure=False (new parameter on get_user_access_token_classified) so a timer NEVER deletes a token, emits token_revoked, or mutates the shared ambiguous-streak / cooldown. A dead grant is only surfaced (proactive dashboard pending-consent badge); the authoritative revoke stays on the lazy-dispatch path where a real user action justifies it. Because the row survives, a spurious server-wide invalid_grant (an AS maintenance window) self-heals — the badge is dropped on the tick the grant works again. - Keepalive: force-refreshes a grant whose refresh token has sat un-exercised past user_token_refresh_keepalive_seconds (default 1800s) even while the access token is still fresh, so a provider that ages out idle refresh tokens can't expire one between a user's real sessions. - Surfaces dead grants once per transition, pinning the pair only after a durable badge write so a failed persist retries rather than being lost. First sweep runs after a short startup grace so a restart surfaces a downed grant within seconds, not a full cadence later. Cadence <= 0 disables the sweep; a positive value is floored (30s) so a misconfigured tiny cadence can't turn the loop into a busy-loop. Reuses the per-key refresh lock, so a keepalive force cannot double-refresh against a concurrent dispatch. Storage: add list_mcp_user_token_reconcile_targets() returning (user_id, server_name, COALESCE(last_refreshed, created)) — expiry-unfiltered, no ciphertext projected — on the protocol, sqlite, and postgres backends. Tests: the sweep's no-auth invisibility (zero DB / AS calls with no oauth_user server), observe-only non-destruction (token kept and shared streak untouched on a background permanent / ambiguous failure), keepalive gating, badge persist-then-pin retry, self-heal on recovery, cadence clamp / disable, and the storage enumerator. |
||
|
Patrick Buckley
|
62034378c6 |
fix(skills): address Copilot review — task_agent doc refs + gate fail-closed on missing storage
- task_agent.json referenced a non-existent `skill(action='search', query=...)` in two spots. The discovery tool is `skills` and the action is `find` (`search` is an activation value, not an action), so the guidance would mislead the model. Corrected both to `skills(action='find', query='...')`, matching the tool's own error strings. - _high_risk_skill_denied returned "" (allow) when get_storage() is None — an asymmetry with the fail-closed lookup-exception path added earlier. A risk gate that can't verify the tier must DENY, not wave the skill through, so storage-unavailable (None) now denies too; both paths share one denial. |
||
|
Patrick Buckley
|
bcb8c5ab88 |
fix(skills): harden task_agent / persona / skill activation from whole-PR review
Two independent multi-agent reviews of the branch (high, then max effort) found authority-confinement and robustness defects the per-step reviews could not see. This commit addresses every confirmed finding. task_agent turned out to be the surface that lagged its siblings on nearly every axis. Risk gate (most severe): - task_agent(skill=...) never enforced the high/critical-risk PRINCIPAL-load- only gate that skills(load) / spawn_workstream / spawn_batch enforce, so a model could route around it by delegating activation to a sub-agent. Enforce it inline in _prepare_task on the row already fetched (no re-query, no drift between get_skill_by_name and get_prompt_template_by_name). - _high_risk_skill_denied now fails CLOSED on a storage fault: deny, never wave the skill through. Denying (not returning "") also keeps spawn_batch's per-row partial-success intact under a transient blip. - (first round) extracted _high_risk_skill_denied onto spawn_workstream / spawn_batch, closing the coordinator-side bypass. Persona confinement (Principle 7 attenuation on the task_agent edge): - A restrictive persona now attenuates the sub-agent's TOOLS, not just its identity text — the tool lever is frozen into the item and filtered before _run_agent. - Honor ALL FOUR persona levers on the sub-agent, not two: a child persona's mcp-off and memory-off levers now drop MCP tools (mcp__* + read_resource / use_prompt) and the memory tool, matching a main session under the persona. - Cap the sub-agent by the PARENT session's own persona grant too, so a restricted principal cannot escalate authority by spawning. - Add persona to the task_agent judge/audit func_args projection (policy + audit parity with spawn). - Persona-resolution failures defer to a clean tool error (try/except mirroring _validate_child_persona) instead of an opaque "internal error". Substitution / capability: - substitute_args=False for capability contexts (defaults, task_agent) so a literal $ARGUMENTS / $N in a body is preserved, not blanked; env vars still resolve. The literal-$ARGUMENTS scan is deferred behind that guard (skipped on every capability render). - Drop the CLAUDE_SKILL_DIR alias (canonical TURNSTONE_SKILL_DIR only). That name also lives in bash, where turnstone-as-a-node-inside-Claude-Code must not shadow the host's value; claiming it in the prompt but deferring in bash diverged the two surfaces (a review finding). turnstone now claims it in neither surface. The CLAUDE_SESSION_ID / CLAUDE_EFFORT prompt aliases stay (pure prompt values, no bash-namespace collision). Skills-as-context: - DEFAULT (always-on) skills stay in the identity system message — the standing baseline, never a mid-session cache-bust; only a NAMED applied skill moves to the user-role capability message. This shrinks the pending model-adherence eval surface to the named-skill move alone. Cleanups: consolidate a duplicated rationale comment; correct the now-stale "task agents are not persona-filtered" note. PRE-MERGE GATE unchanged: the §7 Q1 model-adherence eval (named-skill move, this branch vs main) is not runnable in-tree and must clear before merge. |
||
|
Patrick Buckley
|
fec5067fcd |
feat(skills): gate model-initiated load of high/critical-risk skills
A skill can auto-fire tools (auto_approve + allowed_tools) once loaded, so letting the model activate a risky skill through skills(action='load') is an injection-steerable lane that widens authority behind a rubber-stampable approval. Deny it: high/critical-risk skills are now PRINCIPAL-load-only -- the model gets a clean error pointing the user at /skill, and the operator loads such skills explicitly (handle_command /skill and cli --skill call set_skill directly and bypass this gate). The scanner-computed risk_level is the gate signal -- it already escalates for the auto_approve + allowed_tools authority the create path warns about -- so no new column or migration is needed. The check can only DENY, never widen, so it is safe by construction (HYPOTHESIS.md Principle 7 / design section 5.5). Remaining step-5 follow-ups, out of scope here: persisting the literal disable-model-invocation frontmatter field for arbitrary author-marked skills (needs a column) and deprecating the vestigial variables mechanism. |