45 Commits

Author	SHA1	Message	Date
Patrick Buckley	cc84f9d176	fix(memory): harden project scope authorization and consistency	2026-08-11 21:58:21 -07:00
Patrick Buckley	480a1426b3	Fail-closed history-commit handoff (#1005) ... * fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964	2026-08-11 04:18:36 -07:00
Patrick Buckley	7a06f5e8bc	refactor(session): make ModelLane the provider boundary (#979) (#989) ... * refactor(session): make ModelLane the provider boundary (#979) ## Summary This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot. - Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding. - Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call. - Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references. - Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results. - Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts. ## Validation - SQLite suite: 11,188 passed, 9 skipped, 10 deselected - PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected - Live backend: 3 passed - SSE recovery: 6 passed; browser recovery harness passed all scenarios - Ruff: clean; 595 files correctly formatted - mypy: 243 source files clean - TypeScript: typecheck/build and 35 tests passed - OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte - `git diff --check` and Git LFS integrity clean Closes #979. * fix(deps): update nanoid for GHSA-2v37-7h3g-55p8 Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation. Validation: - npm ci - npm audit --audit-level=moderate: 0 vulnerabilities - TypeScript typecheck and build - TypeScript tests: 35 passed * fix(test): assert canonical model registry URLs Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation. Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.	2026-08-08 16:13:35 -07:00
Patrick Buckley	33ace975d2	feat(models): default-deny governance and admin UI for per-alias backend auth ... Follow-up to the per-alias Entra OBO/app-identity backend auth: the console write path now applies default-deny field classification, the admin shelf gains full backend-auth support, and the session/registry rebind machinery is hardened for config changes landing under live sessions. Console write gate: - Default-deny classification: any non-neutral change to a row that is or becomes dynamic requires admin.mcp plus validation; the provably auth-neutral columns are enumerated (MODEL_AUTH_NEUTRAL_FIELDS) and a live-schema classification test forces every future column to be classified. The derivation is a pure function (_derive_auth_gate) with unit-pinned exclusivity invariants. - Two-tier validation mirroring the MCP oauth_obo validator: the row tier (audience allow-list) runs on every gated write; the posture tier (OIDC configured, token store present) runs on pair changes and on enable-arming. - Pure-disable carve-out: disabling a dynamic row is de-escalation and is never blocked — admin.models suffices and validation is skipped, including for rows with corrupt or skewed stored values. - Capabilities are compared canonically (key order, integral floats), the audience compare normalizes both sides, and staging an audience on a static row is refused on both write twins. - Calibrate writes the capabilities column under an enforced confinement invariant with a compare-and-swap persist. Admin shelf: - Backend-auth section with a per-open constraints fetch (GET /model-definitions/auth-constraints: audience allow-list, grant profile, dynamic modes), datalist audience suggestions, server-defined modes preserved on round-trip, and permission-aware visibility built on cache-skew-safe helpers shared through auth.js. - Refused live-registry swaps surface as an amber registry_warning on the write, delete, reload, and calibrate responses; audit rows carry auth_gated / auth_disarmed markers visible in the audit view. Registry and sessions: - The encryption-key requirement for dynamic auth is enforced inside ModelRegistry.reload() itself — nodes refuse with 503 and the console records coord_registry_error — and reload bumps the generation before the map swap so a racing reader can never pair a stale generation with new maps. - resolve()/resolve_binding() return the generation from inside the registry lock; sessions rebind per send on generation change with atomic client/provider/config commits, fallback-first handling of removed or unconstructable aliases, and judge/limiter resets only when the binding actually changed. - Mint refusals record per-user causes surfaced in the per-turn heartbeat logs; misconfiguration warnings are deduplicated with bounded state. Verification: 10417 tests (99 added on this branch), a 71-scenario browser harness over the real admin shelf, and a live rfc8693 token-exchange e2e run (MCP legs verified end to end; the model-leg scope gap is tracked as #955 under a narrow known-gap signature). Closes #950.	2026-08-03 20:11:28 -07:00
Patrick Buckley	1224b02d03	fix(compaction): review round 8 — seam obligations become primitives ... Eight rounds of findings against the defer-and-drain seam shared one generator: N sites each hand-copying M obligations (spawn discipline, the order-barrier pair, backpressure, best-effort emission, the client settle matrix), with every review finding an empty (site x obligation) cell. This round makes each obligation a single primitive: - The order barrier is Workstream.send_barrier_active() — one definition of the two-term pair (pending entries OR drain alive), consulted by the /send route, the coordinator adapter, and the queued-nudge wake gate, which previously carried only the list term and let a synthetic wake jump an acknowledged send during the claimed-entry window. _PendingSend moved to workstream.py beside the invariant that justifies the drain-alive term; the pending fields got precise types and worker_kind became a Literal, so a typo'd "command" comparison is now a type error instead of a silently never-firing defer guard. - _defer_send probes the barrier before constructing anything, bounds acceptance at 10 pending (the interjection queue's own backpressure contract — unbounded acceptance pinned message + attachment bytes per entry for a whole command window and then ran one unattended turn each), and spawns the drain with rollback: a Thread.start failure pops the just-accepted entry and answers the retryable queue_full instead of 500ing after registration (a phantom the client could neither see nor retract, dispatched later as duplicate turns). start() deliberately stays inside the lock, unlike session_worker's outside-lock discipline: this slot is is_alive()-gated, false for a constructed-but-unstarted thread, so an outside-lock start would open a double-drain window. - A /command whose worker never spawned answers 503 {"status": "error"} (spec + docs + a pane error arm) instead of the generic 200 ok that told SDK callers their /clear ran. - The compaction lifecycle emitter is raise-proof at its single dispatch tail: a raising duck-typed hook degrades to a lost render, never a lost end event — previously a raising on_error or a raising failed-end emit left every pane a frozen progress bar, and a raising SUCCESS end after the committed swap fabricated a failed end. - The client settle matrix lives once: composer_queue's settleSendResponse owns every /send response arm for both panes (the near-verbatim twins were already drifting), parsePriority is shared, and the busy stamp is centralized in setBusy(b, source) with "server" as the fail-safe default. Deferred sends release the composer (no worker exists for them; retracting the chip no longer strands the pane in Stop mode), queue_full on an idle-looking pane removes the optimistic bubble and restores busy (the refusal can now fire with no worker and no drain to ever emit a state event), and the pre-bind settle buffer is TTL-based — a burst of deferred dispatches parked this tab's own raced settle first, where the old size cap evicted exactly it. - The command backstop / console proxy timeout inequality is enforced by a test importing both named constants (both proxy_client constructions, startup and the mTLS re-create); the compaction card wears blue (magenta is reserved for the MCP surface); the redundant TerminalUI.on_compaction override is gone (the inherited protocol default is the policy site).	2026-07-17 11:28:44 -07:00
Patrick Buckley	fd5d3efb43	fix(compaction): review round 7 — drain crash/order/settle rows, protocol-default fallback, bool event-id guard ... Completes the defer-and-drain seam against the matrix rows round 6 never enumerated (the defer contract itself took no hits): - Crash row: a claimed entry survives a dispatch crash — the per-iteration handler re-inserts it at head (claim-flagged so a claim-section failure can't duplicate it), backs off ~1s, retries. The last-resort handler spawns no successor (Thread.start fails under the exact exhaustion that reaches it): the route's ensure-drain stays the single spawn site, so single-flight is structural and a dead drain revives on the next defer. - Order row: the pending list is the order authority. The /send route pre-checks pending/drain-alive under the same lock acquisition that appends (one _defer_send helper serves the barrier and command-window triggers); the coordinator adapter refuses via its return value; the queued-nudge wake gate yields to pending sends and is re-armed by the drain's clean exit — which covers lists emptied by pure retraction — as well as every deferred turn's exit; retry-after-rewind is a documented accepted overtake; init/create is fresh-ws-by-construction. - Client settle row: queued responses carry "deferred": true (SendResponse + regenerated openapi-server.json, status enumeration completed); bind(el, msgId, {deferred, attachedCount}) replaces the _deferredAttachments expando; the idle sweep skips deferred and unbound chips; the shared dispatch attempt emits pane-tier message_dispatched (folded: true for interjection fold-ins — the chip clears only its deferred flag and keeps a live x while DELETE still genuinely retracts); settles that beat bind() park in a bounded buffer; an idle-thinking pane retro-converts its optimistic bubble into a real queued chip instead of presenting a parked message as sent. Rejection polling waits on the slot flags — one dispatch attempt per slot-state change, not 4 Hz. - SessionUI.on_compaction's protocol stub became a real default body (the classic on_info rendering): explicit subclasses inherit protocol members as real methods, which defeated _compaction_event's getattr fallback for exactly the pre-1.8 embedders it serves. - _coerce_event_id() rejects bools (isinstance(True, int) is True) at all three duck-typed event-id coercions: the compaction marker stamp, on_system_turn's persisted return, and _ui_event_id. - Quick-command backstop 60s -> 25s, under the console proxy's 30s so the degraded "running" answer can traverse a proxied pane (which now surfaces it); /resume docs drop the fictional history SSE event (clear_ui + REST re-fetch is the contract); /send response docs match the wire.	2026-07-17 11:28:44 -07:00
Patrick Buckley	1dbf7f410c	feat(compaction): lifecycle events, web progress card, history re-render ... Compaction becomes visible: a first-class 'compaction' SSE lifecycle (start/progress/end, compaction_id-correlated, superseded-flagged ends) replaces the loose info lines; both web panes render a progress-bar card that settles into a persistent result card, re-rendered after reload via the /history projection of the compaction marker row. Slash commands echo as command chips instead of fake user turns. The enabling rework: /command dispatches onto the workstream worker slot (the old inline path blocked the node's event loop for whole compactions and let /clear interleave with live turns). Busy refusals answer 409; quick commands are awaited loop-natively with a 60s backstop; /compact is fire-and-forget. Sends during a command window park in the /send route and dispatch full-fidelity afterwards — the interjection queue (length cap, cross-user guard, identity-swap hazards) is unreachable there — with a compaction-aware client abort bound shared by both panes. compact_now() carries send()'s full generation discipline; Stop aborts the in-flight summary HTTP stream via a generation-scoped cancel ref; force-abandoned compactions retire at their next checkpoint and their stragglers are fenced off every surface (panes, pill latch, CLI). Every session retry backoff is cancel-aware via one shared helper. Docs, OpenAPI spec, and both SDKs updated. Verified: 9457-test non-live suite, JS pin suites, headless-Chrome reducer harness; five unprimed multi-agent review rounds (correctness trend 15/6/6/4/4) with plan-level design passes on every fix round.	2026-07-17 11:28:44 -07:00
Patrick Buckley	fa1ba2cc01	fix(api): type initial_message_status as a Literal enum ... str | None under-specified the field: the implementation and the TS SDK union both constrain it to queue_full / refused_closed, and the Literal projects a proper enum into the generated OpenAPI spec so clients reject unexpected values. Specs regenerated.	2026-07-07 16:42:36 -07:00
Patrick Buckley	e60c19befd	fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds ... Wake path: - Denial metacog nudge moves to the tool channel so it drains with the denied tool batch instead of the next user-message seam. - wake_workstream_if_pending: shared wake gate for watch fires on already-idle workstreams (no IDLE transition for the watcher to observe), wired as wake_fn at every set_watch_runner site via the shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT — after eviction+restore an id-keyed manager lookup would miss). - session_worker exit backstop re-runs the wake gate the moment worker ownership clears: IDLE fans out on the worker thread, so transition-time wakes always landed on the reuse path and no-op'd (the coordinator idle_children strand). - deliver_wake_nudge_from_queue contains GenerationCancelled — it is the wake worker's run() closure and only Exception is caught downstream. Watch delivery: - Terminal fires that cannot reach their workstream are HELD and redelivered on min(interval, 60s) without re-running the command, bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own max_polls across cycles; the poll charge commits durably at hold time so restarts stay budget-bounded. - Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES cap, presence-only re-check under the lock, detection-only stall alerts (reclaiming a wedged admission would trade capped degradation for total poll-pool collapse). - Permanent-vs-transient restore taxonomy: corrupt persona stamp and genuinely-missing history (confirmed by a raising storage probe — the resume loader swallows read blips into []) deactivate the watch immediately; everything else holds and retries. - Cancel-race defense: delivery paths re-check is_watch_active before stashing/dispatching, cancel paths write the row BEFORE forget_terminal_dispatched, the HTTP cancel endpoint clears runner state, and a per-tick sweep bounds the residual stash-after-clear interleaving to one check_interval. - Abandon/exhaustion commits are write-then-clear so storage that can read but not write retries the row write instead of re-running the command every cycle; the fresh-fire unrestorable path stashes before its deactivation write for the same reason. Registry follows identity: - The dispatch registry is keyed by _ws_id at registration time; every rebind now moves it: non-fork resume() and /new go through _follow_watch_registration (new key live before the old is removed, never stealing a registration another live session holds), removals are owner-checked so tearing down a watch-restore shell or a resumed-away session cannot unregister a live pane, the restore shell yields to a registration that appears mid-restore, CLI --resume registers after the successful resume, and both the open path and the detail-GET lazy rehydrate wire the registration. Teardown gating and backpressure honesty: - cleanup_session_ui marks ws._closed FIRST under ws._lock — every teardown path (close, close_idle, evict, delete, discard) funnels through it — and session_worker.send re-checks under the same lock, so a wake can never spawn a worker on a torn-down workstream. - Create responses carry initial_message_status when the initial message could not be delivered (queue_full / refused_closed) instead of reading as success; staged attachments survive for the retry; /send surfaces a closed workstream as 404 rather than queue_full. Docs/spec: OpenAPI artifacts regenerated; api-reference documents the new create-response field; TS SDK type extended. Tests: ~30 new pins (cancel races, budget durability across restarts, owner-checked registry moves, teardown gating, stall alerts, backpressure surfaces, wait_until final re-check); wide subsystem sweep green (2353 passed).	2026-07-07 16:42:36 -07:00
Patrick Buckley	8da5dc3f5a	docs(api): regenerate OpenAPI specs for the details-list contract ... The checked-in openapi-server.json / openapi-console.json still described the removed singular pending_approval_detail field; re-run generate-types.py so the reference specs carry the pending_approval_details list + cycle_id. Retire two docstring references to the deleted singular serializer and one stale comment.	2026-07-05 01:57:54 -07:00
Patrick Buckley	54ebb24374	feat(personas): core stamping, four-lever application, create/spawn/SDK threading; remove /creative; add --persona ... The persona resolved at creation is snapshotted into workstream_config (five keys, all-or-none) and applied ONLY from the stamp — the personas table is never read post-create, so edits/archives never touch existing workstreams, and a corrupt stamp fails construction loudly instead of silently reverting to a default envelope. Legacy pre-063 workstreams carry no keys and keep today's behavior byte-for-byte. The four levers (turnstone/core/personas.py holds the codec): 1. Base override — compose_system_message(base_override=...) replaces exactly the BASE module; ENV/CONTEXT/TOOLS/POLICIES keep composing so mandatory prompt policies ride on top of every persona. This also closes the old /creative hole where the fork bypassed composition (no CONTEXT, no DB policies). 2. Tool visibility — the allowlist intersects both the composition name set (TOOLS block self-suppresses, tool-gated policies drop, the memory advisory drops) and the END of _get_active_tools so the wire never advertises hidden tools. tool_search in the set = soft (discovered tools union with the allowlist via the session's expanded-names set); absent = hard (the whole pathway is disabled, covering provider-native defer_loading, which has no synthetic name to filter). Persona sets force client-side tool search. 3. MCP gate (session-wide) — an MCP-off persona drops the client reference at construction: no merge into _tools OR _task_tools, no listeners, refresh callbacks inert, resource/prompt catalogs gone. 4. Memory (own hands only) — no recall injection, memory-directed nudges suppressed (MEMORY_NUDGE_TYPES; behavioural nudges keep firing), memory tool hidden. _task_tools is NOT filtered; compaction spill/markers are never persona-gated, and the post-compaction recall pointer is emitted only when the recall tool is actually visible. Threading: the create handler resolves once (explicit name -> 400 on unknown/disabled/kind-mismatch; empty -> the kind's default; pre-seed DB -> unstamped legacy) and stamps via constructor kwargs + config keys + the workstreams.persona column; SessionManager.open threads the stamp pre-construction exactly like the saved model alias. Non-fork resume adopts the target's stamp so _save_config can't clobber it. spawn / spawn_batch gain a persona arg with prep-time validation (children are interactive-kind; omitted = kind default, never the parent's). Python + TS SDKs, OpenAPI specs, the picker feed GET /v1/api/personas (authed, no perm), and console admin CRUD /api/admin/personas (persona.* perms, archive-only — no DELETE) round out the surface. BREAKING: /creative is removed (the REPL command now points at the writer persona); turnstone --persona <name> is the replacement. Also fixes the CLI session factory, which TypeErrored on the project_id kwarg the shared InteractiveAdapter passes unconditionally.	2026-07-03 00:29:26 -07:00
Patrick Buckley	2169559d6e	feat(projects): governed project containers — memory scope, grouping, manage UI (#724) ... * feat(projects): governed project containers — memory scope, grouping, manage UI A workstream can attach to a project: a first-class, shareable resource container that owns a `project` memory scope, groups conversations, and is managed from the console. Storage / migration 062: projects + project_members tables, workstreams. project_id, and the memory type default project→general; grants project.{create,read,write,delete} (admin-default). Recall + writes: project memory is recalled iff the workstream is attached AND the user has access (owner ∨ member ∨ public-for-read), resolved once at session construction; coordinators recall it too. New saves default to the project when attached + writable; the save and delete paths are write-gated; deleting a project purges its scoped memory; archived projects aren't recalled. Access = RBAC capability ∧ per-project ACL (auth.resolve_project_access, a single-fetch resolver); visibility changes, member management, and delete are owner-only. API: project CRUD routes on both the server and console; project_id threaded through workstream creation, spawn inheritance, the cluster-create proxy, the dashboard / snapshot / coordinator row builders, and the collector deltas. UI: a project picker with an inline "+ New project" creator in every creation box (console launcher + standalone dialog + dashboard); group-by-project in the rail; a project badge in the composer and on dashboard rows; a console manage tab (list + create/edit + members shelves). The admin Memories view gains coordinator/project scope filters and human scope labels (name, not hex). The memory tool schema documents the project scope and the attach-aware default. * fix(projects): client refresh hardening, creator race guard, SDK project_id Addresses PR #724 review feedback plus two bugs found while validating it. - projects.js refreshProjects: a non-OK status (e.g. 403 when the caller lacks project.read) or a network/parse error no longer blanks the cache or masquerades as "no projects" -- the prior cache is preserved, the failure is recorded (new projectsError()) and warned. Honors the long-standing "a transient error can't blank the rail" docstring. - projects.js _fp: the fingerprint separators were raw control bytes, which made git treat the whole file as binary (no reviewable diff). Rewritten as escape sequences instead of raw bytes -- behavior is byte-identical at runtime. - project_creator.js: createProject() could reject unhandled (authFetch throws on network/401; r.json() throws on a non-JSON body), leaving the widget stuck busy/disabled. Added a .catch, plus a generation guard so a create whose widget was cancelled/reopened mid-flight drops its result instead of selecting a project the user backed out of. - types.ts: add project_id to CreateWorkstreamRequest / WorkstreamInfo / DashboardWorkstream to match the server schemas (was SDK-invisible). - test_project_api.py: move side-effecting HTTP calls out of asserts so the requests run even under python -O. * fix(projects): JSON.stringify the cache fingerprint, drop control-byte separators _fp joined fields/rows on raw NUL/SOH bytes, which made projects.js read as binary to git. Replace with a collision-proof, escape-free JSON.stringify encoding -- same change-detection semantics, zero embedded control characters.	2026-06-26 17:24:06 -07:00
Patrick Buckley	110d44b07e	refactor(tools): remove man, math, and plan_agent built-in tools ... `man` and `math` duplicated capabilities already reachable through `bash`; `plan_agent` is better expressed as a `task_agent` running a planning skill, and carried a large amount of special-case machinery (plan-review gate, refinement loop, per-kind model routing). Removing all three shrinks the tool surface and cuts per-call token cost. Also removed, as dead-once-the-tools-are-gone: - the `math` sandbox executor (`turnstone.core.sandbox`) and its `[sandbox]` extra; the eval analyst now runs bash-only - the read-only `AGENT_TOOLS` sub-agent tool set and the `agent` tool-metadata key (`task_agent`/`TASK_AGENT_TOOLS` retained) - the plan-review protocol end to end: the `on_plan_review` UI hook, `resolve_plan`, `POST /v1/api/plan` + `POST /v1/api/route/plan`, the `plan_review`/`plan_resolved` SSE events, and their Python SDK / TypeScript SDK / OpenAPI / frontend / Discord+Slack bindings - the `model.plan_alias` / `model.plan_effort` settings and the registry `plan_model` / `plan_effort` routing fields TOOLS 31->28, TASK_AGENT_TOOLS 13->11; COORDINATOR_TOOLS unchanged. BREAKING CHANGE: removes the `man`, `math`, `plan_agent` tools, the plan-review SSE/HTTP/SDK surface, and the plan_* model-routing settings from the experimental 1.6 line.	2026-05-31 19:54:43 -07:00
Patrick Buckley	c80354880a	feat(api): enrich saved-workstream list with model/skill/context fields ... GET /v1/api/workstreams/saved returned only ws_id/alias/title/created/ updated/message_count — too little to drive the planned saved-list table redesign. Add seven fields, all sourced from already-persisted data (no migration): - state, kind, node_id: columns on the workstreams table - model_alias, launch_skill: from workstream_config via LEFT JOIN - child_count: COUNT of child workstreams via parent_ws_id - context_tokens: most recent usage_events prompt size for the workstream - context_ratio: context-window occupancy (context_tokens / model context window), computed in the handler so the NULL / zero-window cases stay explicit and identical across both storage backends context_window comes from a model_definitions join; aliases defined only in config.toml are absent there, so context_ratio degrades to 0.0 rather than reporting bogus occupancy. The Python SDK reuses the Pydantic model; the TypeScript SDK OpenAPI snapshot and hand-maintained interface are updated. Tests cover the new storage columns (including NULL-when-absent), the handler ratio math + zero-window degradation, and the SDK enriched round-trip.	2026-05-29 16:05:17 -07:00
Patrick Buckley	3e5633f440	chore(sdk): regenerate OpenAPI snapshots from current specs ... openapi-server.json / openapi-console.json had drifted well behind build_server_spec() / build_console_spec() — the committed snapshots are regenerated periodically (via sdk/typescript/scripts/generate-types.py) rather than on every schema-changing PR, so accumulated additions (skill parsing, pending-approval items, model-definition CRUD, etc.) had not been captured. This resyncs both with no code changes.	2026-05-29 12:35:51 -07:00
Patrick Buckley	dea2729292	refactor(coordinator): rename task_list → tasks, doc/prompt sweep (#437) ... Four themes from a coordinator-feature shakedown: 1. Correctness fixes (return shapes / examples / behavior) - tools_coordinator.md: drop fake skill names from spawn examples; fix wrong kwarg ``node_id=`` → ``target_node=``. - wait_for_workstream.json: document ``message`` + ``truncated`` per-ws fields (always enriched in the client; the JSON shape lagged the docstring). - cancel_workstream.json: document the conditional ``dropped`` payload — ``was_running`` always present when ``dropped`` is, ``pending_approval`` and ``queued_messages`` conditional sub-shapes. - spawn_workstream.json: document full return shape including ``routing_strategy ∈ {rendezvous, target_node, resume}`` and ``status``. - close_all_children.json: clarify ``skipped`` covers BOTH hard-deleted children AND already-closed-and-evicted children (wire shape doesn't distinguish); drop incorrect "echoed back in response" claim — server returns ``{status, closed, failed, skipped}``, never echoes ``reason``. - console/server.py: comment in ``_fanout_on_children`` clarifying that the 400 "No session" branch fires for cancel-cascade callers and is unreachable from close_all_children (close handler 404s instead). - coordinator_client._utc_now_iso(): switch to bare ISO format matching the rest of the storage row format used in the codebase. 2. Tightened the 11 longest tool descriptions (~23% cut on the coord set). Removed ALL-CAPS emphasis, normalised em-dashes, dropped informal phrasing. No new claims. 3. Removed static approval annotations from descriptions. Approval is governed at runtime by the unified ``approve_tools`` body and admin-defined ``tool_policies`` (#436); static "Auto-approved" / "Approval required" / per-action approval tags become a stale signal. Field names (``pending_approval``) and operational verb behaviour ("cancel unblocks pending approvals") stay. 4. Renamed ``task_list`` coord tool → ``tasks``. The previous name compounded the bare word ``task`` (which collides with chat-template channels on local models — same reason ``task_agent`` carries the suffix); the plural form sidesteps the collision and reads more accurately, since the tool acts on the whole list rather than a single task. Sweep covers tool JSON, Python methods (5 client methods + 2 session methods + 1 helper + 1 constant), audit event name (``task_list.update`` → ``tasks.update``), log tag (``task_list.corrupt_envelope`` → ``tasks.corrupt_envelope``), frontend SSE event matcher, prompts, docs, and tests. CHANGELOG entry added. Plus: dropped the ENV block (Output Environment / Available rendering / Formatting principles) from coordinator system prompts. Coordinators orchestrate rather than render rich output to the user, so the rendering capability matrix is not actionable for them. Coord prompt drops ~29% (6309 → 4493 chars). SDK regeneration via ``generate-types.py`` updates both ``openapi-console.json`` (the rename's downstream change) and ``openapi-server.json`` (PR #436 drift — its merge added ``pending_approval_detail`` + ``recent_auto_approvals`` fields to the Python schemas but didn't regenerate the JSON artifact). ## Behavior changes (operator-visible) - Audit event name: ``task_list.update`` → ``tasks.update``. Audit dashboards / SIEM filters / log greps that pinned the old prefix should update. - SSE ``tool_result`` events now ship ``name="tasks"`` for the scratchpad tool. The bundled coord-tree UI is updated atomically; external consumers reading SSE events by tool name need to update. - Existing task envelopes in production storage have ``+00:00`` timestamps from the old ``_utc_now_iso``. New writes are bare; old rows are not backfilled. Within an envelope you may briefly see mixed formats until each row is re-touched. No code path string-compares timestamps within an envelope, so this is cosmetic. ## Validation - ``ruff check`` + ``ruff format --check`` clean - ``mypy turnstone/`` clean (175 source files) - ``pytest -m "not live"`` — 4679 passed, 3 deselected	2026-04-27 22:51:31 -07:00
Patrick Buckley	b8e51fa9ed	fix(api): add DequeueRequest schema for DELETE /workstreams/{ws_id}/send ... Copilot review on PR #422 flagged that the DELETE-on-send (dequeue) EndpointSpec declared no request_model, so the generated OpenAPI showed no requestBody for an operation that *requires* a JSON body with ``msg_id`` and 400s when it's missing. - Add ``DequeueRequest`` to ``server_schemas.py`` with the single required ``msg_id: str`` field. - Wire ``request_model=DequeueRequest`` and ``response_model= StatusResponse`` on the DELETE EndpointSpec; trim the now-redundant inline body example from the description. - Re-import the schema in ``server_spec.py`` and add the entry to ``_ALL_MODELS`` so the OpenAPI components list carries it. - Regenerate ``openapi-server.json``. Sibling thread on the close EndpointSpec was already addressed in 4000ae2 (request_model=CloseWorkstreamRequest). 4558 tests passing under ``-m "not live"``; ruff + mypy clean.	2026-04-26 22:14:22 -07:00
Patrick Buckley	5874159ffd	fix(close): require non-empty body, restore CloseWorkstreamRequest ... Copilot caught three real issues in PR #422 review, all clustered around the close request body contract: 1. The interactive close handler runs with ``supports_close_reason=True``, which calls ``read_json_or_400(request)`` — an empty / non-JSON body returns ``400 {"error": "Invalid JSON body"}``. The previous SDK fix sent NO body via ``json_body=None``, which would 400 against a real server. The mock-transport test silently masked it because the mock answered without inspecting the body. 2. The doc said the body was empty (or ``{}``), with no mention of the optional ``reason`` field, its 512-byte cap, or the credential-redaction guard. 3. The Pydantic schema for close was deleted outright; OpenAPI and SDKs lost their typed shape for the optional ``reason``. Changes: - ``turnstone/api/server_schemas.py``: reintroduce ``CloseWorkstreamRequest`` with a single optional ``reason: str | None = None`` field. Docstring documents the must-be-valid-JSON contract and notes that coord ignores the body (``supports_close_reason=False``). - ``turnstone/api/server_spec.py``: re-import the schema, point the close ``EndpointSpec`` at it via ``request_model=``, restore the ``_ALL_MODELS`` entry. OpenAPI JSON regenerated. - ``turnstone/sdk/server.py``: ``close_workstream`` (sync + async) gains an optional ``reason: str | None = None`` parameter and always sends ``json_body={}`` (or ``{"reason": ...}``) so the body is never empty. Adds a regression test (``test_close_workstream_sends_valid_json_body``) that inspects the raw transport content rather than relying on a path-keyed mock — the kind of check that would have caught this bug pre-merge. - ``sdk/typescript/src/server.ts``: ``closeWorkstream`` gains an optional ``opts.reason`` parameter; reintroduce ``CloseWorkstreamRequest`` interface in ``types.ts`` and re-export from ``index.ts``. - ``docs/api-reference.md``: close section documents the JSON-body requirement, the ``reason`` field, the 512-byte cap, the multibyte-safe behavior, the credential-redaction guard, and the non-string-coercion path. - ``CHANGELOG.md``: amend the 1.5.0 BREAKING block to reflect the schema reintroduction (slim form, ``reason`` optional) instead of the prior "removed outright" claim. 4558 tests passing under ``-m "not live"`` (was 4557 — +1 from the regression test). ruff + mypy clean.	2026-04-26 22:14:22 -07:00
Patrick Buckley	ad0e7ce6eb	docs: mark 1.5.0 legacy URL surface removal ... CHANGELOG [Unreleased] / Removed (BREAKING — 1.5.0) block calling out the legacy URL family removal with the swap table. Doc passes on api-reference.md (per-endpoint sections rewritten with path parameters and slimmer body shapes), architecture.md (handler-list diagram and console-proxy URL example), console.md (URL-rewriting JS shim docstring + SSE proxy example), and the two PlantUML diagrams (11-console-data-flow, 16-channel-architecture). Also picks up two test-side stragglers from step 5 that referenced the legacy adapters in a docstring + a stale /v1/api/events SSE test: turn into path-keyed equivalents. OpenAPI JSON dump regenerated to reflect the catalog edits from step 3. After this commit: - 4557 tests passing under -m "not live" - ruff + mypy clean on turnstone/ tests/ sdk/ - grep for "/v1/api/send", "/v1/api/approve", "/v1/api/cancel", "/v1/api/workstreams/close" returns zero hits across turnstone/ sdk/ docs/ tests/ (excluding CHANGELOG.md, which intentionally documents the old shape). - grep for make_legacy_body_keyed_adapter, make_legacy_query_keyed_adapter, _make_method_dispatch, close_legacy returns zero hits.	2026-04-26 22:14:22 -07:00
Patrick Buckley	6572437c5d	refactor(server): rename dashboard row id → ws_id for v1 row-shape consistency ... The /v1/api/dashboard endpoint was the last workstream-listing surface keyed on `id` rather than `ws_id`. The Stage 2 list-verb lift converged the active list (`/v1/api/workstreams`) and saved list (`/v1/api/workstreams/saved`) on `ws_id` but explicitly left dashboard alone to keep that PR's diff focused. This lands the same rename on the remaining endpoint so v1 row shape is consistent across the family. Scope kept narrow: - Pydantic `DashboardWorkstream` and TS SDK `DashboardWorkstream` interface both rename `id: str/string` → `ws_id`. - The bundled web UI (`turnstone/ui/static/app.js`) is the only consumer reading `dashboard.workstreams[].id` and is updated atomically. - Console `_fetch_live_block` (cluster-inspect's projection over a remote node's dashboard payload at `turnstone/console/server.py`) flips its `entry.get("id")` lookup to `entry.get("ws_id")`. - Drive-by: stale `id` example in `docs/api-reference.md` for the earlier `/v1/api/workstreams` rename also fixed. `_build_node_snapshot` (the global-events SSE node_snapshot payload consumed by the cluster collector) deliberately stays on `id` — it's part of a separate cluster-row family (collector → cluster_workstreams → console UI) that is internally consistent on `id` and would need its own coordinated sweep. CHANGELOG documents the bounded blast radius. Tests: 4554 passing (-m "not live"). ruff + mypy clean.	2026-04-26 20:00:45 -07:00
Patrick Buckley	acbe18d5f5	docs: apply Copilot review feedback on PR #419 ... Server-side history endpoint declared ``error_codes=[404]`` but the lifted ``make_history_handler`` factory can also return: - ``400`` on empty ``ws_id`` (defensive — Starlette routing makes it unreachable in practice, but the factory has the branch). - ``500`` on the ``cfg.list_kind is None`` misconfig gate added in the /review fix-up (defense-in-depth fail-loud; both production cfgs wire ``list_kind`` so the gate doesn't fire today). - ``503`` via ``cfg.manager_lookup`` when the kind's manager isn't available (interactive's lookup never returns 503; coord's can). Updated ``server_spec.py`` to ``[400, 404, 500, 503]`` per Copilot's suggestion — matches the existing detail entry's shape so the two endpoints document the same possible-error envelope. Caught the parallel asymmetry on ``console_spec.py``: history was ``[403, 404, 503]`` but the lifted factory's misconfig + empty- ws_id branches reach coord too. Updated to ``[400, 403, 404, 500, 503]`` — same factory body, same possible responses, plus ``403`` from coord's ``admin.coordinator`` permission gate. Regenerated ``openapi-{server,console}.json``. No code changes; spec metadata only. Tests + lint + mypy unchanged.	2026-04-26 15:48:26 -07:00
Patrick Buckley	d555816016	refactor(core): lift history + detail verb bodies across both kinds (Stage 2 verb lift) ... Last verb-shape lift before v1.5.0 stable can tag. Adds two new factories to ``turnstone/core/session_routes.py``: - ``make_history_handler(cfg)`` — body lifted from coord's ``coordinator_history`` near-verbatim. ``?limit=`` query param defaults to 100, clamps to [1, 500], malformed values fall back to 100. Storage operations (``get_workstream`` on the storage-fallback path, ``load_messages`` for the row read) now run via ``asyncio.to_thread`` (was inline pre-lift on coord). - ``make_detail_handler(cfg)`` — body lifted from coord's ``coordinator_detail``. Lazy-rehydrates a closed/evicted workstream via ``mgr.open()`` on miss; mirrors :func:`make_open_handler`'s exception envelope (``ValueError`` → 503 with the session-factory's remediation text; bare ``Exception`` → correlation_id'd 500 with the per-kind noun via ``cfg.audit_action_prefix``). NO new ``SessionEndpointConfig`` fields — the factories reuse ``permission_gate``, ``manager_lookup``, ``not_found_label``, ``audit_action_prefix``, and (for history's storage-fallback kind check) ``list_kind`` — all already wired by both production lifespans for the list/saved factories. Coord side: ``coordinator_history`` and ``coordinator_detail`` standalone handler bodies removed from ``console/server.py``; ``register_session_routes`` now wires ``history=make_history_handler(coord_endpoint_config)`` and ``detail=make_detail_handler(coord_endpoint_config)``. Interactive side: GAINS both endpoints as a feature gain. Pre-lift interactive had no ``GET /v1/api/workstreams/{ws_id}`` and no ``GET /v1/api/workstreams/{ws_id}/history`` — SDK consumers had to subscribe to ``/events`` SSE just to read display fields or message rows. The same lifted factories are wired with the interactive endpoint config; cross-kind isolation is preserved on both sides (history via ``cfg.list_kind`` storage-fallback gate + fail-loud-on-misconfig 500; detail via ``mgr.open()``'s internal kind check). Pydantic schemas: ``CoordinatorDetailResponse`` / ``CoordinatorHistoryResponse`` removed from ``console_schemas.py``; ``WorkstreamDetailResponse`` / ``WorkstreamHistoryResponse`` added to ``server_schemas.py`` (mirrors the list lift's pattern for ``WorkstreamInfo``). Both server and console OpenAPI specs reference the unified schemas; ``server_spec.py`` gains ``EndpointSpec`` entries for the new interactive endpoints. TS SDK gains both interfaces in ``sdk/typescript/src/types.ts``; ``openapi-{server,console}.json`` regenerated. Tests: 6 new coord regression/parity tests in ``test_coordinator_endpoints.py`` (limit clamping, cross-kind 404 on storage fallback, storage-only history, detail 503 on session-factory misconfig, detail 500 with correlation_id on unexpected rehydrate failure, history swallows ``load_messages`` exception → 200 with empty messages). 10 new interactive parity tests in ``test_workstream_endpoints.py`` (``TestHistoryInteractive`` + ``TestDetailInteractive``). 1 new openapi spec test pinning the server-side ``?limit=`` query param. Total: ``4490 → 4491`` after the new exception-swallow regression test landed. ``ruff check`` clean, ``mypy`` clean on touched files. /review pipeline (4 finders → verify → dedupe) caught 1 Minor defense-in-depth (bug-1/sec-1, merged: ``make_history_handler`` fail-closed gate when ``cfg.list_kind is None``, mirroring ``make_saved_handler``'s same gate) + 1 Minor test-helper rename (q-1: ``_interactive_history_cfg`` → ``_interactive_endpoint_cfg``) + 4 Nits (q-2 unused fixture parameter, q-3 CHANGELOG TS SDK mention, q-4 missing exception-swallow regression test, q-5 misleading test comment) — all addressed in the same commit.	2026-04-26 15:48:26 -07:00
Patrick Buckley	edf52016ac	refactor(core): lift list + saved verb bodies across both kinds (Stage 2 verb lift) ... New ``make_list_handler(cfg)`` and ``make_saved_handler(cfg)`` factories in ``turnstone/core/session_routes.py`` replace four pre-lift bodies (interactive ``list_workstreams`` + ``list_saved_workstreams``; coord ``coordinator_list`` + ``coordinator_saved``). Same factory + capability-flag pattern as the merged cancel / open / events / create lifts. Four new ``SessionEndpointConfig`` fields: - ``list_resolve_titles: ListResolveTitles | None`` — bulk lookup ``(ws_ids) -> {ws_id: title-or-None}``. Interactive wires ``get_workstream_display_names`` (new bulk helper added on the storage layer + memory.py); the lifted body resolves every active row in ONE ``SELECT ... WHERE ws_id IN (...)`` instead of the pre-lift N+1 (one SELECT per row). - ``list_kind: WorkstreamKind | None`` — explicit kind classifier for the saved-list storage filter. Replaces the initial draft's ``audit_action_prefix == "coordinator"`` string compare which would have silently leaked INTERACTIVE rows for any future kind whose audit prefix didn't match. Required when a kind mounts list/saved; misconfig surfaces as a 500 with a clear log line. - ``saved_state_filter: str | None`` — coord wires ``"closed"``; interactive wires ``None``. - ``saved_loaded_lookup: SavedLoadedLookup | None`` — coord-only defence-in-depth filter that excludes ws_ids in the warm pool. Behaviour changes (all observable in CHANGELOG): - **Active-list row shape converges on always-include** ``{ws_id, name, state, kind, parent_ws_id, user_id}``. Interactive renames ``id`` → ``ws_id``; both kinds populate every field (coord adds kind + parent_ws_id; interactive adds user_id). - **Top-level response key converges on ``"workstreams"``** on both endpoints. Coord ``coordinators`` key removed — coord is a 1.5.0aN-only surface (never shipped stable) so the convergence has no compat shim; SDK / frontend consumers swap once. - **Storage + manager-lock work moved off the event loop on interactive**. ``list_workstreams_with_history`` runs through ``asyncio.to_thread`` on both kinds (matches coord's pre-existing perf-2 pattern from the saved-coordinators review); ``mgr.list_all`` + per-row work also offloaded. - **N+1 storage round-trips on /v1/api/workstreams eliminated**. Pre-lift interactive resolved the alias for every active row in a separate SELECT (up to 50 round-trips per dashboard refresh on a saturated node). Lifted body issues one bulk SELECT. Pydantic schemas: ``WorkstreamInfo.id`` renamed → ``ws_id``, ``WorkstreamInfo.user_id`` field added. ``CoordinatorInfo`` and ``CoordinatorListResponse`` removed (folded into the unified ``WorkstreamInfo`` / ``ListWorkstreamsResponse``). OpenAPI spec snapshots regenerated. TS SDK types updated (``WorkstreamInfo`` interface gains ws_id + the always-include fields); TS test mock + assertion updated to match. ``GET /v1/api/dashboard`` is intentionally NOT in this PR's scope and still returns rows keyed on ``id``. Tracked as a separate cleanup PR (tombstone-note added at the dashboard handler). /review pipeline run; the four Major findings + one Minor + six nits all addressed in the same commit: - M1: TS SDK ``WorkstreamInfo`` interface stale (id: string) → renamed + fields added. - M2: TS SDK test masked the type-mismatch with stale mock → updated. - M3: N+1 alias resolution on active list → bulk ``get_workstream_display_names`` helper + ``list_resolve_titles`` bulk cfg hook. - M4: Missing interactive parity regression test for unified row shape → mirror of coord's added in test_server_authz.py. - Mi1: ``audit_action_prefix`` string-compare deriving kind → explicit ``cfg.list_kind: WorkstreamKind`` field. - Six nits: redundant inner asyncio import, forward-ref quotes on Awaitable, duplicated frontend comments, dashboard ``id`` field has no tombstone-note, empty-coord_mgr short-circuit on ``saved_loaded_lookup``. 4512 tests passing; ruff + mypy clean.	2026-04-26 13:11:07 -07:00
Patrick Buckley	4a72b2ce19	build(sdk): regen openapi specs + bump TS SDK to 0.4.0 ... Stage 2 Priority 0 Step 0.5 follow-on. The handwritten Python OpenAPI spec already moved to ``/v1/api/workstreams/`` in the URL sweep commit; this just regenerates ``sdk/typescript/openapi-{server,console}.json`` from those specs so generated TS callers see the new paths. Bumps the TS SDK to 0.4.0 to flag the URL-shape break for any 1.5.0aN-era consumer of the experimental coord client. Python SDK needs no change — it never exposed the coord HTTP surface. TS typecheck + 32 vitest tests pass.	2026-04-24 16:56:06 -07:00
Patrick Buckley	58d20f4012	chore(coord): remove spawn-quota subsystem (#403) ... * chore(coord): remove spawn-quota subsystem The quota gate was operator-level safety per its own comments, not a security boundary, and never fired in a week of heavy use. Runaway coordinator spawns are already bounded by max_active slot exhaustion, which surfaces to the coord LLM as a tool error — same operational shape, one fewer moving part. Precedes the Stage 1 SessionManager unification so the coord tool doesn't inherit quota bookkeeping. Upgraded deployments with the three removed settings persisted will log three "Skipping invalid setting" warnings on startup and otherwise degrade cleanly; a follow-up migration to delete the rows would silence that noise. * chore(migrations): drop stale coord spawn-quota settings rows (047) Clears persisted rows for the three ConfigStore keys removed in the previous commit so upgraded deployments don't log "Skipping invalid setting" warnings on every startup. Downgrade is a no-op — the rows were operator-set values, and a rollback to pre-1.5.0 code falls back to the registry defaults for any key not present.	2026-04-23 20:09:12 -07:00
Patrick Buckley	7d61f9a37c	feat(coordinator): phase 8 PR A — spawn_batch + close_all_children batch tools (#386) ... * feat(coordinator): phase 8 PR A — spawn_batch + close_all_children batch tools Adds two model-facing batch tools so a coordinator can fan out without burning one approval per child: - `spawn_batch` — create up to 10 child workstreams in a single approval. Serialised spawns so sibling ordering (by created_at) stays deterministic. Returns `{results: {idx: {ws_id, name, node_id, status}}, denied: [{idx, reason}]}`. Per-item validation / spawn failures surface in `denied[]`; the batch hard-errors on >10 rather than silent truncation. - `close_all_children` — soft-close every direct child in one approval. Server-side Sem(16) fan-out via `coord_client.close_workstream`; `reason` propagates to every closed child's audit + workstream_config. Response mirrors `stop_cascade`'s cascade idiom: `{closed, failed, skipped}` where `skipped` is upstream-404 / already-gone. Shipped infra: - New console endpoint `POST /v1/api/coordinator/{ws_id}/close_all_children` (gated `admin.coordinator`, `allow_service_bypass=False`, 512-char reason cap, `coordinator.closed_all_children` audit). - Shared `_fanout_on_children` helper — both `stop_cascade` and `close_all_children` now delegate to it (one place to own the snapshot → semaphore-gather → bucket-split skeleton). - `CoordinatorClient.close_all_children(reason)` plus a `_post_url` seam that `_post` now reuses (no more duplicated transport-error handling). - `_emit_batch_event` — best-effort SSE emitter modelled on `_emit_wait_event`. Emits `batch_started` / `batch_ended` pairs keyed by call_id. Throttled `batch_progress` deferred to a follow-up. - OpenAPI request + response schemas, endpoint spec entry, TS SDK regenerated. - Persona doc (`tools_coordinator.md`) covers the two new patterns. Bulk-endpoint shape policy (codified in PR C later): split by semantic category — `{results, denied, truncated}` for bulk-read / bulk-create-with-payload (cluster/ws/live, spawn_batch), `{<bucket>, failed, skipped}` for cascade-mutation (stop_cascade, close_all_children). No retrofit needed on stop_cascade. Tests: new `test_coordinator_close_all_children.py` (8 endpoint tests), expanded `test_coordinator_tools.py` (session-side prepare/exec, coord_client=None guards, batch SSE events), expanded `test_coordinator_client.py` (route map, client method, transport errors), tool-count assertions updated. Deferred (not this PR): per-item selective-deny approval UI, throttled batch_progress SSE, coordinator-skills doc + bulk-endpoints doc (PR C), spawn budget / rate limit (PR B). * fix(coordinator): address PR #386 copilot review - coordinator_client.close_all_children: pass the unformatted path template as log_path so telemetry aggregates don't fragment per session (ws_id still lives in the real URL). - session.py: drop dead spawned_ids accumulator in _exec_spawn_batch — leftover from an eager-register path that got removed earlier. - close_all_children tool JSON: document the 512-char server-side cap on reason and that reason is echoed back in the response payload. Added maxLength:512 on the schema property so the LLM sees the constraint. - CoordinatorCloseAllChildrenRequest: add Field(max_length=512) so the OpenAPI schema reflects the runtime 400-on-overflow constraint.	2026-04-18 23:50:25 -07:00
Patrick Buckley	9826ea15c5	feat(coordinator): phase 7 — governance + skill metadata + cross-cutt… (#383) ... * feat(coordinator): phase 7 — governance + skill metadata + cross-cutting invariants Combines three stacked sub-PRs into a single coordinator phase-7 shipment against the phase-7 plan doc. The sub-PR structure (0 / A / B) preserved on individual branches for reviewer drill-down; this branch is the one reviewers should merge. ## Sub-PR 0 — service-auth boundary invariants Shared helpers and contracts that lock the console ↔ node service-auth boundary so later authz surfaces use them by construction. - ``_effective_user_filter(request)`` in both ``turnstone.console.server`` and ``turnstone.server`` with a shared ``DENY_EMPTY_SUB`` sentinel on ``turnstone.core.auth``. Three-way return — admin/service bypass, scoped caller uid, or fail-closed sentinel on blank sub. Four callsite migrations (``_coordinator_rows``, ``coordinator_children``, ``coordinator_metrics``, ``cluster_ws_live_bulk``). - ``StorageBackend`` class docstring codifies the tenancy contract (every list/count/aggregate method must accept ``user_id: str | None = None`` and push ``WHERE user_id = :user_id`` into SQL) and the ``_mapping`` row-access contract. New ``turnstone.testing.row_contract`` ships ``assert_row_like()``. - ``_verify_collector_service_scope`` probes an upstream node at boot with ``expected_node_id=_scope-probe_``; a 409 proves the scope gate was passed, a 403/401 sets ``collector_scope_error`` and causes ``cluster_snapshot`` / ``cluster_events_sse`` to return 503 with a remediation hint. Probe URL allowlist rejects non-http(s) schemes and 169.254.0.0/16 hosts. - 4xx log-level floor on ``_NodeDashboardCache.get``, ``_fetch_live_block``, and ``_proxy_sse`` — dotted-hierarchy prefixes with bounded body previews. ``_bounded_body_preview`` and ``_bounded_stream_preview`` strip control chars. ## Sub-PR A — coordinator governance core Mid-session governance surface for coordinator workstreams. - **Trusted-session mode.** New ``coordinator.trust.send`` permission (migration 042). ``ChatSession.set_trust_send`` / ``revoke_tools`` methods with a ``_governance_lock``. ``POST /v1/api/coordinator/{ws_id}/trust {send: bool}`` double-gated on ``admin.coordinator`` AND ``coordinator.trust.send`` with ``allow_service_bypass=False`` so service tokens can't escalate. ``_prepare_send_to_workstream`` auto-approves sends whose target is in the coordinator's own subtree; foreign ws_ids still require approval. ``_is_own_subtree`` checks both ``parent_ws_id`` AND ``user_id`` to defend against cross-tenant row corruption. - **Audit-layer credential redaction.** ``record_audit`` walks ``detail`` (dicts, lists, tuples, sets, frozensets; keys too) and routes every string through ``redact_credentials`` + a C0 control-char scrub. New kw-only ``raw_detail=True`` opt-out. ``_has_any_string`` fast-path. Audit action registry extended with the four new governance sub-prefixes. - **Mid-session revocation + cascading stop.** ``POST /v1/api/coordinator/{ws_id}/restrict {revoke: [...]}`` caps 256 entries / 128 chars; ``_prepare_tool`` short-circuits with a tool-error. ``POST /v1/api/coordinator/{ws_id}/stop_cascade`` cancels the coord's in-flight generation then dispatches ``cancel_workstream`` for every direct child in parallel via ``asyncio.gather`` bounded by ``Semaphore(16)``. Per-child outcomes split into ``cancelled`` / ``failed`` / ``skipped`` (404 = already-gone rather than dispatch-broken). Both endpoints apply ``allow_service_bypass=False`` on the admin gate. - **Shared plumbing.** ``_resolve_coord_session`` helper collapses the handler prelude three endpoints shared. ``_emit_coord_audit`` wraps ``record_audit`` in a dedicated ``ThreadPoolExecutor`` (``app.state.audit_executor``) so audit bursts don't starve cancel dispatches. ``_require_json_object`` guards body parsing so non- object JSON returns 400 instead of 500. ## Sub-PR B — skill metadata governance - **Description validator (migration 043).** ``prompt_templates`` rows now require a non-empty ``description``. Existing empty rows get backfilled with a ``"Skill: <name>"`` placeholder on upgrade. The installer (``admin_skill_discover``) and MCP prompt sync both synthesise a placeholder when the upstream description is blank so non-admin write paths satisfy the invariant. - **Skill kind classifier (migration 044).** New ``prompt_templates.kind`` column (``interactive`` / ``coordinator`` / ``any``; defaults to ``any``). New ``turnstone.core.skill_kind.SkillKind`` StrEnum is the single source of truth; Pydantic schemas type ``kind`` as ``SkillKind`` (OpenAPI advertises the enum) and the handler validator catches the ValueError. ``list_skills_filtered`` gains a ``kinds: list[str] | None = None`` SQL filter. ``CoordinatorClient.list_skills`` defaults to ``kinds=["coordinator", "any"]`` so interactive-only skills are hidden from the orchestrator. - **``scan_status`` → ``risk_level`` rename (migration 045).** Lossless column rename to align with ``IntentVerdict.risk_level`` terminology. Swept storage (both backends + schema + protocol), handlers, API schemas, tool JSON, generated OpenAPI specs, TypeScript SDK types, frontend (``governance.js``), tests, and English prose in ``docs/judge.md`` + ``docs/tools.md``. The user-facing on-load warning now reads ``has risk level: {risk_tier}``. Tool JSON's ``risk_level`` enum corrected to the scanner's actual taxonomy (``safe / low / medium / high / critical``; was the never-shipped ``clean / flagged / unscanned / pending``). Historical migration 021 left untouched. ## Migrations 042 (``coordinator.trust.send`` perm — PR A) 043 (description backfill — PR B) 044 (``kind`` column add — PR B) 045 (``scan_status`` → ``risk_level`` rename — PR B) All four use position-anchored permission strings / host-side parse-filter-rejoin on downgrade where SQL ``REPLACE`` could corrupt prefix-overlapping values. ## Verification - ``ruff check turnstone tests`` clean. - ``mypy turnstone`` clean on 165 source files. - ``pytest -m "not live"``: 4431 passed (+85 over the phase-6 baseline). Includes +32 tests in ``tests/test_service_auth_boundary.py`` and +38 in ``tests/test_coordinator_governance.py``; shared fixtures extracted to ``tests/_coord_test_helpers.py``. - Generated OpenAPI JSON (``sdk/typescript/openapi-{console,server}.json``) regenerated via ``sdk/typescript/scripts/generate-types.py``; zero ``scan_status`` occurrences remaining outside the historical migration 021 and the rename migration 045. ## Security reviews Both reviews flagged by the phase-7 plan (items 1 + 5, plus 0a's refuse-to-serve gate) ran through the multi-stage ``/review`` pipeline twice per sub-PR; all confirmed findings landed in-branch. * fixup(phase-7): CI lint + PR #383 review fixups Addresses the lint CI failure (ruff format) plus 12 findings from the two automated PR reviewers. Copilot: - ``_sqlite.list_installed_skill_urls`` / ``_postgresql.list_installed_skill_urls`` used positional row indexing (``r[0]``/``r[1]``/``r[2]``) while this same PR's ``StorageBackend`` class docstring forbids it. Switched both to ``r._mapping["..."]`` access. - ``list_skills.json`` previously advertised ``risk_level=""`` as a filter for unscanned skills, but the implementation treats empty strings as "no filter". Clarified the tool description to say omit the filter entirely to include unscanned rows, and added an explicit ``enum`` on the parameter restricting it to the scanner tiers. ``_prepare_list_skills`` keeps the ``strip() or None`` normalisation — unscanned filtering now has an unambiguous contract. - ``test_storage_skills_filtered.test_risk_level_filter`` used the legacy ``clean`` / ``flagged`` values from the pre-rename column. Rewritten with the scanner's actual taxonomy (``safe`` / ``high``). github-code-quality (CodeQL): - ``test_deny_sentinel_is_singleton`` previously asserted ``cs.DENY_EMPTY_SUB is cs.DENY_EMPTY_SUB`` — an identical-expression comparison. Rewritten as two separate ``from ... import ... as`` aliases (``FIRST_READ`` / ``SECOND_READ``) so the identity check is between distinct bindings. - ``test_restrict_empty_revoke_is_noop_but_audits`` unpacked ``state`` without using it. Renamed to ``_state``. - Mixed import styles in ``test_service_auth_boundary.py`` — the file previously used both ``import turnstone.console.server as cs`` and ``from turnstone.console.server import ...`` for the same module (same story for ``turnstone.core.auth`` and ``turnstone.server``). Consolidated to the ``from X import Y`` style used elsewhere in the file; the ``_fetch_live_block`` test now patches via pytest's ``monkeypatch`` fixture instead of a manual rebind through a module alias. CI: - ``ruff format`` reformatted one line in ``tests/test_coordinator_endpoints.py``. Verification: ruff check + mypy clean (166 files); 4459 non-live pytest pass. * fix(tests): swap asyncio marker for anyio in service-auth boundary tests PR #383 CI caught that the 13 ``@pytest.mark.asyncio`` decorators I added in ``test_service_auth_boundary.py`` are an off-convention choice — the rest of the repo uses ``@pytest.mark.anyio`` (148 sites vs my 13). The CI environment pulls in ``anyio`` but not ``pytest-asyncio``, so every async test in this one file was failing with "async def functions are not natively supported". It passed locally by accident — my dev venv happens to have pytest-asyncio installed ambiently. Swapped all 13 marker sites to ``@pytest.mark.anyio``. No functional change; the tests run under the same default asyncio backend anyio provides. Verification: ruff + mypy clean (166 files); 4459 non-live pytest pass.	2026-04-18 10:20:19 -07:00
Patrick Buckley	42e99d6990	docs: update tools, architecture, SDK for v0.9.2 changes ... - docs/tools.md: batch edit_file (edits array), bash stderr prefix, math sandbox extras, output truncation - docs/judge.md: JSON secret detection in output guard - docs/architecture.md: state_change now sent to per-workstream SSE - README.md: [sandbox] extras group in requirements - TypeScript SDK: StateChangeEvent type, type guard, exports - OpenAPI specs regenerated	2026-03-29 06:06:30 -07:00
Patrick Buckley	da5bf90a4b	feat: model detect button, capabilities API, and model dropdowns (#215) ... * feat: model detect button, capabilities API, and model dropdowns Admin Models tab: add Detect button that probes a model endpoint to verify reachability, list available models, detect context_window, and identify server type (llama.cpp/vLLM/SGLang/OpenAI/Anthropic). Add static capability lookup endpoint for auto-filling form fields when a known model name is entered. Add known-models endpoint for datalist autocomplete suggestions. Add "openai-compatible" as a third provider option for local servers, keeping the OpenAI SDK under the hood but suppressing capability auto-fill and known-model suggestions. Replace free-text model input with a select dropdown in both console and server new-workstream modals, populated from a new lightweight GET /v1/api/models endpoint. New endpoints: - POST /v1/api/admin/model-definitions/detect - GET /v1/api/admin/model-capabilities - GET /v1/api/admin/model-capabilities/known - GET /v1/api/models (both console and server) * fix: accumulate signature_delta for Anthropic thinking blocks (#214) The streaming path captured thinking_delta events but not signature_delta, leaving the signature empty on round-trip and causing 400 errors on multi-turn conversations with thinking enabled. * fix: address PR review — empty base_url, capability leak, response schemas - Don't pass empty base_url to OpenAI client (falls back to SDK default) - Return None from lookup_model_capabilities for openai-compatible provider - Only use static capability table for known models in _detect_openai_compat, avoiding misleading 200k default for unknown local models - Add AvailableModelInfo + ListAvailableModelsResponse schemas to both console_spec and server_spec - Regenerate TypeScript SDK OpenAPI snapshots * fix: apply same known-model guard to Anthropic context_window detection Only report context_window from the static capability table when the Anthropic model is actually known, matching the OpenAI path fix. * ui: add autocomplete hint to Model ID label in admin modal * fix: use explicit kwargs for OpenAI() to satisfy strict mypy	2026-03-29 03:50:21 -07:00
Patrick Buckley	4f6ef13ce9	fix: cancel button race condition with stream abort and force cancel (#202) ... The cancel endpoint emitted a 'cancelled' SSE event before the worker thread terminated. The frontend transitioned to "send" mode prematurely, so the next send got rejected with "Already processing a request." Backend: - Providers expose SDK stream handle via cancel_ref parameter so cancel() can close the HTTP connection and unblock iteration - Generation counter prevents orphaned threads from mutating messages or clearing cancel state after force cancel - _check_cancelled() added between retry attempts in _try_stream - Server polls (async, non-blocking) for cancelled worker to exit - Force cancel (force:true) abandons stuck worker, keeps cancel event set so subprocesses are killed, guards against spurious SSE events Frontend: - 'cancelled' shows "Cancelling..." then escalates to "Force Stop" after 2s for a harder cancel that abandons the worker immediately - 10s safety timeout auto-recovers if stream_end never arrives - busy_error re-enables stop button instead of showing send - Timeout cleanup in disconnectSSE, stream_end, and force .then() - Layout shift prevention (min-width, white-space: nowrap) - aria-label updates for accessibility Tests: - 7 new tests: stream close, error suppression, cancel_ref population, transport error conversion, non-cancel exception propagation, retry cancellation check	2026-03-28 19:26:37 -07:00
Patrick Buckley	86b404177b	chore: bump version to 0.8.4 ... - feat: split-pane layout for chat UI (#127) - fix: enforce CSS min dimensions during split handle drag - feat: add OpenShell sandbox policy for turnstone-server (#128) - fix: collector JWT expiry causes silent workstream data wipe (#126) - fix: auto-titler SSE event + SSE reconnection after restart (#125) - fix: wire resume_ws through console + expose max_ws in heartbeat (#124)	2026-03-18 18:13:18 -07:00
Patrick Buckley	ec3454ee2e	fix: wire resume_ws through console + expose max_ws in heartbeat (#124) ... * fix: wire resume_ws through console + expose max_ws in heartbeat Console create_workstream handler now reads resume_ws from the request body and passes it to CreateWorkstreamMessage on all three dispatch paths (pool, auto, explicit). Previously resume only worked via channel router and direct CLI — the console layer never plumbed it through. Server /health now includes max_ws from WorkstreamManager. Bridge reads it on startup and includes it in heartbeat metadata so the console's _pick_best_node gets accurate capacity instead of always defaulting to 10. Collector also updates max_ws on subsequent heartbeats (not just discovery). Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks fixed for new max_workstreams property access in /health. * fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id Add _fetch_server_metadata() so bridge reads max_ws from /health even when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats would advertise max_ws=10 regardless of actual server config. Add 3 test cases verifying resume_ws flows through all three console dispatch paths (directed, pool, auto-select).	2026-03-18 14:24:10 -07:00
Patrick Buckley	341d2f604f	fix: address PR #118 review feedback ... - Regenerate OpenAPI snapshots (openapi-console.json) to include license and compatibility fields in SkillInfo/CreateSkillRequest/UpdateSkillRequest - Omit version from create/update payloads when blank so server applies default "1.0.0" instead of storing empty string - Push enabled_only + limit filters into list_skills_by_activation storage query (protocol, SQLite, PostgreSQL) instead of loading all rows and filtering in Python; session.py now passes enabled_only=True, limit=30 - License length cap ([:128]) was already applied in previous commit	2026-03-17 16:09:06 -07:00
Patrick Buckley	75eda9a096	feat: unified skills system — merge prompt templates + workstream tem… (#106) ... * feat: unified skills system — merge prompt templates + workstream templates Evolves prompt_templates into a first-class skills entity and merges workstream templates into the same model, collapsing two concepts into one. Migration 021: 21 new columns on prompt_templates (skills metadata, security scan fields, session config from WS templates), skill_resources table for bundled files, skill_versions table for auto-snapshot version history. Data migration converts existing WS templates into skills with name collision handling, migrates version history, renames workstreams and scheduled_tasks columns, cleans orphaned permissions, drops old tables. Key changes: - All public interfaces renamed: templates → skills (API, CLI, SDK, UI) - Session config (model, temperature, token_budget, auto_approve, etc.) now lives on the skill and is applied at workstream creation - /skill slash command, set_skill() API, --skill CLI flag - BM25 skill search via SkillSearchManager for activation="search" skills - Admin UI: Skills tab with collapsible Session Config section, description subtitles, activation/origin/MCP badges, pagination - Shared validation helper (_parse_skill_session_config) for DRY CRUD - Version history with auto-snapshot on every edit + API endpoint - Cascade delete (resources + versions) on skill removal - Security: range validation, activation allowlist, fail-closed enabled check, duplicate name 409, readonly guard, JSON validation - 77 new tests across storage, runtime, search, API integration, and migration behavior verification (2521 total) * fix: address Copilot review + rename admin.templates → admin.skills - Skip skill lookup when resume_ws is set (avoids spurious 400) - Fix _applied_skill_version mismatch (1 in both workstreams table and session) - Remove stale template field from MQ protocol diagram - Rename admin.templates permission to admin.skills everywhere (runtime, frontend, tests, docs) with migration step for persisted role data - Fix stale /api/templates references in docs and diagrams - Update docstrings/comments for skills terminology * fix: address Copilot round 2 — skill version lineage + stale doc refs - Compute actual skill version from skill_versions count (not hardcoded 1) - Use same version in both workstreams table and session metadata - Fix response payload example: "templates" → "skills" key - Fix "Each template summary" → "Each skill summary"	2026-03-16 14:47:06 -07:00
Patrick Buckley	80e1924d7f	feat: enable prompt caching for Anthropic and OpenAI providers (#104) ... * feat: enable prompt caching for Anthropic and OpenAI providers Activate automatic prompt caching on both LLM providers to reduce input token costs on multi-turn conversations. Anthropic gets cache_control: ephemeral (90% savings on cache hits), OpenAI GPT-5.x gets 24h extended cache retention (free). Cache metrics flow end-to-end through the entire data pipeline: provider → session → server SSE → MQ protocol → storage → Prometheus metrics → admin Usage tab. - AnthropicProvider: top-level cache_control on all requests, extract cache_creation_input_tokens and cache_read_input_tokens from streaming and non-streaming responses - OpenAIProvider: prompt_cache_retention=24h for GPT-5.x, extract cached_tokens from usage.prompt_tokens_details - UsageInfo: new cache_creation_tokens and cache_read_tokens fields - Migration 020: add cache columns to usage_events table - Storage: record_usage_event and query_usage updated (sqlite + pg) - Metrics: turnstone_tokens_total{type="cache_creation|cache_read"} - Server: on_status passes cache tokens to SSE, storage, and metrics - MQ: StatusEvent carries cache fields through bridge - SDKs: Python and TypeScript StatusEvent types updated - OpenAPI: UsageBreakdownItem schema includes cache fields - Console UI: Usage tab shows cache write/read as secondary readouts with visual separator, dimmed when zero - 16 new tests, docs and 3 diagrams updated * fix: address Copilot review feedback - Fix MQ protocol diagram clipping by switching to vertical package layout (inbound on top, outbound below) with package aliases - Regenerate OpenAPI snapshots to include cache_creation_tokens and cache_read_tokens on UsageBreakdownItem - Replace fragile MagicMock(spec=[]) + del pattern with types.SimpleNamespace in cache metrics missing-attributes test	2026-03-16 01:59:55 -07:00
Patrick Buckley	20df7b3034	feat: OIDC SSO authentication with PKCE, auto-provisioning, and role … (#71) ... * feat: OIDC SSO authentication with PKCE, auto-provisioning, and role mapping Add OpenID Connect as a fourth authentication method, enabling single sign-on via any OIDC provider (Okta, Azure AD, Google, Keycloak). Opt-in via env vars (TURNSTONE_OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET). Security: - Authorization Code Flow with PKCE (S256) - State/nonce parameters with database-backed pending store (multi-node safe) - JWKS signature validation with async fetch + key rotation retry - Algorithm allowlist from JWKS key (not token header) prevents confusion - Identity matching exclusively by (issuer, sub) — prevents account takeover - password_enabled=false enforced server-side, not just UI - Rate limiting on both authorize and callback endpoints - OIDC users get "!oidc" password sentinel (bcrypt rejects naturally) - ID token validated for iss, aud, exp, nonce Features: - Auto-provisioning with username deduplication on first login - Claim-based role mapping with IdP demotion propagation (revokes stale roles) - "Continue with [Provider]" SSO button on login page - OIDC-only mode hides password form - Setup wizard required before OIDC login (admin bootstrap) Storage: migration 018 (oidc_identities + oidc_pending_states tables), 8 new protocol methods on both SQLite and PostgreSQL backends. 66 new tests (2273 total). * fix: address PR #71 review feedback (18 items) Bugs fixed: - OIDC success redirect now fetches permissions via new /auth/whoami endpoint before completing login (fixes permission-gating in UI) - Remove double decodeURIComponent on oidc_error (URLSearchParams already decodes; extra call throws on stray %) - Authorize rate limiter returns redirect instead of JSON 429 (endpoint reached via browser navigation, not fetch) - Lazy JWKS fetch in callback when startup discovery failed (IdP recovery without restart) - Startup exception handlers now log with exc_info=True - PostgreSQL pop_oidc_pending_state uses DELETE...RETURNING for true atomicity (eliminates TOCTOU) Behavior: - New OIDC users without role mapping get builtin-viewer by default (assigned_by="oidc-default", not revoked by role sync) Documentation fixes: - Role mapping: sync semantics (add + revoke stale), not "additive only" - PASSWORD_ENABLED=false blocks ALL password logins including admin - Algorithm: asymmetric allowlist, not per-key derivation - PlantUML diagram updated for role revocation API spec fixes: - Removed error_codes=[302] from callback (302 is success redirect) - Added /auth/whoami to both server + console specs - Regenerated TypeScript SDK OpenAPI snapshots (23 + 51 paths) * fix: address PR #71 round 2 review feedback (10 items) Rate limiting: - Authorize endpoint now calls record() after check() so the rate limiter actually counts attempts (was a no-op before) OIDC resilience: - Split startup try/except: discovery failure disables OIDC, JWKS prefetch failure leaves OIDC enabled for lazy retry on first login - JWKS unavailable message changed to "temporarily unavailable" (was misleadingly "not configured") - create_oidc_pending_state raises on collision instead of OR IGNORE (prevents silent insert drop on state collision) - SQLite pop_oidc_pending_state uses BEGIN IMMEDIATE for write lock (eliminates TOCTOU race) Frontend: - OIDC error display deferred 300ms so showLogin()'s async status fetch doesn't clear it via _switchMode → _clearError API spec: - OIDC authorize/callback endpoints now declare response_code=302 - Added AuthWhoamiResponse Pydantic model for /auth/whoami - Regenerated TypeScript SDK OpenAPI snapshots Documentation: - Diagram: JWKS "cached at startup, refreshed on-demand" (was "hourly") - Added TODO(tech-debt) comments on Host header redirect_uri sites	2026-03-15 03:44:18 -07:00
Patrick Buckley	376da3d084	feat: prompt template tech debt — tests, read-only endpoints, double-… (#67) ... * feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal Close test coverage gaps for prompt templates: - Resume with deleted template: verifies graceful degradation (template_content=None, warning logged) - Threading safety: concurrent set_template/init_system_messages with no race conditions - Factory passthrough: template kwarg propagation through WorkstreamManager.create() Add read-only template listing endpoints (read scope, no content exposed): - GET /v1/api/templates — prompt template summaries (name, category, is_default, origin) - GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model) - Available on both server and console; Python + TypeScript SDK methods added - Console creation modal switched from admin endpoint to read-scope endpoint Eliminate double-load inefficiency in workstream creation: - Template validation moved before mgr.create() (no create-then-rollback on invalid template) - template kwarg plumbed through WorkstreamManager.create() and session factory - _SessionFactory Protocol added for proper mypy typing Add workstream creation modal to server web UI: - Name, model, template dropdown, ws_template/profile dropdown - Instrument panel aesthetic: gradient top border, blur backdrop, amber accent - Focus trap, Escape/Enter keyboard handling, loading state, error display - WCAG AA contrast compliance, reduced-motion support * fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates() to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint. Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types. Regenerate openapi-server.json and openapi-console.json snapshots. Addresses Copilot review feedback on PR #67. * fix: skip template pre-validation when resuming a workstream When resume_ws is set, the request's template field is irrelevant — resume() restores the template from workstream_config. Pre-validating a stale template name would incorrectly return 400 before the resume even runs. Addresses Copilot review feedback on PR #67.	2026-03-15 02:09:31 -07:00
Patrick Buckley	19abc0cc65	feat: admin MCP Servers tab — database-backed MCP server management w… (#62) ... * feat: admin MCP Servers tab — database-backed MCP server management with live status Add MCP Servers admin tab (14th tab, System group) for managing MCP server definitions via the database instead of static JSON config files. Storage: `mcp_servers` table (migration 016), 6 CRUD methods on both SQLite and PostgreSQL backends, `MCP_SERVER_MUTABLE` field allowlist. Config priority chain: DB rows (if any enabled) → CLI `--mcp-config` → `mcp.config_path` setting → none. Nodes auto-load from DB on startup via `load_mcp_config(storage=)`. Hot-reload: `reconcile_sync(storage)` diffs running servers against DB — adds missing, removes stale, reconnects changed. `_db_managed` set tracks DB-sourced servers so config-file servers (MCP_CONFIG env) are never removed by reconcile. Per-server `AsyncExitStack` for clean teardown. Reload pattern: console writes to DB then signals nodes via `POST /_internal/mcp-reload` (update by reference, no config payload). Console admin API: 7 endpoints under `/v1/api/admin/mcp-servers` (CRUD + reload + import), `admin.mcp` permission, secret masking (env/headers replaced with *** unless ?reveal=true), audit log sanitization. Unified view: tab merges DB-managed servers with config-sourced servers detected on nodes. Config servers shown as read-only rows with "config" badge — no edit/delete. Admin UI: 7-column grid with magenta status dots, transport badges, single-column create/edit modal, paste-based JSON import (mcpServers format), detail modal with per-node status. Mobile 3-column collapse, reduced-motion support, backdrop-click dismiss, focus trapping. SDKs: 7 methods on Python (async+sync) and TypeScript SDKs. Also fixes: Settings tab permission gate (admin.users → admin.settings), _ALL_PERMISSIONS list in governance.js (5 missing permissions added), _internal/mcp-reload added to APPROVE_PATHS. Docs: architecture.md (14 tabs), api-reference.md (7 endpoints), 20-mcp-architecture.puml updated with admin-driven lifecycle. 66 new tests (2232 total). * fix: address Copilot review feedback on MCP admin PR - Docs: fix "merges both sources" → "first-match-wins priority" (architecture.md) - Validation: require command for stdio, url for streamable-http transport - Validation: check args/headers/env types in import handler before storing - Schema: add transport/command/url to McpServerStatus, source to McpServerDetail - Thread safety: move all remove_server_sync mutations onto MCP event loop thread - Regenerate OpenAPI JSON snapshots for TypeScript SDK	2026-03-14 17:02:50 -07:00
Patrick Buckley	67f43a7ee0	feat: [memory] REST API endpoints + SDK methods + docs (#56) ... * feat: [memory] REST API endpoints + SDK methods + docs Server API (4 endpoints): - GET /v1/api/memories — list with type/scope/scope_id/limit filters - POST /v1/api/memories — save (upsert) with validation - POST /v1/api/memories/search — search by query (read scope) - DELETE /v1/api/memories/{name} — delete by name+scope Console admin API (4 endpoints): - GET /v1/api/admin/memories — list all memories - GET /v1/api/admin/memories/search — search with ?q= param - GET /v1/api/admin/memories/{memory_id} — get by ID - DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit Storage: add delete_structured_memory_by_id, add mem_type filter to count_structured_memories. Auth: memory DELETE requires write scope, admin.memories permission added to valid set + builtin-admin role. Python SDK: list_memories, save_memory, search_memories, delete_memory on both server (async+sync) and console (async+sync) clients. TypeScript SDK: matching methods + types on both clients. Pydantic schemas with Literal type/scope validation, OpenAPI endpoint specs on both servers. 33 endpoint tests + 8 auth scope tests. Docs: docs/memory.md feature guide, api-reference.md endpoint docs, 23-memory-architecture.puml diagram. Also fixes stray `total: int` on CreateChannelUserRequest. * fix: [memory] address PR review — cross-user scope, schema types, snapshots Security: user-scoped memory endpoints now bind scope_id to the authenticated user's identity. Providing a mismatched scope_id returns 403, preventing cross-user memory access on all 4 server endpoints. Schema: MemoryInfo response uses MemoryType/MemoryScope Literals. SearchMemoriesRequest uses filter Literals (empty string allowed). Limit query params declare schema_type="integer" for correct OpenAPI. Regenerate sdk/typescript/openapi-{server,console}.json snapshots. Update count_structured_memories docstring for mem_type param. Fix fallback response to use normalized name after save. 6 new security tests for user-scope access control.	2026-03-14 02:28:47 -07:00
Patrick Buckley	02d9c5c797	feat: workstream templates — behavioral profiles for workstream creation (#49) ... * feat: workstream templates — behavioral profiles for workstream creation Workstream templates define the complete configuration for workstream creation: system prompt, model, auto-approve policy, per-tool auto-approve, temperature, reasoning effort, max tokens, agent max turns, token budget, and completion notifications. Applied once at creation time (snapshot, not live binding). Auto-versioning captures pre-update state on every edit. Schema & storage: - workstream_templates + workstream_template_versions tables (migration 011) - ws_template_id/ws_template_version columns on workstreams table - ws_template column on scheduled_tasks table - Full CRUD + versioning on SQLite and PostgreSQL backends - prompt_template_hash (SHA-256) for drift detection Runtime: - Template resolution before mgr.create() for model override - Post-creation settings application (prompt, temperature, approval, budget) - Token budget enforcement in session.send() — 80% warning, approval gate at 100% via __budget_override__ synthetic tool - WebUI.auto_approve_tools server-side per-tool auto-approve - Prompt template drift detection (hash comparison, log warning on mismatch) Integration: - ws_template field on CreateWorkstreamMessage, bridge, channel router, scheduler dispatch, MQ client - Console admin "WS Templates" tab (11th) with CRUD, version history modal - Profile dropdown on workstream creation modal - WS template dropdown on scheduler create/edit modals - Prompt template name validation on ws_template create/update - 7 console admin API endpoints + read-only summary endpoint - Full OpenAPI spec entries in console_spec.py - Python SDK (sync + async) and TypeScript SDK methods - Pydantic schemas for all request/response models Docs & diagrams: - New 21-ws-template-architecture.puml sequence diagram - Updated governance, storage, MQ protocol diagrams + PNGs - Updated architecture.md, governance.md, api-reference.md, console.md, sdk.md 48 new tests (1788 total). mypy clean. ruff clean. * fix: address PR #49 review feedback - auto_approve_tools uses approval_label (not just func_name) for consistency with tool policy evaluation - inline system_prompt from ws_template persisted as _ws_template_system_prompt in workstream_config, restored on resume (previously lost because _template_content wasn't persisted) - budget gate (__budget_override__) no longer bypassed by blanket auto_approve — requires explicit approval or tool policy allow - diagram 21 field list corrected (removed tool_search/threshold, added prompt_template_hash/notify_on_complete) * fix: address PR #49 review feedback (round 2) - Grant admin.ws_templates permission in migration 011 (tab was hidden) - Center WS template modals and fix radio button alignment - Skip template validation when ws_template overrides prompt - Guard against empty version snapshots on no-op updates - Replace setTimeout race with Promise chain in schedule ws_template select - Validate numeric fields in admin create/update handlers (400 not 500) - Add ws_template to TypeScript OpenAPI specs - Use typed Pydantic response models in SDK ws_template methods	2026-03-12 21:06:51 -07:00
Patrick Buckley	2f7f70825b	feat: wire prompt templates into session startup with full creation-p… (#47) ... * feat: wire prompt templates into session startup with full creation-path support Prompt templates (prompt_templates table) now have runtime effect: - is_default=true templates auto-apply as system message content, concatenated in name order before user instructions - Per-workstream template selection via --template CLI flag, template field on POST /v1/api/workstreams/new, console creation modal dropdown, scheduled task config, and channel adapter config - {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass regex (prevents cross-variable injection) - /template slash command for runtime switching, persisted across resume - set_template() public API on ChatSession Security hardening: - MCP sync resets is_default=False on content update (prevents compromised server from injecting defaults) - 32KB content cap on template create/update + defensive truncation - Template existence validation returns 400 before workstream creation - Single-pass regex eliminates cross-variable expansion Template field plumbed through all creation paths: CLI, server API, MQ protocol/bridge, console backend, scheduler dispatch, channel router, MQ client. Migration 010 adds template column to scheduled_tasks. Frontend: console workstream modal template dropdown, scheduler create/edit template field, governance template UI variables auto-detected from content (read-only display replaces editable input). Focus trap and Enter-key accessibility fixes in workstream modal. Docs: governance.md template runtime section, api-reference.md template field, governance + MCP architecture diagrams updated. Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests. * fix: address PR #47 review feedback - Defer template validation until after resume_ws — a bad template name no longer 400s when resume would have ignored it anyway - Add template field to OpenAPI JSON specs (openapi-server.json, openapi-console.json) for SDK/docs consistency - Validate template existence in schedule create and update endpoints — reject unknown template names with 400 instead of allowing schedules that would silently fail at dispatch time	2026-03-12 16:57:31 -07:00
Patrick Buckley	be165c1971	feat: MCP resource and prompt discovery with read_resource tool (#44) ... * feat: MCP resource and prompt discovery with read_resource tool Extends MCPClientManager with resource and prompt discovery alongside existing tool support. Resources and prompts are discovered on connect, cached per-server with copy-on-write rebuilds, and refreshed via push notifications, periodic polling, or manual /mcp refresh. New read_resource built-in tool reads MCP resources by URI. Requires user approval (same as MCP tool calls) since resources are served by external MCP servers. Resource catalog injected into system message with XML delimiters. Error messages sanitized to prevent leaking server internals to the model. Prompt discovery stores prefixed names (mcp__server__prompt) and exposes get_prompt_sync() for future use_prompt tool (Chunk D). /mcp command now shows tools, resources, and prompts. Docs and diagrams updated. * feat: MCP prompt governance sync with origin tracking and readonly guards Migration 009 adds origin, mcp_server, and readonly columns to prompt_templates. MCP prompts discovered by MCPClientManager are automatically synced into the governance table as read-only templates with origin="mcp". Sync engine handles: create on connect, update on prompt refresh, delete when prompts are removed from server. Manual templates take precedence on name collision (MCP prompt skipped with warning). Admin API returns 403 on update/delete of readonly templates. Console UI shows MCP origin badge and disables edit/delete buttons. Storage backends gain get_prompt_template_by_name, list_prompt_templates_by_origin, and delete_prompt_templates_by_server methods. Also addresses PR #44 review feedback: concurrent.futures.TimeoutError handling in sync dispatch, XML-escape resource catalog descriptions, resource template entries excluded from _resource_map, URI collision warnings, needs_periodic capability-aware computation, malformed JSON primary key fallback for read_resource. * feat: use_prompt tool, prompt catalog, and PR review hardening New use_prompt built-in tool invokes MCP prompt templates by name, expanding them into messages. Requires user approval (external MCP servers). Prompt catalog injected into system message with XML delimiters (up to 30 prompts, HTML-escaped). Prompt listener registered in session for catalog rebuild on changes. Addresses PR #44 review feedback: - _init_system_messages() now uses copy-on-write (build locally, assign atomically) so background thread callbacks never see partial system messages - sync_prompts_to_storage() serialized behind _sync_lock to prevent races between set_storage() (main thread) and MCP background thread - shutdown() clears listener lists to release callback references Docs and diagrams updated for 18 built-in tools. * feat: granular tool policies for MCP resources, prompts, and tools Policy evaluation now uses approval_label (falling back to func_name) for fnmatch pattern matching, enabling fine-grained per-URI and per-server policies: - read_resource: mcp_resource__{normalized_uri} - use_prompt: mcp__{server}__{prompt} (prefixed name) - MCP tools: mcp__{server}__{tool} (was static "mcp_tool") URI normalization resolves .. path segments to prevent traversal bypasses in policy matching. Resource templates filtered from system message catalog (not directly readable). use_prompt arguments validated as dict with string coercion. TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly fields. Governance docs updated with MCP policy patterns. * feat: MCP visibility in server and console UIs Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts counts. Server UI status bar shows magenta MCP indicator with tooltip. Console cluster status bar shows MCP metrics with magenta LED dot. Console node detail view shows per-node MCP summary. Console collector aggregates MCP counts across nodes in overview. Uses var(--magenta) design token with new --magenta-glow for theme adaptation. ARIA roles on MCP status elements. Tooltips on console MCP metric labels. Node MCP summary hidden on mobile (< 700px). New diagram: 20-mcp-architecture.puml covering full MCP lifecycle (connection, discovery, refresh, governance sync, policy, UI). * fix: McpStatus in health schema, count properties, catalog name fidelity Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so typed clients see the mcp field from /health. Addresses Copilot review feedback: - resource_count/prompt_count properties avoid list allocation on /health and /metrics polls - get_tools/resources/prompts return shallow-copied dicts to prevent callers from mutating internal cache - Prompt names and arg names in system message catalog are NOT HTML-escaped (model must use exact strings in use_prompt calls); only descriptions are escaped * fix: OpenAPI spec McpStatus + diagram approval column accuracy Adds McpStatus schema and optional mcp field to HealthResponse in openapi-server.json, matching the Python schema and TypeScript types. Fixes tool pipeline diagram: math, web_fetch, web_search correctly shown as auto-approve (not "Yes" for approval).	2026-03-12 14:49:58 -07:00
Patrick Buckley	fd507c6a3c	feat: generation cancellation — stop button, cancel API, cooperative … (#40) ... * feat: generation cancellation — stop button, cancel API, cooperative cancel Add cooperative cancellation via threading.Event on ChatSession. The cancel signal is set from outside the worker thread (HTTP handler, MQ bridge, or Escape key) and checked at defined checkpoints: per streaming chunk, before tool execution, inside bash commands, and at each sub-agent turn. Core: GenerationCancelled(BaseException) exception, cancel()/_check_cancelled() methods, partial content preservation in _stream_response, clean rollback in send() with idle state emission (no re-raise). Server: POST /v1/api/cancel endpoint, CancelledEvent SSE emission, worker thread safety net. Frontend: Stop button (■ Stop) with send/stop swap via setBusy(), Escape key shortcut, cancelled event handler. Accessible: aria-label, focus-visible override, light theme contrast, non-color differentiation. MQ: CancelMessage inbound type, bridge _handle_cancel routed handler. SDK: cancel() on Python async+sync clients, CancelledEvent in Python+TypeScript event registries, isCancelledEvent type guard. OpenAPI: CancelRequest schema + endpoint spec. Docs: API reference, architecture, SDK docs updated. Diagrams: conversation turn, tool pipeline, MQ protocol, workstream states, SDK architecture. * fix: address PR #40 review feedback - setBusy() now resets stopBtn.disabled so stop button is re-enabled on next generation after a successful cancel - Gate cancel side effects (resolve_approval, resolve_plan, cancelled SSE event) on worker_thread.is_alive() to avoid spurious events when idle - Add /v1/api/cancel endpoint and CancelRequest schema to TypeScript openapi-server.json to keep it in sync with Python-generated spec	2026-03-10 20:43:52 -07:00
Patrick Buckley	fb190f8977	Normalize session_id into ws_id as sole persistent identity (#29) ... * Normalize session_id into ws_id as sole persistent identity Eliminate the separate session_id concept. The workstream ID (ws_id) is now the single identity used for both real-time routing and conversation persistence, removing a layer of indirection that was 1:1 in practice and buggy on resume (stale pointers, orphaned rows). Schema changes (migration 006): - Drop sessions table; add alias/title columns to workstreams - Rename conversations.session_id → ws_id - Rename session_config table → workstream_config (ws_id column) - Data migration remaps existing conversations to ws_id Storage/API renames: - register_session → register_workstream (already existed, merged) - save_message/load_messages now keyed by ws_id - resolve_session → resolve_workstream - ChatSession.session_id property → ws_id - ChatSession.resume_session() → resume() - resume_session field → resume_ws - SessionResumedEvent → WorkstreamResumedEvent - /api/sessions → /api/workstreams/saved - /sessions slash command → /workstreams - --session-retention-days → --retention-days Channel eviction recovery simplified: reuses old ws_id directly instead of get_session_id_by_ws() reverse lookup. * Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec - docs/channels.md: "resumes the session" → "resumes the workstream", "Session resumed:" → "Resumed:", "old session was pruned" → "old workstream was pruned" - docs/api-reference.md: "Each session object" → "Each saved workstream object", field descriptions updated, removed stale node_id field - sdk/typescript/openapi-server.json: fully regenerated from Python models — removes all stale session_id properties from WorkstreamInfo, DashboardWorkstream, CreateWorkstreamResponse schemas	2026-03-07 12:49:07 -08:00
Patrick Buckley	5ee539c983	Add Python and TypeScript client SDKs for server and console APIs (#19) ... * Add Python and TypeScript client SDKs for server and console APIs Python SDK (turnstone/sdk/) with sync + async clients for both server and console APIs. Returns Pydantic models directly, streams SSE events as typed dataclasses. 27 event types with registry-based deserialization. High-level send_and_wait() for request-response patterns. TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses fetch + ReadableStream for SSE parsing. Discriminated union event types with type guards. Same API surface as Python SDK. 63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at docs/sdk.md with SDK architecture diagram. * Address PR #19 review feedback + fix lint - Fix consume_task leak in send_and_wait when send() raises (try/finally) - Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout - Add signal param to TS streamSSE for cancellation support - Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF - Fix generate-types.py sys.path (parents[3] not parents[2]) - Document token ignored when httpx_client provided - Document TS timeout units as milliseconds - Fix stale docstring in test_sdk_sse.py - Fix import sorting (ruff I001)	2026-03-03 21:22:24 -08:00