mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
cc84f9d176a9a67cccbbce8fd862421fdb9a773a
50 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
cc84f9d176 | fix(memory): harden project scope authorization and consistency | ||
|
Patrick Buckley
|
480a1426b3 |
Fail-closed history-commit handoff (#1005)
* fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964 |
||
|
Patrick Buckley
|
98e96ab5f3 |
Add per-alias model concurrency admission (#990)
* feat(models): add per-alias concurrency admission Add registry-backed FIFO admission limits with queue-aware deadlines and full-stream leases. Expose max_concurrency through storage, admin configuration, OpenAPI, documentation, and diagrams, with role and live backend count coverage. * fix(api): omit null concurrency schema default Keep max_concurrency optional for presence-keyed updates without advertising a null default for its non-null integer OpenAPI shape. |
||
|
Patrick Buckley
|
7a06f5e8bc |
refactor(session): make ModelLane the provider boundary (#979) (#989)
* refactor(session): make ModelLane the provider boundary (#979) ## Summary This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot. - Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding. - Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call. - Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references. - Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results. - Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts. ## Validation - SQLite suite: 11,188 passed, 9 skipped, 10 deselected - PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected - Live backend: 3 passed - SSE recovery: 6 passed; browser recovery harness passed all scenarios - Ruff: clean; 595 files correctly formatted - mypy: 243 source files clean - TypeScript: typecheck/build and 35 tests passed - OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte - `git diff --check` and Git LFS integrity clean Closes #979. * fix(deps): update nanoid for GHSA-2v37-7h3g-55p8 Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation. Validation: - npm ci - npm audit --audit-level=moderate: 0 vulnerabilities - TypeScript typecheck and build - TypeScript tests: 35 passed * fix(test): assert canonical model registry URLs Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation. Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy. |
||
|
Patrick Buckley
|
8605c9783d |
feat(models): rfc8693_obo auth mode, per-alias exchange scopes, identity-keyed mint cache
Adds the dedicated `rfc8693_obo` model auth mode (#955): model definitions gain an `obo_scopes` column (migration 069), the mint threads the scopes to the token-exchange leg (RFC 8693), and every dynamic mode pins its grant leg — a mode is a dialect commitment, not a hint the deployment profile resolves. Exchange-capable IdPs refuse an audience whose scope was not requested; this closes the structurally unmintable model-OBO path on token-exchange deployments. The model mint-cache is identity-keyed on the owning definition's alias (`__model_obo__:<alias>` per user, `__model_app__:<alias>` under the shared app principal), matching the MCP discipline where rows key on the unique server name. The bearer's shape lives in the row's audience/scopes columns and the freshness gate compares it on every read, so a re-aimed alias refuses its old row and overwrites the same key in place. Admin lifecycle (rename, re-aim, scope change, delete) purges a definition's own rows through one shared helper — sound because one definition owns each key; a sibling's rows are untouchable by construction. Cooldown and backoff additionally key on the dispatch shape, so an operator's config repair is an instant clean slate. Cause records, cooldowns, locks and memoization are per-alias end to end, and the session heartbeat reads refusal causes under the same keys. Console: default-deny write gating for dynamic rows (value-diff over the full column ladder, admin.mcp escalation, a never-blockable pure-disable carve-out), a two-tier validator (audience allow-list on every write; deployment-posture checks when the pair is chosen), one shared scopes parser whose omit-unchanged arm keeps over-cap DB-direct residue rows disarmable without ungating real changes, and served constraints (dynamic/scopes/app-identity mode lists, mode-to-profile pairing) so the shelf tracks the registry by data. The admin shelf gains the mode option, a scopes input with residue affordances, pairing-aware option greying, and a derived auth badge. Registry load refuses control characters in alias, audience, and scopes — including the C0 separator block that str.split() would silently collapse — and the C0/DEL class has one exported spelling shared by every surface. Profile-mismatch visibility warns at reload and boot with the mode-correct cause, gated on OIDC being enabled. Breaking: a stored `entra_obo` alias on a deployment whose `[oidc] obo_grant_profile` is `rfc8693` (or the inverse pairing) no longer mints via the profile-driven overload — the mint refuses before any IdP traffic with cause `grant_profile_mismatch`, and the `model.auth_fail_closed` policy governs static fallback. Such rows never minted usefully on scope-gating IdPs; the shelf now surfaces the pairing and the per-turn heartbeat names the refusal cause. Live-verified end to end: scoped token exchange mints, the warm cache serves with zero IdP calls, and the mode/profile mismatch refuses with zero IdP traffic (scripts/obo-e2e/keycloak_e2e.sh); the refresh-redemption profile's E1-E7 hold via scripts/obo-e2e/entra_e2e.py. Closes #955. |
||
|
Patrick Buckley
|
33ace975d2 |
feat(models): default-deny governance and admin UI for per-alias backend auth
Follow-up to the per-alias Entra OBO/app-identity backend auth: the console write path now applies default-deny field classification, the admin shelf gains full backend-auth support, and the session/registry rebind machinery is hardened for config changes landing under live sessions. Console write gate: - Default-deny classification: any non-neutral change to a row that is or becomes dynamic requires admin.mcp plus validation; the provably auth-neutral columns are enumerated (MODEL_AUTH_NEUTRAL_FIELDS) and a live-schema classification test forces every future column to be classified. The derivation is a pure function (_derive_auth_gate) with unit-pinned exclusivity invariants. - Two-tier validation mirroring the MCP oauth_obo validator: the row tier (audience allow-list) runs on every gated write; the posture tier (OIDC configured, token store present) runs on pair changes and on enable-arming. - Pure-disable carve-out: disabling a dynamic row is de-escalation and is never blocked — admin.models suffices and validation is skipped, including for rows with corrupt or skewed stored values. - Capabilities are compared canonically (key order, integral floats), the audience compare normalizes both sides, and staging an audience on a static row is refused on both write twins. - Calibrate writes the capabilities column under an enforced confinement invariant with a compare-and-swap persist. Admin shelf: - Backend-auth section with a per-open constraints fetch (GET /model-definitions/auth-constraints: audience allow-list, grant profile, dynamic modes), datalist audience suggestions, server-defined modes preserved on round-trip, and permission-aware visibility built on cache-skew-safe helpers shared through auth.js. - Refused live-registry swaps surface as an amber registry_warning on the write, delete, reload, and calibrate responses; audit rows carry auth_gated / auth_disarmed markers visible in the audit view. Registry and sessions: - The encryption-key requirement for dynamic auth is enforced inside ModelRegistry.reload() itself — nodes refuse with 503 and the console records coord_registry_error — and reload bumps the generation before the map swap so a racing reader can never pair a stale generation with new maps. - resolve()/resolve_binding() return the generation from inside the registry lock; sessions rebind per send on generation change with atomic client/provider/config commits, fallback-first handling of removed or unconstructable aliases, and judge/limiter resets only when the binding actually changed. - Mint refusals record per-user causes surfaced in the per-turn heartbeat logs; misconfiguration warnings are deduplicated with bounded state. Verification: 10417 tests (99 added on this branch), a 71-scenario browser harness over the real admin shelf, and a live rfc8693 token-exchange e2e run (MCP legs verified end to end; the model-leg scope gap is tracked as #955 under a narrow known-gap signature). Closes #950. |
||
|
metaclassing
|
9adde920d4 |
feat(models): per-alias backend auth via Entra OBO and app identity (#898)
Adds a per-alias `auth_mode` on model definitions so a model backend can authenticate to an Entra-fronted gateway with a per-request minted token instead of one shared static API key, letting the gateway attribute calls to the actual user or to the app as a machine identity. - `static` (default, unchanged) sends the stored `api_key`. - `entra_obo` mints a per-user On-Behalf-Of token for `obo_audience` from the caller's captured refresh credential. - `entra_app` mints an app-identity token via the client-credentials grant, and covers userless turns that OBO cannot. Reuses the existing OBO grant legs, refresh-token rotation CAS, cluster advisory lock and the `mcp_user_tokens` mint-cache, keyed under synthetic `__model_obo__:<audience>` / `__model_app__:<audience>` rows. The token binds at the call site through `client.with_options(api_key=...)` so each SDK emits it on its own auth path rather than through header injection. Migration 068 adds `auth_mode` and `obo_audience`. Both are additive and existing rows default to `static`, so behaviour is unchanged unless an alias opts in. Operator controls: `model.auth_audience_allowlist` is an exact-match allow-list that gates which audiences may be configured and denies all by default, and changing a mode or audience requires `admin.mcp`. `model.auth_fail_closed` decides whether a failed mint may fall back to an explicitly configured static key. A delegated call with no user, or a dynamic alias with no real static key, always refuses. Two changes here apply regardless of whether any alias opts in: - Storage and app state are now wired into the console MCP client manager. This fixes per-user `oauth_user` / `oauth_obo` dispatch for coordinator-hosted sessions, which previously raised `RuntimeError` on first call because `set_app_state` was only ever called on the node. - Unattended watch restores and `--resume` resolve the persisted workstream owner instead of constructing the session under an empty principal. A workstream with no owner is now a permanent refusal rather than an anonymous, auto-approved run. |
||
|
Patrick Buckley
|
176f6a9631 |
chore(sdk): regenerate the console spec for the tasks schema
Picks up the needs_user status description and the new note property from console_schemas.py. Spec-only, no API behavior change. The server spec is knowingly left stale here: the generator writes both files unconditionally, and its drift belongs to the change that introduced it rather than to this one. |
||
|
Patrick Buckley
|
9dea2c89f1 |
chore(sdk): regenerate openapi-console.json
The console OpenAPI spec had drifted from build_console_spec(): the committed file was last generated at 1.7.0rc1 and was missing the persona and project_id workstream-creation fields (Personas and Projects, both 1.7) plus the version bump to 1.8.0a2. Regenerate via sdk/typescript/scripts/generate-types.py to resync. Spec-only; no console API behavior change (openapi-server.json was already current). |
||
|
Patrick Buckley
|
e60c19befd |
fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds
Wake path: - Denial metacog nudge moves to the tool channel so it drains with the denied tool batch instead of the next user-message seam. - wake_workstream_if_pending: shared wake gate for watch fires on already-idle workstreams (no IDLE transition for the watcher to observe), wired as wake_fn at every set_watch_runner site via the shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT — after eviction+restore an id-keyed manager lookup would miss). - session_worker exit backstop re-runs the wake gate the moment worker ownership clears: IDLE fans out on the worker thread, so transition-time wakes always landed on the reuse path and no-op'd (the coordinator idle_children strand). - deliver_wake_nudge_from_queue contains GenerationCancelled — it is the wake worker's run() closure and only Exception is caught downstream. Watch delivery: - Terminal fires that cannot reach their workstream are HELD and redelivered on min(interval, 60s) without re-running the command, bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own max_polls across cycles; the poll charge commits durably at hold time so restarts stay budget-bounded. - Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES cap, presence-only re-check under the lock, detection-only stall alerts (reclaiming a wedged admission would trade capped degradation for total poll-pool collapse). - Permanent-vs-transient restore taxonomy: corrupt persona stamp and genuinely-missing history (confirmed by a raising storage probe — the resume loader swallows read blips into []) deactivate the watch immediately; everything else holds and retries. - Cancel-race defense: delivery paths re-check is_watch_active before stashing/dispatching, cancel paths write the row BEFORE forget_terminal_dispatched, the HTTP cancel endpoint clears runner state, and a per-tick sweep bounds the residual stash-after-clear interleaving to one check_interval. - Abandon/exhaustion commits are write-then-clear so storage that can read but not write retries the row write instead of re-running the command every cycle; the fresh-fire unrestorable path stashes before its deactivation write for the same reason. Registry follows identity: - The dispatch registry is keyed by _ws_id at registration time; every rebind now moves it: non-fork resume() and /new go through _follow_watch_registration (new key live before the old is removed, never stealing a registration another live session holds), removals are owner-checked so tearing down a watch-restore shell or a resumed-away session cannot unregister a live pane, the restore shell yields to a registration that appears mid-restore, CLI --resume registers after the successful resume, and both the open path and the detail-GET lazy rehydrate wire the registration. Teardown gating and backpressure honesty: - cleanup_session_ui marks ws._closed FIRST under ws._lock — every teardown path (close, close_idle, evict, delete, discard) funnels through it — and session_worker.send re-checks under the same lock, so a wake can never spawn a worker on a torn-down workstream. - Create responses carry initial_message_status when the initial message could not be delivered (queue_full / refused_closed) instead of reading as success; staged attachments survive for the retry; /send surfaces a closed workstream as 404 rather than queue_full. Docs/spec: OpenAPI artifacts regenerated; api-reference documents the new create-response field; TS SDK type extended. Tests: ~30 new pins (cancel races, budget durability across restarts, owner-checked registry moves, teardown gating, stall alerts, backpressure surfaces, wait_until final re-check); wide subsystem sweep green (2353 passed). |
||
|
Patrick Buckley
|
8da5dc3f5a |
docs(api): regenerate OpenAPI specs for the details-list contract
The checked-in openapi-server.json / openapi-console.json still described the removed singular pending_approval_detail field; re-run generate-types.py so the reference specs carry the pending_approval_details list + cycle_id. Retire two docstring references to the deleted singular serializer and one stale comment. |
||
|
Patrick Buckley
|
b65e5cae0e |
docs(personas): accuracy sweep — spec models, protocol contracts, page corrections
Spec models now describe what the endpoints do: ListPersonasResponse declares the tool_inventory the shelf depends on, both console create models declare persona, CreatePersonaRequest declares org_id, and UpdatePersonaRequest documents the null-vs-absent split (null clears base_prompt/tool_allowlist, null on flags/kinds is ignored). Console OpenAPI regenerated. Protocol contracts match the implementations: update_persona's return covers the no-op case, create_persona's raises-list is complete, and both extended row-shape docstrings gain their tail columns plus the append-only rule. The workstreams.persona comments say slug, not display name. Page corrections from the docs review: personas.md documents the creative_mode-to-writer migration conversion, the mid-session /resume MCP-lever behavior, visibility-based nudge gating, the soft-set prompt-cache cost, and the executive tool list — and drops internal jargon. The changelog entry moves under [Unreleased] with the house breaking-marker style and the auto-conversion note. coordinator-skills and the API tour stop using persona to mean framing; governance, api-reference, sdk, console, tools, and memory pick up the new permission family, endpoints, kwargs, picker, and lever caveats. |
||
|
Patrick Buckley
|
54ebb24374 |
feat(personas): core stamping, four-lever application, create/spawn/SDK threading; remove /creative; add --persona
The persona resolved at creation is snapshotted into workstream_config (five keys, all-or-none) and applied ONLY from the stamp — the personas table is never read post-create, so edits/archives never touch existing workstreams, and a corrupt stamp fails construction loudly instead of silently reverting to a default envelope. Legacy pre-063 workstreams carry no keys and keep today's behavior byte-for-byte. The four levers (turnstone/core/personas.py holds the codec): 1. Base override — compose_system_message(base_override=...) replaces exactly the BASE module; ENV/CONTEXT/TOOLS/POLICIES keep composing so mandatory prompt policies ride on top of every persona. This also closes the old /creative hole where the fork bypassed composition (no CONTEXT, no DB policies). 2. Tool visibility — the allowlist intersects both the composition name set (TOOLS block self-suppresses, tool-gated policies drop, the memory advisory drops) and the END of _get_active_tools so the wire never advertises hidden tools. tool_search in the set = soft (discovered tools union with the allowlist via the session's expanded-names set); absent = hard (the whole pathway is disabled, covering provider-native defer_loading, which has no synthetic name to filter). Persona sets force client-side tool search. 3. MCP gate (session-wide) — an MCP-off persona drops the client reference at construction: no merge into _tools OR _task_tools, no listeners, refresh callbacks inert, resource/prompt catalogs gone. 4. Memory (own hands only) — no recall injection, memory-directed nudges suppressed (MEMORY_NUDGE_TYPES; behavioural nudges keep firing), memory tool hidden. _task_tools is NOT filtered; compaction spill/markers are never persona-gated, and the post-compaction recall pointer is emitted only when the recall tool is actually visible. Threading: the create handler resolves once (explicit name -> 400 on unknown/disabled/kind-mismatch; empty -> the kind's default; pre-seed DB -> unstamped legacy) and stamps via constructor kwargs + config keys + the workstreams.persona column; SessionManager.open threads the stamp pre-construction exactly like the saved model alias. Non-fork resume adopts the target's stamp so _save_config can't clobber it. spawn / spawn_batch gain a persona arg with prep-time validation (children are interactive-kind; omitted = kind default, never the parent's). Python + TS SDKs, OpenAPI specs, the picker feed GET /v1/api/personas (authed, no perm), and console admin CRUD /api/admin/personas (persona.* perms, archive-only — no DELETE) round out the surface. BREAKING: /creative is removed (the REPL command now points at the writer persona); turnstone --persona <name> is the replacement. Also fixes the CLI session factory, which TypeErrored on the project_id kwarg the shared InteractiveAdapter passes unconditionally. |
||
|
Patrick Buckley
|
2169559d6e |
feat(projects): governed project containers — memory scope, grouping, manage UI (#724)
* feat(projects): governed project containers — memory scope, grouping, manage UI
A workstream can attach to a project: a first-class, shareable resource
container that owns a `project` memory scope, groups conversations, and is
managed from the console.
Storage / migration 062: projects + project_members tables, workstreams.
project_id, and the memory type default project→general; grants
project.{create,read,write,delete} (admin-default).
Recall + writes: project memory is recalled iff the workstream is attached AND
the user has access (owner ∨ member ∨ public-for-read), resolved once at session
construction; coordinators recall it too. New saves default to the project when
attached + writable; the save and delete paths are write-gated; deleting a
project purges its scoped memory; archived projects aren't recalled.
Access = RBAC capability ∧ per-project ACL (auth.resolve_project_access, a
single-fetch resolver); visibility changes, member management, and delete are
owner-only.
API: project CRUD routes on both the server and console; project_id threaded
through workstream creation, spawn inheritance, the cluster-create proxy, the
dashboard / snapshot / coordinator row builders, and the collector deltas.
UI: a project picker with an inline "+ New project" creator in every creation
box (console launcher + standalone dialog + dashboard); group-by-project in the
rail; a project badge in the composer and on dashboard rows; a console manage
tab (list + create/edit + members shelves). The admin Memories view gains
coordinator/project scope filters and human scope labels (name, not hex). The
memory tool schema documents the project scope and the attach-aware default.
* fix(projects): client refresh hardening, creator race guard, SDK project_id
Addresses PR #724 review feedback plus two bugs found while validating it.
- projects.js refreshProjects: a non-OK status (e.g. 403 when the caller
lacks project.read) or a network/parse error no longer blanks the cache
or masquerades as "no projects" -- the prior cache is preserved, the
failure is recorded (new projectsError()) and warned. Honors the
long-standing "a transient error can't blank the rail" docstring.
- projects.js _fp: the fingerprint separators were raw control bytes,
which made git treat the whole file as binary (no reviewable diff).
Rewritten as escape sequences instead of raw bytes -- behavior is
byte-identical at runtime.
- project_creator.js: createProject() could reject unhandled (authFetch
throws on network/401; r.json() throws on a non-JSON body), leaving the
widget stuck busy/disabled. Added a .catch, plus a generation guard so a
create whose widget was cancelled/reopened mid-flight drops its result
instead of selecting a project the user backed out of.
- types.ts: add project_id to CreateWorkstreamRequest / WorkstreamInfo /
DashboardWorkstream to match the server schemas (was SDK-invisible).
- test_project_api.py: move side-effecting HTTP calls out of asserts so
the requests run even under python -O.
* fix(projects): JSON.stringify the cache fingerprint, drop control-byte separators
_fp joined fields/rows on raw NUL/SOH bytes, which made projects.js read as binary to git. Replace with a collision-proof, escape-free JSON.stringify encoding -- same change-detection semantics, zero embedded control characters.
|
||
|
Patrick Buckley
|
3e5633f440 |
chore(sdk): regenerate OpenAPI snapshots from current specs
openapi-server.json / openapi-console.json had drifted well behind build_server_spec() / build_console_spec() — the committed snapshots are regenerated periodically (via sdk/typescript/scripts/generate-types.py) rather than on every schema-changing PR, so accumulated additions (skill parsing, pending-approval items, model-definition CRUD, etc.) had not been captured. This resyncs both with no code changes. |
||
|
Patrick Buckley
|
dea2729292 |
refactor(coordinator): rename task_list → tasks, doc/prompt sweep (#437)
Four themes from a coordinator-feature shakedown:
1. Correctness fixes (return shapes / examples / behavior)
- tools_coordinator.md: drop fake skill names from spawn examples;
fix wrong kwarg ``node_id=`` → ``target_node=``.
- wait_for_workstream.json: document ``message`` + ``truncated``
per-ws fields (always enriched in the client; the JSON shape
lagged the docstring).
- cancel_workstream.json: document the conditional ``dropped``
payload — ``was_running`` always present when ``dropped`` is,
``pending_approval`` and ``queued_messages`` conditional sub-shapes.
- spawn_workstream.json: document full return shape including
``routing_strategy ∈ {rendezvous, target_node, resume}`` and
``status``.
- close_all_children.json: clarify ``skipped`` covers BOTH
hard-deleted children AND already-closed-and-evicted children
(wire shape doesn't distinguish); drop incorrect "echoed back
in response" claim — server returns ``{status, closed, failed,
skipped}``, never echoes ``reason``.
- console/server.py: comment in ``_fanout_on_children`` clarifying
that the 400 "No session" branch fires for cancel-cascade
callers and is unreachable from close_all_children (close
handler 404s instead).
- coordinator_client._utc_now_iso(): switch to bare ISO format
matching the rest of the storage row format used in the codebase.
2. Tightened the 11 longest tool descriptions (~23% cut on the
coord set). Removed ALL-CAPS emphasis, normalised em-dashes,
dropped informal phrasing. No new claims.
3. Removed static approval annotations from descriptions.
Approval is governed at runtime by the unified ``approve_tools``
body and admin-defined ``tool_policies`` (#436); static
"Auto-approved" / "Approval required" / per-action approval
tags become a stale signal. Field names (``pending_approval``)
and operational verb behaviour ("cancel unblocks pending
approvals") stay.
4. Renamed ``task_list`` coord tool → ``tasks``. The previous name
compounded the bare word ``task`` (which collides with chat-template
channels on local models — same reason ``task_agent`` carries
the suffix); the plural form sidesteps the collision and reads
more accurately, since the tool acts on the whole list rather
than a single task. Sweep covers tool JSON, Python methods (5
client methods + 2 session methods + 1 helper + 1 constant),
audit event name (``task_list.update`` → ``tasks.update``), log
tag (``task_list.corrupt_envelope`` → ``tasks.corrupt_envelope``),
frontend SSE event matcher, prompts, docs, and tests. CHANGELOG
entry added.
Plus: dropped the ENV block (Output Environment / Available
rendering / Formatting principles) from coordinator system
prompts. Coordinators orchestrate rather than render rich output
to the user, so the rendering capability matrix is not actionable
for them. Coord prompt drops ~29% (6309 → 4493 chars).
SDK regeneration via ``generate-types.py`` updates both
``openapi-console.json`` (the rename's downstream change) and
``openapi-server.json`` (PR #436 drift — its merge added
``pending_approval_detail`` + ``recent_auto_approvals`` fields to
the Python schemas but didn't regenerate the JSON artifact).
## Behavior changes (operator-visible)
- Audit event name: ``task_list.update`` → ``tasks.update``.
Audit dashboards / SIEM filters / log greps that pinned the old
prefix should update.
- SSE ``tool_result`` events now ship ``name="tasks"`` for the
scratchpad tool. The bundled coord-tree UI is updated atomically;
external consumers reading SSE events by tool name need to update.
- Existing task envelopes in production storage have ``+00:00``
timestamps from the old ``_utc_now_iso``. New writes are bare;
old rows are not backfilled. Within an envelope you may briefly
see mixed formats until each row is re-touched. No code path
string-compares timestamps within an envelope, so this is
cosmetic.
## Validation
- ``ruff check`` + ``ruff format --check`` clean
- ``mypy turnstone/`` clean (175 source files)
- ``pytest -m "not live"`` — 4679 passed, 3 deselected
|
||
|
Patrick Buckley
|
acbe18d5f5 |
docs: apply Copilot review feedback on PR #419
Server-side history endpoint declared ``error_codes=[404]`` but the
lifted ``make_history_handler`` factory can also return:
- ``400`` on empty ``ws_id`` (defensive — Starlette routing makes
it unreachable in practice, but the factory has the branch).
- ``500`` on the ``cfg.list_kind is None`` misconfig gate added in
the /review fix-up (defense-in-depth fail-loud; both production
cfgs wire ``list_kind`` so the gate doesn't fire today).
- ``503`` via ``cfg.manager_lookup`` when the kind's manager isn't
available (interactive's lookup never returns 503; coord's can).
Updated ``server_spec.py`` to ``[400, 404, 500, 503]`` per Copilot's
suggestion — matches the existing detail entry's shape so the two
endpoints document the same possible-error envelope.
Caught the parallel asymmetry on ``console_spec.py``: history was
``[403, 404, 503]`` but the lifted factory's misconfig + empty-
ws_id branches reach coord too. Updated to
``[400, 403, 404, 500, 503]`` — same factory body, same possible
responses, plus ``403`` from coord's ``admin.coordinator``
permission gate.
Regenerated ``openapi-{server,console}.json``. No code changes;
spec metadata only. Tests + lint + mypy unchanged.
|
||
|
Patrick Buckley
|
d555816016 |
refactor(core): lift history + detail verb bodies across both kinds (Stage 2 verb lift)
Last verb-shape lift before v1.5.0 stable can tag. Adds two new
factories to ``turnstone/core/session_routes.py``:
- ``make_history_handler(cfg)`` — body lifted from coord's
``coordinator_history`` near-verbatim. ``?limit=`` query param
defaults to 100, clamps to [1, 500], malformed values fall back
to 100. Storage operations (``get_workstream`` on the
storage-fallback path, ``load_messages`` for the row read) now
run via ``asyncio.to_thread`` (was inline pre-lift on coord).
- ``make_detail_handler(cfg)`` — body lifted from coord's
``coordinator_detail``. Lazy-rehydrates a closed/evicted
workstream via ``mgr.open()`` on miss; mirrors
:func:`make_open_handler`'s exception envelope (``ValueError``
→ 503 with the session-factory's remediation text; bare
``Exception`` → correlation_id'd 500 with the per-kind noun
via ``cfg.audit_action_prefix``).
NO new ``SessionEndpointConfig`` fields — the factories reuse
``permission_gate``, ``manager_lookup``, ``not_found_label``,
``audit_action_prefix``, and (for history's storage-fallback
kind check) ``list_kind`` — all already wired by both production
lifespans for the list/saved factories.
Coord side: ``coordinator_history`` and ``coordinator_detail``
standalone handler bodies removed from ``console/server.py``;
``register_session_routes`` now wires
``history=make_history_handler(coord_endpoint_config)`` and
``detail=make_detail_handler(coord_endpoint_config)``.
Interactive side: GAINS both endpoints as a feature gain. Pre-lift
interactive had no ``GET /v1/api/workstreams/{ws_id}`` and no
``GET /v1/api/workstreams/{ws_id}/history`` — SDK consumers had to
subscribe to ``/events`` SSE just to read display fields or
message rows. The same lifted factories are wired with the
interactive endpoint config; cross-kind isolation is preserved on
both sides (history via ``cfg.list_kind`` storage-fallback gate
+ fail-loud-on-misconfig 500; detail via ``mgr.open()``'s internal
kind check).
Pydantic schemas: ``CoordinatorDetailResponse`` /
``CoordinatorHistoryResponse`` removed from ``console_schemas.py``;
``WorkstreamDetailResponse`` / ``WorkstreamHistoryResponse`` added
to ``server_schemas.py`` (mirrors the list lift's pattern for
``WorkstreamInfo``). Both server and console OpenAPI specs
reference the unified schemas; ``server_spec.py`` gains
``EndpointSpec`` entries for the new interactive endpoints. TS
SDK gains both interfaces in ``sdk/typescript/src/types.ts``;
``openapi-{server,console}.json`` regenerated.
Tests: 6 new coord regression/parity tests in
``test_coordinator_endpoints.py`` (limit clamping, cross-kind 404
on storage fallback, storage-only history, detail 503 on
session-factory misconfig, detail 500 with correlation_id on
unexpected rehydrate failure, history swallows
``load_messages`` exception → 200 with empty messages). 10 new
interactive parity tests in ``test_workstream_endpoints.py``
(``TestHistoryInteractive`` + ``TestDetailInteractive``). 1 new
openapi spec test pinning the server-side ``?limit=`` query param.
Total: ``4490 → 4491`` after the new exception-swallow
regression test landed. ``ruff check`` clean, ``mypy`` clean on
touched files.
/review pipeline (4 finders → verify → dedupe) caught 1 Minor
defense-in-depth (bug-1/sec-1, merged: ``make_history_handler``
fail-closed gate when ``cfg.list_kind is None``, mirroring
``make_saved_handler``'s same gate) + 1 Minor test-helper rename
(q-1: ``_interactive_history_cfg`` → ``_interactive_endpoint_cfg``)
+ 4 Nits (q-2 unused fixture parameter, q-3 CHANGELOG TS SDK
mention, q-4 missing exception-swallow regression test, q-5
misleading test comment) — all addressed in the same commit.
|
||
|
Patrick Buckley
|
edf52016ac |
refactor(core): lift list + saved verb bodies across both kinds (Stage 2 verb lift)
New ``make_list_handler(cfg)`` and ``make_saved_handler(cfg)``
factories in ``turnstone/core/session_routes.py`` replace four
pre-lift bodies (interactive ``list_workstreams`` +
``list_saved_workstreams``; coord ``coordinator_list`` +
``coordinator_saved``). Same factory + capability-flag pattern as
the merged cancel / open / events / create lifts.
Four new ``SessionEndpointConfig`` fields:
- ``list_resolve_titles: ListResolveTitles | None`` — bulk lookup
``(ws_ids) -> {ws_id: title-or-None}``. Interactive wires
``get_workstream_display_names`` (new bulk helper added on the
storage layer + memory.py); the lifted body resolves every active
row in ONE ``SELECT ... WHERE ws_id IN (...)`` instead of the
pre-lift N+1 (one SELECT per row).
- ``list_kind: WorkstreamKind | None`` — explicit kind classifier
for the saved-list storage filter. Replaces the initial draft's
``audit_action_prefix == "coordinator"`` string compare which
would have silently leaked INTERACTIVE rows for any future kind
whose audit prefix didn't match. Required when a kind mounts
list/saved; misconfig surfaces as a 500 with a clear log line.
- ``saved_state_filter: str | None`` — coord wires ``"closed"``;
interactive wires ``None``.
- ``saved_loaded_lookup: SavedLoadedLookup | None`` — coord-only
defence-in-depth filter that excludes ws_ids in the warm pool.
Behaviour changes (all observable in CHANGELOG):
- **Active-list row shape converges on always-include** ``{ws_id,
name, state, kind, parent_ws_id, user_id}``. Interactive renames
``id`` → ``ws_id``; both kinds populate every field (coord adds
kind + parent_ws_id; interactive adds user_id).
- **Top-level response key converges on ``"workstreams"``** on
both endpoints. Coord ``coordinators`` key removed — coord is a
1.5.0aN-only surface (never shipped stable) so the convergence
has no compat shim; SDK / frontend consumers swap once.
- **Storage + manager-lock work moved off the event loop on
interactive**. ``list_workstreams_with_history`` runs through
``asyncio.to_thread`` on both kinds (matches coord's pre-existing
perf-2 pattern from the saved-coordinators review); ``mgr.list_all``
+ per-row work also offloaded.
- **N+1 storage round-trips on /v1/api/workstreams eliminated**.
Pre-lift interactive resolved the alias for every active row in a
separate SELECT (up to 50 round-trips per dashboard refresh on a
saturated node). Lifted body issues one bulk SELECT.
Pydantic schemas: ``WorkstreamInfo.id`` renamed → ``ws_id``,
``WorkstreamInfo.user_id`` field added. ``CoordinatorInfo`` and
``CoordinatorListResponse`` removed (folded into the unified
``WorkstreamInfo`` / ``ListWorkstreamsResponse``). OpenAPI spec
snapshots regenerated. TS SDK types updated (``WorkstreamInfo``
interface gains ws_id + the always-include fields); TS test
mock + assertion updated to match.
``GET /v1/api/dashboard`` is intentionally NOT in this PR's scope
and still returns rows keyed on ``id``. Tracked as a separate
cleanup PR (tombstone-note added at the dashboard handler).
/review pipeline run; the four Major findings + one Minor + six
nits all addressed in the same commit:
- M1: TS SDK ``WorkstreamInfo`` interface stale (id: string) →
renamed + fields added.
- M2: TS SDK test masked the type-mismatch with stale mock → updated.
- M3: N+1 alias resolution on active list → bulk
``get_workstream_display_names`` helper + ``list_resolve_titles``
bulk cfg hook.
- M4: Missing interactive parity regression test for unified row
shape → mirror of coord's added in test_server_authz.py.
- Mi1: ``audit_action_prefix`` string-compare deriving kind →
explicit ``cfg.list_kind: WorkstreamKind`` field.
- Six nits: redundant inner asyncio import, forward-ref quotes on
Awaitable, duplicated frontend comments, dashboard ``id`` field
has no tombstone-note, empty-coord_mgr short-circuit on
``saved_loaded_lookup``.
4512 tests passing; ruff + mypy clean.
|
||
|
Patrick Buckley
|
9ed8b1e0b5 |
refactor(core): lift create verb body across both kinds (Stage 2 verb lift)
New ``make_create_handler(cfg, *, audit_emit=None)`` factory in ``turnstone/core/session_routes.py`` consumes five new ``SessionEndpointConfig`` fields (``create_supports_attachments``, ``create_supports_user_id_override``, ``create_validate_request``, ``create_build_kwargs``, ``create_post_install``) and replaces both ``create_workstream`` and ``coordinator_create`` bodies. Same factory + capability-flag pattern as the merged cancel / open / events lifts. ``_validate_and_save_uploaded_files`` lifted to ``turnstone.core.attachments`` so both processes call one kind-agnostic implementation. Coord parity gains (§ Post-P3 reckoning item #1 + carry-forward): - Create-time attachments: multipart parsing, validate+save+rollback, ``attachment_ids`` on the response. Coord adapter ``send`` doesn't yet reserve attachments at create time, so the rows save as pending and the next ``/send`` picks them up via the standard send-with-attachments path. - Disabled-skill rejection (matches interactive's pre-lift gate). - Always-include response shape ``{ws_id, name, resumed, message_count, attachment_ids}`` populated with default ``False``/``0``/``[]`` on the fields coord doesn't fill. - 200 status (was 201). - Audit-emit failures swallow + warning log instead of 500. Both kinds converge on the manager-at-capacity 429, factory-misconfig 503, and correlation_id'd 500 for unexpected ``mgr.create`` failure (interactive lifted up to coord's safer error envelope). Three /review fixes folded in: - ``notify_targets`` malformed input gates at the validator (400) instead of bubbling out of post_install as a 500 — pre-fix the workstream had already been created + audited + broadcast by the time the validation raised. - Skill-lookup storage failures now share the correlation_id'd 500 path with ``mgr.create`` (was masquerading as 400 "Skill not found"). - Whitespace-only ``skill`` field treated as empty (matches pre-lift coord). CHANGELOG entry under [Unreleased] documents every observable behaviour change. OpenAPI spec regenerated. Three new coord regression tests (create-time-attachments save pending rows, always-include parity fields, disabled-skill rejection) plus one interactive regression test (notify_targets 400). 4500 tests passing. |
||
|
Patrick Buckley
|
02e4a01207 |
fix(core,server): apply Copilot review feedback on PR #411
Five fixes from Copilot's review of Stage 2 P1.5 — all preserve
behaviour, narrow docstring claims, and round out the response shape:
* **session_routes.py:supports_attachments docstring** — claimed
the handler "accepts only ``{"message": ...}``" when ``False``,
but the implementation silently ignores ``attachment_ids``
rather than rejecting. Updated wording to say the
attachment-resolution block short-circuits and any
``attachment_ids`` are silently ignored. Behaviour unchanged
(silent-ignore is the right choice for forward compat — clients
passing ``attachment_ids`` speculatively to a not-yet-lit-up
kind shouldn't get a 400).
* **session_routes.py:queue_full response shape** — restored the
always-include guarantee for ``attached_ids`` /
``dropped_attachment_ids``. The queue_full path now returns
``attached_ids: []`` and ``dropped_attachment_ids: list(requested_ids)``
so SDK consumers don't have to branch on status.
* **server.py:_interactive_spawn_metrics guard** — added
``_ws_turn_tool_calls`` to the ``hasattr`` chain. Previously
the guard checked ``_ws_lock`` + ``_ws_messages`` and then
unconditionally assigned ``_ws_turn_tool_calls`` — would
raise on a SessionUI subclass with the first two but not the
third.
* **console_spec.py:coord_send error_codes** — added 409
(the 'session UI not available' branch in
``make_send_handler`` returns 409, but the spec didn't list
it). OpenAPI spec regenerated; TS SDK types refreshed.
* **session_routes.py:tenant_check docstring** — claimed
interactive uses ``_require_ws_access`` with "404 on owner
mismatch", but the helper now delegates to
``resolve_workstream_owner`` which explicitly does NOT enforce
row-level ownership (trusted-team semantics; 404s only on
missing rows). Updated wording to match.
|
||
|
Patrick Buckley
|
e0c78e2aec |
test,docs: coord attachment + queue parity tests + spec regen + CHANGELOG
Five new TestCoordinatorAttachments tests in ``tests/test_coordinator_endpoints.py`` exercising the lifted attachment surface end-to-end on coord: * upload → list round-trip * get_content returns raw bytes with text/plain forced for text * delete removes pending entries and clears them from the listing * send with attachment_ids consumes pending under the send_id token * send response carries attached_ids / dropped_attachment_ids even on plain-text sends (unified shape parity) The existing ``_coord_endpoint_config`` fixture grew capability flags to mirror the production console wiring, and ``_make_client`` now mounts the four coord attachment routes via ``make_attachment_handlers``. OpenAPI specs regenerated; TS SDK bumped to 0.5.0. CHANGELOG entry under [Unreleased] documents the verb-shape lift, the coord attachment surface coming online, the response-shape change for ``coordinator_send``, the unification of the three lifted classifier / lock helpers under ``turnstone.core.attachments``, and the new SDK helpers. |
||
|
Patrick Buckley
|
4a72b2ce19 |
build(sdk): regen openapi specs + bump TS SDK to 0.4.0
Stage 2 Priority 0 Step 0.5 follow-on. The handwritten Python
OpenAPI spec already moved to ``/v1/api/workstreams/`` in the URL
sweep commit; this just regenerates ``sdk/typescript/openapi-{server,console}.json``
from those specs so generated TS callers see the new paths.
Bumps the TS SDK to 0.4.0 to flag the URL-shape break for any
1.5.0aN-era consumer of the experimental coord client. Python SDK
needs no change — it never exposed the coord HTTP surface.
TS typecheck + 32 vitest tests pass.
|
||
|
Patrick Buckley
|
58d20f4012 |
chore(coord): remove spawn-quota subsystem (#403)
* chore(coord): remove spawn-quota subsystem The quota gate was operator-level safety per its own comments, not a security boundary, and never fired in a week of heavy use. Runaway coordinator spawns are already bounded by max_active slot exhaustion, which surfaces to the coord LLM as a tool error — same operational shape, one fewer moving part. Precedes the Stage 1 SessionManager unification so the coord tool doesn't inherit quota bookkeeping. Upgraded deployments with the three removed settings persisted will log three "Skipping invalid setting" warnings on startup and otherwise degrade cleanly; a follow-up migration to delete the rows would silence that noise. * chore(migrations): drop stale coord spawn-quota settings rows (047) Clears persisted rows for the three ConfigStore keys removed in the previous commit so upgraded deployments don't log "Skipping invalid setting" warnings on every startup. Downgrade is a no-op — the rows were operator-set values, and a rollback to pre-1.5.0 code falls back to the registry defaults for any key not present. |
||
|
Patrick Buckley
|
67aaa236e8 |
feat(coordinator): phase 8 PR B — spawn budget + rate limit + /quota endpoint (#387)
* feat(coordinator): phase 8 PR B — spawn budget + rate limit + /quota endpoint
Adds two complementary controls so a runaway coordinator can't saturate a
cluster's max_active without anyone noticing:
- **Spawn budget** (hard quota) — cap on concurrently active children.
Default 20 per coord. spawn_workstream returns a tool error guiding
the model to close idle children; spawn_batch routes overflow rows to
`denied[]` with partial-success semantics.
- **Spawn rate limit** (soft pacing) — classic token bucket, defaults
5 tokens/minute with burst 10. A rate-limited spawn surfaces a tool
error carrying `retry after Ns` so the model paces itself. Zero
refill rate is honoured as "disable refill" (bucket still honors the
initial burst).
Shipped infra:
- `turnstone/core/spawn_quota.py` — thread-safe `SpawnBudget` +
`TokenBucket`. 15 unit tests.
- `turnstone/core/session.py` — coord-only state built from settings at
__init__. Shared `_eval_spawn_quota(active)` helper drives both the
single-spawn path (wraps the denial reason in `_coord_tool_error`) and
the batch path (annotates `spec["_error"]`). `_count_active_children`
routes through `coord_client.list_children(include_closed=False)` and
fails *open* on lookup error (budget is operator-safety, not security).
- `POST/GET /v1/api/coordinator/{ws_id}/quota` — partial-update admin
endpoint mirroring the /trust + /restrict shape. Accepts either the
nested `spawn_rate` object or flat aliases — supplying both for the
same field returns 400 so the admin UI can't half-migrate silently.
Overrides are in-memory only (die on session reopen). Audits via
`coordinator.quota.updated` with before/after snapshots.
- Settings: `coordinator.spawn_budget`, `coordinator.spawn_rate.tokens_per_minute`,
`coordinator.spawn_rate.burst` with ranges 1..500 / 0..600 / 1..500.
The range bounds are the single source of truth — the endpoint
validators and Pydantic schema both import from `settings_registry.SETTINGS`
so bumping a cap in one place lights up everywhere.
- OpenAPI: `CoordinatorQuotaRequest` / `CoordinatorQuotaResponse` /
`CoordinatorSpawnRateState` schemas + endpoint specs. TS SDK regenerated.
Tests: +15 unit (SpawnBudget + TokenBucket), +17 endpoint (GET + POST
happy paths, range edges, mixed-body rejection, non-object spawn_rate,
service-token refusal), +11 session-side (budget blocks single spawn,
budget batch partial-success, rate batch partial-success, empty-body
reject, mutator live-update, non-coord session has no quota state).
Deferred (not this PR): per-skill scoping via migration 047 +
`prompt_templates.spawn_budget` column. Count-only storage helper
(opportunistic — list_children at budget ≤ 500 is fine behind a
human-gated approval flow).
* fix(coordinator): address PR #387 copilot review
- Budget undercount: _count_active_children used list_children's
LIMIT-then-Python-filter path, so a fan-out with many recently-closed
children could push live rows past the SQL LIMIT and silently
undercount, leaking spawn slots past the budget. Replace with a new
CoordinatorClient.count_active_children that uses
storage.count_workstreams_by_state (SQL aggregate, no pagination,
sums non-terminal states). Tenant-guarded; fails open on storage
error (budget is operator-safety, not a security gate). New client
tests cover the non-terminal count, the closed/deleted exclusion,
the foreign-parent guard, and the fail-open path.
- Service-token bypass on /quota: both GET and POST used the default
allow_service_bypass=True, so a service token whose user_id matched
the coord owner could read or *raise* spawn capacity without the
explicit admin.coordinator grant. Flip both to
allow_service_bypass=False for consistency with /restrict,
/stop_cascade, and /close_all_children.
- OpenAPI contract leak: CoordinatorSpawnRateState was used for both
the request and response shapes, which let generated SDKs imply
clients could POST tokens_available (a read-only bucket reading the
handler ignores). Split into CoordinatorSpawnRateInput (request:
tokens_per_minute + burst only) and CoordinatorSpawnRateState
(response: adds tokens_available). No runtime behaviour change;
SDKs regenerate with two distinct types.
Drops the _ACTIVE_COUNT_SLACK / _ACTIVE_COUNT_MIN_LIMIT constants in
session.py — no longer needed since the new helper takes no limit
argument. Updates the 5 session-side quota tests to stub
count_active_children instead of list_children.
|
||
|
Patrick Buckley
|
7d61f9a37c |
feat(coordinator): phase 8 PR A — spawn_batch + close_all_children batch tools (#386)
* feat(coordinator): phase 8 PR A — spawn_batch + close_all_children batch tools
Adds two model-facing batch tools so a coordinator can fan out without burning one approval per child:
- `spawn_batch` — create up to 10 child workstreams in a single approval. Serialised
spawns so sibling ordering (by created_at) stays deterministic. Returns
`{results: {idx: {ws_id, name, node_id, status}}, denied: [{idx, reason}]}`.
Per-item validation / spawn failures surface in `denied[]`; the batch hard-errors
on >10 rather than silent truncation.
- `close_all_children` — soft-close every direct child in one approval. Server-side
Sem(16) fan-out via `coord_client.close_workstream`; `reason` propagates to every
closed child's audit + workstream_config. Response mirrors `stop_cascade`'s cascade
idiom: `{closed, failed, skipped}` where `skipped` is upstream-404 / already-gone.
Shipped infra:
- New console endpoint `POST /v1/api/coordinator/{ws_id}/close_all_children`
(gated `admin.coordinator`, `allow_service_bypass=False`, 512-char reason cap,
`coordinator.closed_all_children` audit).
- Shared `_fanout_on_children` helper — both `stop_cascade` and `close_all_children`
now delegate to it (one place to own the snapshot → semaphore-gather → bucket-split
skeleton).
- `CoordinatorClient.close_all_children(reason)` plus a `_post_url` seam that
`_post` now reuses (no more duplicated transport-error handling).
- `_emit_batch_event` — best-effort SSE emitter modelled on `_emit_wait_event`.
Emits `batch_started` / `batch_ended` pairs keyed by call_id. Throttled
`batch_progress` deferred to a follow-up.
- OpenAPI request + response schemas, endpoint spec entry, TS SDK regenerated.
- Persona doc (`tools_coordinator.md`) covers the two new patterns.
Bulk-endpoint shape policy (codified in PR C later): split by semantic category —
`{results, denied, truncated}` for bulk-read / bulk-create-with-payload (cluster/ws/live,
spawn_batch), `{<bucket>, failed, skipped}` for cascade-mutation (stop_cascade,
close_all_children). No retrofit needed on stop_cascade.
Tests: new `test_coordinator_close_all_children.py` (8 endpoint tests), expanded
`test_coordinator_tools.py` (session-side prepare/exec, coord_client=None guards,
batch SSE events), expanded `test_coordinator_client.py` (route map, client method,
transport errors), tool-count assertions updated.
Deferred (not this PR): per-item selective-deny approval UI, throttled batch_progress
SSE, coordinator-skills doc + bulk-endpoints doc (PR C), spawn budget / rate limit (PR B).
* fix(coordinator): address PR #386 copilot review
- coordinator_client.close_all_children: pass the unformatted path template
as log_path so telemetry aggregates don't fragment per session (ws_id
still lives in the real URL).
- session.py: drop dead spawned_ids accumulator in _exec_spawn_batch —
leftover from an eager-register path that got removed earlier.
- close_all_children tool JSON: document the 512-char server-side cap on
reason and that reason is echoed back in the response payload. Added
maxLength:512 on the schema property so the LLM sees the constraint.
- CoordinatorCloseAllChildrenRequest: add Field(max_length=512) so the
OpenAPI schema reflects the runtime 400-on-overflow constraint.
|
||
|
Patrick Buckley
|
7c16b0dfa8 |
refactor(routing): replace hash-ring rebalancer with rendezvous (HRW)… (#384)
* refactor(routing): replace hash-ring rebalancer with rendezvous (HRW) hashing Routing was a stored bucket table maintained by a central rebalancer daemon, which shared its liveness primitive (services.last_heartbeat) with the collector — when a heartbeat-fresh node went into a zombie HTTP-handler-broken state, neither the collector nor the rebalancer could self-correct, and the router kept directing traffic at it. Rendezvous hashing makes the route a pure function of (ws_id, live_services) so the heartbeat is the single source of truth and any liveness-eviction propagates to the next route call without a separate state-publication step. The rebalancer's central state has no analogue: the new router computes the per-key node winner on every call, the collector pushes membership updates into the router cache from its discovery thread, and per-route overrides survive on workstream_overrides. Eager workstream migration goes away; in-flight workstreams lazily rehydrate from storage on the new owner — already the dead-node behaviour. * fix(tools): describe rendezvous re-routing on spawn/inspect node_id The first pass overclaimed `node_id` "stays canonical for this workstream's lifetime" — under rendezvous routing the active owner re-derives per-call from live membership, so a node join/drop after spawn can shift it. Tool descriptions now say `node_id` is the spawn-time binding; subsequent ops re-route via rendezvous over the current live-node set; the new owner lazily rehydrates from shared storage; coordinators should re-read with inspect_workstream rather than caching the value. |
||
|
Patrick Buckley
|
9826ea15c5 |
feat(coordinator): phase 7 — governance + skill metadata + cross-cutt… (#383)
* feat(coordinator): phase 7 — governance + skill metadata + cross-cutting invariants
Combines three stacked sub-PRs into a single coordinator phase-7
shipment against the phase-7 plan doc. The sub-PR structure (0 / A /
B) preserved on individual branches for reviewer drill-down; this
branch is the one reviewers should merge.
## Sub-PR 0 — service-auth boundary invariants
Shared helpers and contracts that lock the console ↔ node service-auth
boundary so later authz surfaces use them by construction.
- ``_effective_user_filter(request)`` in both ``turnstone.console.server``
and ``turnstone.server`` with a shared ``DENY_EMPTY_SUB`` sentinel
on ``turnstone.core.auth``. Three-way return — admin/service
bypass, scoped caller uid, or fail-closed sentinel on blank sub.
Four callsite migrations (``_coordinator_rows``,
``coordinator_children``, ``coordinator_metrics``,
``cluster_ws_live_bulk``).
- ``StorageBackend`` class docstring codifies the tenancy contract
(every list/count/aggregate method must accept ``user_id: str |
None = None`` and push ``WHERE user_id = :user_id`` into SQL) and
the ``_mapping`` row-access contract. New
``turnstone.testing.row_contract`` ships ``assert_row_like()``.
- ``_verify_collector_service_scope`` probes an upstream node at boot
with ``expected_node_id=_scope-probe_``; a 409 proves the scope
gate was passed, a 403/401 sets ``collector_scope_error`` and
causes ``cluster_snapshot`` / ``cluster_events_sse`` to return 503
with a remediation hint. Probe URL allowlist rejects non-http(s)
schemes and 169.254.0.0/16 hosts.
- 4xx log-level floor on ``_NodeDashboardCache.get``,
``_fetch_live_block``, and ``_proxy_sse`` — dotted-hierarchy
prefixes with bounded body previews. ``_bounded_body_preview`` and
``_bounded_stream_preview`` strip control chars.
## Sub-PR A — coordinator governance core
Mid-session governance surface for coordinator workstreams.
- **Trusted-session mode.** New ``coordinator.trust.send``
permission (migration 042). ``ChatSession.set_trust_send`` /
``revoke_tools`` methods with a ``_governance_lock``. ``POST
/v1/api/coordinator/{ws_id}/trust {send: bool}`` double-gated on
``admin.coordinator`` AND ``coordinator.trust.send`` with
``allow_service_bypass=False`` so service tokens can't escalate.
``_prepare_send_to_workstream`` auto-approves sends whose target is
in the coordinator's own subtree; foreign ws_ids still require
approval. ``_is_own_subtree`` checks both ``parent_ws_id`` AND
``user_id`` to defend against cross-tenant row corruption.
- **Audit-layer credential redaction.** ``record_audit`` walks
``detail`` (dicts, lists, tuples, sets, frozensets; keys too)
and routes every string through ``redact_credentials`` + a C0
control-char scrub. New kw-only ``raw_detail=True`` opt-out.
``_has_any_string`` fast-path. Audit action registry extended
with the four new governance sub-prefixes.
- **Mid-session revocation + cascading stop.** ``POST
/v1/api/coordinator/{ws_id}/restrict {revoke: [...]}`` caps 256
entries / 128 chars; ``_prepare_tool`` short-circuits with a
tool-error. ``POST /v1/api/coordinator/{ws_id}/stop_cascade``
cancels the coord's in-flight generation then dispatches
``cancel_workstream`` for every direct child in parallel via
``asyncio.gather`` bounded by ``Semaphore(16)``. Per-child
outcomes split into ``cancelled`` / ``failed`` / ``skipped``
(404 = already-gone rather than dispatch-broken). Both endpoints
apply ``allow_service_bypass=False`` on the admin gate.
- **Shared plumbing.** ``_resolve_coord_session`` helper collapses
the handler prelude three endpoints shared. ``_emit_coord_audit``
wraps ``record_audit`` in a dedicated ``ThreadPoolExecutor``
(``app.state.audit_executor``) so audit bursts don't starve cancel
dispatches. ``_require_json_object`` guards body parsing so non-
object JSON returns 400 instead of 500.
## Sub-PR B — skill metadata governance
- **Description validator (migration 043).** ``prompt_templates``
rows now require a non-empty ``description``. Existing empty rows
get backfilled with a ``"Skill: <name>"`` placeholder on upgrade.
The installer (``admin_skill_discover``) and MCP prompt sync both
synthesise a placeholder when the upstream description is blank
so non-admin write paths satisfy the invariant.
- **Skill kind classifier (migration 044).** New
``prompt_templates.kind`` column (``interactive`` / ``coordinator``
/ ``any``; defaults to ``any``). New
``turnstone.core.skill_kind.SkillKind`` StrEnum is the single
source of truth; Pydantic schemas type ``kind`` as ``SkillKind``
(OpenAPI advertises the enum) and the handler validator catches
the ValueError. ``list_skills_filtered`` gains a
``kinds: list[str] | None = None`` SQL filter.
``CoordinatorClient.list_skills`` defaults to
``kinds=["coordinator", "any"]`` so interactive-only skills are
hidden from the orchestrator.
- **``scan_status`` → ``risk_level`` rename (migration 045).**
Lossless column rename to align with ``IntentVerdict.risk_level``
terminology. Swept storage (both backends + schema + protocol),
handlers, API schemas, tool JSON, generated OpenAPI specs,
TypeScript SDK types, frontend (``governance.js``), tests, and
English prose in ``docs/judge.md`` + ``docs/tools.md``. The
user-facing on-load warning now reads ``has risk level:
{risk_tier}``. Tool JSON's ``risk_level`` enum corrected to the
scanner's actual taxonomy (``safe / low / medium / high /
critical``; was the never-shipped ``clean / flagged / unscanned /
pending``). Historical migration 021 left untouched.
## Migrations
042 (``coordinator.trust.send`` perm — PR A)
043 (description backfill — PR B)
044 (``kind`` column add — PR B)
045 (``scan_status`` → ``risk_level`` rename — PR B)
All four use position-anchored permission strings / host-side
parse-filter-rejoin on downgrade where SQL ``REPLACE`` could
corrupt prefix-overlapping values.
## Verification
- ``ruff check turnstone tests`` clean.
- ``mypy turnstone`` clean on 165 source files.
- ``pytest -m "not live"``: 4431 passed (+85 over the phase-6
baseline). Includes +32 tests in ``tests/test_service_auth_boundary.py``
and +38 in ``tests/test_coordinator_governance.py``; shared fixtures
extracted to ``tests/_coord_test_helpers.py``.
- Generated OpenAPI JSON (``sdk/typescript/openapi-{console,server}.json``)
regenerated via ``sdk/typescript/scripts/generate-types.py``; zero
``scan_status`` occurrences remaining outside the historical
migration 021 and the rename migration 045.
## Security reviews
Both reviews flagged by the phase-7 plan (items 1 + 5, plus 0a's
refuse-to-serve gate) ran through the multi-stage ``/review``
pipeline twice per sub-PR; all confirmed findings landed in-branch.
* fixup(phase-7): CI lint + PR #383 review fixups
Addresses the lint CI failure (ruff format) plus 12 findings from the
two automated PR reviewers.
Copilot:
- ``_sqlite.list_installed_skill_urls`` / ``_postgresql.list_installed_skill_urls``
used positional row indexing (``r[0]``/``r[1]``/``r[2]``) while this
same PR's ``StorageBackend`` class docstring forbids it. Switched
both to ``r._mapping["..."]`` access.
- ``list_skills.json`` previously advertised ``risk_level=""`` as a
filter for unscanned skills, but the implementation treats empty
strings as "no filter". Clarified the tool description to say
omit the filter entirely to include unscanned rows, and added an
explicit ``enum`` on the parameter restricting it to the scanner
tiers. ``_prepare_list_skills`` keeps the ``strip() or None``
normalisation — unscanned filtering now has an unambiguous contract.
- ``test_storage_skills_filtered.test_risk_level_filter`` used the
legacy ``clean`` / ``flagged`` values from the pre-rename column.
Rewritten with the scanner's actual taxonomy (``safe`` / ``high``).
github-code-quality (CodeQL):
- ``test_deny_sentinel_is_singleton`` previously asserted
``cs.DENY_EMPTY_SUB is cs.DENY_EMPTY_SUB`` — an identical-expression
comparison. Rewritten as two separate ``from ... import ... as`` aliases
(``FIRST_READ`` / ``SECOND_READ``) so the identity check is between
distinct bindings.
- ``test_restrict_empty_revoke_is_noop_but_audits`` unpacked ``state``
without using it. Renamed to ``_state``.
- Mixed import styles in ``test_service_auth_boundary.py`` — the
file previously used both ``import turnstone.console.server as cs``
and ``from turnstone.console.server import ...`` for the same
module (same story for ``turnstone.core.auth`` and
``turnstone.server``). Consolidated to the ``from X import Y`` style
used elsewhere in the file; the ``_fetch_live_block`` test now
patches via pytest's ``monkeypatch`` fixture instead of a manual
rebind through a module alias.
CI:
- ``ruff format`` reformatted one line in
``tests/test_coordinator_endpoints.py``.
Verification: ruff check + mypy clean (166 files); 4459 non-live
pytest pass.
* fix(tests): swap asyncio marker for anyio in service-auth boundary tests
PR #383 CI caught that the 13 ``@pytest.mark.asyncio`` decorators I
added in ``test_service_auth_boundary.py`` are an off-convention
choice — the rest of the repo uses ``@pytest.mark.anyio`` (148 sites
vs my 13). The CI environment pulls in ``anyio`` but not
``pytest-asyncio``, so every async test in this one file was failing
with "async def functions are not natively supported". It passed
locally by accident — my dev venv happens to have pytest-asyncio
installed ambiently.
Swapped all 13 marker sites to ``@pytest.mark.anyio``. No functional
change; the tests run under the same default asyncio backend anyio
provides.
Verification: ruff + mypy clean (166 files); 4459 non-live pytest
pass.
|
||
|
Patrick Buckley
|
42e99d6990 |
docs: update tools, architecture, SDK for v0.9.2 changes
- docs/tools.md: batch edit_file (edits array), bash stderr prefix, math sandbox extras, output truncation - docs/judge.md: JSON secret detection in output guard - docs/architecture.md: state_change now sent to per-workstream SSE - README.md: [sandbox] extras group in requirements - TypeScript SDK: StateChangeEvent type, type guard, exports - OpenAPI specs regenerated |
||
|
Patrick Buckley
|
da5bf90a4b |
feat: model detect button, capabilities API, and model dropdowns (#215)
* feat: model detect button, capabilities API, and model dropdowns Admin Models tab: add Detect button that probes a model endpoint to verify reachability, list available models, detect context_window, and identify server type (llama.cpp/vLLM/SGLang/OpenAI/Anthropic). Add static capability lookup endpoint for auto-filling form fields when a known model name is entered. Add known-models endpoint for datalist autocomplete suggestions. Add "openai-compatible" as a third provider option for local servers, keeping the OpenAI SDK under the hood but suppressing capability auto-fill and known-model suggestions. Replace free-text model input with a select dropdown in both console and server new-workstream modals, populated from a new lightweight GET /v1/api/models endpoint. New endpoints: - POST /v1/api/admin/model-definitions/detect - GET /v1/api/admin/model-capabilities - GET /v1/api/admin/model-capabilities/known - GET /v1/api/models (both console and server) * fix: accumulate signature_delta for Anthropic thinking blocks (#214) The streaming path captured thinking_delta events but not signature_delta, leaving the signature empty on round-trip and causing 400 errors on multi-turn conversations with thinking enabled. * fix: address PR review — empty base_url, capability leak, response schemas - Don't pass empty base_url to OpenAI client (falls back to SDK default) - Return None from lookup_model_capabilities for openai-compatible provider - Only use static capability table for known models in _detect_openai_compat, avoiding misleading 200k default for unknown local models - Add AvailableModelInfo + ListAvailableModelsResponse schemas to both console_spec and server_spec - Regenerate TypeScript SDK OpenAPI snapshots * fix: apply same known-model guard to Anthropic context_window detection Only report context_window from the static capability table when the Anthropic model is actually known, matching the OpenAI path fix. * ui: add autocomplete hint to Model ID label in admin modal * fix: use explicit kwargs for OpenAI() to satisfy strict mypy |
||
|
Patrick Buckley
|
4f6ef13ce9 |
fix: cancel button race condition with stream abort and force cancel (#202)
The cancel endpoint emitted a 'cancelled' SSE event before the worker thread terminated. The frontend transitioned to "send" mode prematurely, so the next send got rejected with "Already processing a request." Backend: - Providers expose SDK stream handle via cancel_ref parameter so cancel() can close the HTTP connection and unblock iteration - Generation counter prevents orphaned threads from mutating messages or clearing cancel state after force cancel - _check_cancelled() added between retry attempts in _try_stream - Server polls (async, non-blocking) for cancelled worker to exit - Force cancel (force:true) abandons stuck worker, keeps cancel event set so subprocesses are killed, guards against spurious SSE events Frontend: - 'cancelled' shows "Cancelling..." then escalates to "Force Stop" after 2s for a harder cancel that abandons the worker immediately - 10s safety timeout auto-recovers if stream_end never arrives - busy_error re-enables stop button instead of showing send - Timeout cleanup in disconnectSSE, stream_end, and force .then() - Layout shift prevention (min-width, white-space: nowrap) - aria-label updates for accessibility Tests: - 7 new tests: stream close, error suppression, cancel_ref population, transport error conversion, non-cancel exception propagation, retry cancellation check |
||
|
Patrick Buckley
|
04c50568e9 |
feat: add priority column for skill ordering control (#144)
* feat: add priority column for skill ordering control Add priority INTEGER DEFAULT 0 column to prompt_templates (migration 024). Skills with activation="default" are now ordered by priority ASC, name ASC instead of name-only. Admins can set priority via create/update API. Lower values run first. Priority is editable on readonly/installed skills. 4 new tests. Python SDK, TypeScript SDK, and Pydantic models updated. * fix: address review — apply priority ordering to list_default_templates list_default_templates() still ordered by name only, so priority had no effect on default skill execution order. Update both SQLite and PostgreSQL backends to order by (priority, name). * fix: address review — regenerate OpenAPI snapshot, add default template ordering test Regenerate openapi-console.json to include priority field on skill models. Add test_list_default_templates_ordered_by_priority to verify the execution path for default skills respects priority ordering. |
||
|
Patrick Buckley
|
86b404177b |
chore: bump version to 0.8.4
- feat: split-pane layout for chat UI (#127) - fix: enforce CSS min dimensions during split handle drag - feat: add OpenShell sandbox policy for turnstone-server (#128) - fix: collector JWT expiry causes silent workstream data wipe (#126) - fix: auto-titler SSE event + SSE reconnection after restart (#125) - fix: wire resume_ws through console + expose max_ws in heartbeat (#124) |
||
|
Patrick Buckley
|
ec3454ee2e |
fix: wire resume_ws through console + expose max_ws in heartbeat (#124)
* fix: wire resume_ws through console + expose max_ws in heartbeat Console create_workstream handler now reads resume_ws from the request body and passes it to CreateWorkstreamMessage on all three dispatch paths (pool, auto, explicit). Previously resume only worked via channel router and direct CLI — the console layer never plumbed it through. Server /health now includes max_ws from WorkstreamManager. Bridge reads it on startup and includes it in heartbeat metadata so the console's _pick_best_node gets accurate capacity instead of always defaulting to 10. Collector also updates max_ws on subsequent heartbeats (not just discovery). Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks fixed for new max_workstreams property access in /health. * fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id Add _fetch_server_metadata() so bridge reads max_ws from /health even when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats would advertise max_ws=10 regardless of actual server config. Add 3 test cases verifying resume_ws flows through all three console dispatch paths (directed, pool, auto-select). |
||
|
Patrick Buckley
|
341d2f604f |
fix: address PR #118 review feedback
- Regenerate OpenAPI snapshots (openapi-console.json) to include license and compatibility fields in SkillInfo/CreateSkillRequest/UpdateSkillRequest - Omit version from create/update payloads when blank so server applies default "1.0.0" instead of storing empty string - Push enabled_only + limit filters into list_skills_by_activation storage query (protocol, SQLite, PostgreSQL) instead of loading all rows and filtering in Python; session.py now passes enabled_only=True, limit=30 - License length cap ([:128]) was already applied in previous commit |
||
|
Patrick Buckley
|
88085c29ff |
fix: normalize install response + review fixes
Address 5 Copilot review items + code review findings:
- Normalize install endpoint to always return envelope response:
{installed: [...], skipped: [...], total: N} — eliminates dual
response shape (single SkillInfo vs batch). Breaking change to
install endpoint response, SDKs and OpenAPI spec updated.
- Add SkillInstallResponse + SkillInstallSkipped Pydantic models
- POST /resources spec now correctly documents response_code=201
- SQLite count_skill_resources_bulk chunks IN clause at 900 to stay
under SQLITE_MAX_VARIABLE_NUMBER (999)
- Fix installDiscoveredSkill() JS handler for envelope response
- Add error key to 409 duplicate response for error handler compat
- Update Python SDK install_skill return type (dict, not SkillInfo)
- Add TypeScript SkillInstallResponse + SkillInstallSkipped types
- Regenerate openapi-console.json
- Update all install tests for envelope response shape
|
||
|
Patrick Buckley
|
4b44d88401 |
fix: harden batch skill install — 7 review items + OpenAPI snapshot
- Race condition: wrap create_prompt_template in try/except, append to skipped on conflict instead of crashing - HTTP timeout: per-request timeout (10s+5s connect) instead of shared 15s pool; parallelize SKILL.md and resource fetches with semaphore (5 concurrent) - Branch detection: return branch_explicit from _parse_github_url(), eliminate duplicated regex matching and type: ignore comments - Content-length: check len(resp.content) after fetch instead of unreliable content-length header; add size check in batch path - Rate limits: _check_rate_limit() inspects x-ratelimit-remaining, raises actionable error on 403, warns when remaining < 10 - Root resources: fix _find_resource_files skipping root-level resources like scripts/foo.sh for root SKILL.md - resource_count: pass accurate count in update and install responses - Regenerate openapi-console.json with new resource endpoints |
||
|
Patrick Buckley
|
c28bfc1e58 |
feat: skill discovery — search and install skills from external sources (#111)
* feat: skill discovery — search and install skills from external sources Add discovery UI and API for finding and installing skills from skills.sh registries and GitHub repositories with one-click install, SKILL.md frontmatter parsing, and security scan integration. Core modules: - skill_parser.py: ParsedSkill dataclass, parse_skill_md() with YAML frontmatter support (Anthropic + Hermes tag formats), name validation - skill_sources.py: SkillsShClient (async search + resolve), fetch_skill_from_github (SKILL.md + bundled resource fetching with 256KB cap, text extension filter, GitHub API tree traversal) API: - GET /v1/api/admin/skills/discover — search with installed annotation and scan_status for installed skills - POST /v1/api/admin/skills/install — fetch, parse, duplicate check, create with origin="source" readonly=true, store resources, audit Also fixes pre-existing bug where _skill_to_response omitted scan_status, scan_report, scan_version fields — scan tier badges in the installed skills table were silently empty despite data existing in storage. Admin UI: pill toggle (Installed/Discover), discovery cards with scan tier badges, GitHub import modal with proper focus trap/Escape/backdrop, scoped selectors preventing MCP↔Skills cross-tab state corruption. SDK: discover_skills() + install_skill() on Python (async+sync) and TypeScript console clients. 48 new tests across 3 test files. All 2632 tests pass. * fix: address copilot review — 404 vs 502, O(n) lookups, branch fallback - SkillNotFoundError subclass: install returns 404 when SKILL.md is missing, 502 only for connectivity/upstream errors - get_skill_by_source_url() + list_installed_skill_urls(): indexed storage lookups replace O(n) full-table scans with content blobs - Default branch fallback: tries main then master when URL doesn't specify a branch - Path normalization: strip trailing slash once, remove redundant candidate - SDK install_skill() returns typed SkillInfo with response_model - Tree size guard: skip resource tree if response >2MB |
||
|
Patrick Buckley
|
e71ea38953 |
feat: output guard data pipeline — persist assessments, SSE events, a… (#110)
* feat: output guard data pipeline — persist assessments, SSE events, admin UI
Complete the output guard pipeline: persist assessments for v2 calibration,
surface warnings in every UI layer, and add scan badges to admin skills tab.
Storage: migration 022 adds output_assessments table (flags, risk_level,
annotations, output_length, redacted — raw output never stored) and
scan_version column on prompt_templates. Three new protocol methods with
SQLite + PostgreSQL implementations.
Server/CLI: on_output_warning now persists assessments fire-and-forget.
CLI shows flags, annotations, and redaction notice. Session emits on_info
warning when high/critical scan_status skill is loaded.
MQ: OutputWarningEvent dataclass + bridge SSE forwarding.
Web UI: output_warning SSE handler with inline warning rendering
(role="alert" for accessibility), semantic risk colors.
Console admin: scan badges on skills list (dedicated scope-scan-* CSS with
green/yellow/red risk vocabulary), scan report breakdown in edit modal with
4-axis scores, POST /admin/skills/{id}/rescan endpoint, GET
/admin/output-assessments endpoint with date-filtered pagination.
Security fixes: ReDoS in connection string regex ([^@\s]+ → [^:@\s]+),
negative limit bypass in all admin endpoints (max(1, ...)), to_dict()
excludes sanitized output by default.
False-positive fixes: credentials pattern anchored to path context,
env secret key check restricted to key portion only.
* fix: address PR #110 review — list redaction, test fixture, OpenAPI snapshot
Per-part redaction: evaluate each text part independently in structured
output instead of joining all parts and replacing only the first one.
Fix test annotations default from "{}" to "[]" matching schema.
Regenerate TypeScript OpenAPI snapshots for new admin endpoints.
|
||
|
Patrick Buckley
|
75eda9a096 |
feat: unified skills system — merge prompt templates + workstream tem… (#106)
* feat: unified skills system — merge prompt templates + workstream templates Evolves prompt_templates into a first-class skills entity and merges workstream templates into the same model, collapsing two concepts into one. Migration 021: 21 new columns on prompt_templates (skills metadata, security scan fields, session config from WS templates), skill_resources table for bundled files, skill_versions table for auto-snapshot version history. Data migration converts existing WS templates into skills with name collision handling, migrates version history, renames workstreams and scheduled_tasks columns, cleans orphaned permissions, drops old tables. Key changes: - All public interfaces renamed: templates → skills (API, CLI, SDK, UI) - Session config (model, temperature, token_budget, auto_approve, etc.) now lives on the skill and is applied at workstream creation - /skill slash command, set_skill() API, --skill CLI flag - BM25 skill search via SkillSearchManager for activation="search" skills - Admin UI: Skills tab with collapsible Session Config section, description subtitles, activation/origin/MCP badges, pagination - Shared validation helper (_parse_skill_session_config) for DRY CRUD - Version history with auto-snapshot on every edit + API endpoint - Cascade delete (resources + versions) on skill removal - Security: range validation, activation allowlist, fail-closed enabled check, duplicate name 409, readonly guard, JSON validation - 77 new tests across storage, runtime, search, API integration, and migration behavior verification (2521 total) * fix: address Copilot review + rename admin.templates → admin.skills - Skip skill lookup when resume_ws is set (avoids spurious 400) - Fix _applied_skill_version mismatch (1 in both workstreams table and session) - Remove stale template field from MQ protocol diagram - Rename admin.templates permission to admin.skills everywhere (runtime, frontend, tests, docs) with migration step for persisted role data - Fix stale /api/templates references in docs and diagrams - Update docstrings/comments for skills terminology * fix: address Copilot round 2 — skill version lineage + stale doc refs - Compute actual skill version from skill_versions count (not hardcoded 1) - Use same version in both workstreams table and session metadata - Fix response payload example: "templates" → "skills" key - Fix "Each template summary" → "Each skill summary" |
||
|
Patrick Buckley
|
80e1924d7f |
feat: enable prompt caching for Anthropic and OpenAI providers (#104)
* feat: enable prompt caching for Anthropic and OpenAI providers
Activate automatic prompt caching on both LLM providers to reduce input
token costs on multi-turn conversations. Anthropic gets cache_control:
ephemeral (90% savings on cache hits), OpenAI GPT-5.x gets 24h extended
cache retention (free). Cache metrics flow end-to-end through the entire
data pipeline: provider → session → server SSE → MQ protocol → storage →
Prometheus metrics → admin Usage tab.
- AnthropicProvider: top-level cache_control on all requests, extract
cache_creation_input_tokens and cache_read_input_tokens from streaming
and non-streaming responses
- OpenAIProvider: prompt_cache_retention=24h for GPT-5.x, extract
cached_tokens from usage.prompt_tokens_details
- UsageInfo: new cache_creation_tokens and cache_read_tokens fields
- Migration 020: add cache columns to usage_events table
- Storage: record_usage_event and query_usage updated (sqlite + pg)
- Metrics: turnstone_tokens_total{type="cache_creation|cache_read"}
- Server: on_status passes cache tokens to SSE, storage, and metrics
- MQ: StatusEvent carries cache fields through bridge
- SDKs: Python and TypeScript StatusEvent types updated
- OpenAPI: UsageBreakdownItem schema includes cache fields
- Console UI: Usage tab shows cache write/read as secondary readouts
with visual separator, dimmed when zero
- 16 new tests, docs and 3 diagrams updated
* fix: address Copilot review feedback
- Fix MQ protocol diagram clipping by switching to vertical package
layout (inbound on top, outbound below) with package aliases
- Regenerate OpenAPI snapshots to include cache_creation_tokens and
cache_read_tokens on UsageBreakdownItem
- Replace fragile MagicMock(spec=[]) + del pattern with
types.SimpleNamespace in cache metrics missing-attributes test
|
||
|
Patrick Buckley
|
20df7b3034 |
feat: OIDC SSO authentication with PKCE, auto-provisioning, and role … (#71)
* feat: OIDC SSO authentication with PKCE, auto-provisioning, and role mapping Add OpenID Connect as a fourth authentication method, enabling single sign-on via any OIDC provider (Okta, Azure AD, Google, Keycloak). Opt-in via env vars (TURNSTONE_OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET). Security: - Authorization Code Flow with PKCE (S256) - State/nonce parameters with database-backed pending store (multi-node safe) - JWKS signature validation with async fetch + key rotation retry - Algorithm allowlist from JWKS key (not token header) prevents confusion - Identity matching exclusively by (issuer, sub) — prevents account takeover - password_enabled=false enforced server-side, not just UI - Rate limiting on both authorize and callback endpoints - OIDC users get "!oidc" password sentinel (bcrypt rejects naturally) - ID token validated for iss, aud, exp, nonce Features: - Auto-provisioning with username deduplication on first login - Claim-based role mapping with IdP demotion propagation (revokes stale roles) - "Continue with [Provider]" SSO button on login page - OIDC-only mode hides password form - Setup wizard required before OIDC login (admin bootstrap) Storage: migration 018 (oidc_identities + oidc_pending_states tables), 8 new protocol methods on both SQLite and PostgreSQL backends. 66 new tests (2273 total). * fix: address PR #71 review feedback (18 items) Bugs fixed: - OIDC success redirect now fetches permissions via new /auth/whoami endpoint before completing login (fixes permission-gating in UI) - Remove double decodeURIComponent on oidc_error (URLSearchParams already decodes; extra call throws on stray %) - Authorize rate limiter returns redirect instead of JSON 429 (endpoint reached via browser navigation, not fetch) - Lazy JWKS fetch in callback when startup discovery failed (IdP recovery without restart) - Startup exception handlers now log with exc_info=True - PostgreSQL pop_oidc_pending_state uses DELETE...RETURNING for true atomicity (eliminates TOCTOU) Behavior: - New OIDC users without role mapping get builtin-viewer by default (assigned_by="oidc-default", not revoked by role sync) Documentation fixes: - Role mapping: sync semantics (add + revoke stale), not "additive only" - PASSWORD_ENABLED=false blocks ALL password logins including admin - Algorithm: asymmetric allowlist, not per-key derivation - PlantUML diagram updated for role revocation API spec fixes: - Removed error_codes=[302] from callback (302 is success redirect) - Added /auth/whoami to both server + console specs - Regenerated TypeScript SDK OpenAPI snapshots (23 + 51 paths) * fix: address PR #71 round 2 review feedback (10 items) Rate limiting: - Authorize endpoint now calls record() after check() so the rate limiter actually counts attempts (was a no-op before) OIDC resilience: - Split startup try/except: discovery failure disables OIDC, JWKS prefetch failure leaves OIDC enabled for lazy retry on first login - JWKS unavailable message changed to "temporarily unavailable" (was misleadingly "not configured") - create_oidc_pending_state raises on collision instead of OR IGNORE (prevents silent insert drop on state collision) - SQLite pop_oidc_pending_state uses BEGIN IMMEDIATE for write lock (eliminates TOCTOU race) Frontend: - OIDC error display deferred 300ms so showLogin()'s async status fetch doesn't clear it via _switchMode → _clearError API spec: - OIDC authorize/callback endpoints now declare response_code=302 - Added AuthWhoamiResponse Pydantic model for /auth/whoami - Regenerated TypeScript SDK OpenAPI snapshots Documentation: - Diagram: JWKS "cached at startup, refreshed on-demand" (was "hourly") - Added TODO(tech-debt) comments on Host header redirect_uri sites |
||
|
Patrick Buckley
|
376da3d084 |
feat: prompt template tech debt — tests, read-only endpoints, double-… (#67)
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal Close test coverage gaps for prompt templates: - Resume with deleted template: verifies graceful degradation (template_content=None, warning logged) - Threading safety: concurrent set_template/init_system_messages with no race conditions - Factory passthrough: template kwarg propagation through WorkstreamManager.create() Add read-only template listing endpoints (read scope, no content exposed): - GET /v1/api/templates — prompt template summaries (name, category, is_default, origin) - GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model) - Available on both server and console; Python + TypeScript SDK methods added - Console creation modal switched from admin endpoint to read-scope endpoint Eliminate double-load inefficiency in workstream creation: - Template validation moved before mgr.create() (no create-then-rollback on invalid template) - template kwarg plumbed through WorkstreamManager.create() and session factory - _SessionFactory Protocol added for proper mypy typing Add workstream creation modal to server web UI: - Name, model, template dropdown, ws_template/profile dropdown - Instrument panel aesthetic: gradient top border, blur backdrop, amber accent - Focus trap, Escape/Enter keyboard handling, loading state, error display - WCAG AA contrast compliance, reduced-motion support * fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates() to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint. Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types. Regenerate openapi-server.json and openapi-console.json snapshots. Addresses Copilot review feedback on PR #67. * fix: skip template pre-validation when resuming a workstream When resume_ws is set, the request's template field is irrelevant — resume() restores the template from workstream_config. Pre-validating a stale template name would incorrectly return 400 before the resume even runs. Addresses Copilot review feedback on PR #67. |
||
|
Patrick Buckley
|
19abc0cc65 |
feat: admin MCP Servers tab — database-backed MCP server management w… (#62)
* feat: admin MCP Servers tab — database-backed MCP server management with live status Add MCP Servers admin tab (14th tab, System group) for managing MCP server definitions via the database instead of static JSON config files. Storage: `mcp_servers` table (migration 016), 6 CRUD methods on both SQLite and PostgreSQL backends, `MCP_SERVER_MUTABLE` field allowlist. Config priority chain: DB rows (if any enabled) → CLI `--mcp-config` → `mcp.config_path` setting → none. Nodes auto-load from DB on startup via `load_mcp_config(storage=)`. Hot-reload: `reconcile_sync(storage)` diffs running servers against DB — adds missing, removes stale, reconnects changed. `_db_managed` set tracks DB-sourced servers so config-file servers (MCP_CONFIG env) are never removed by reconcile. Per-server `AsyncExitStack` for clean teardown. Reload pattern: console writes to DB then signals nodes via `POST /_internal/mcp-reload` (update by reference, no config payload). Console admin API: 7 endpoints under `/v1/api/admin/mcp-servers` (CRUD + reload + import), `admin.mcp` permission, secret masking (env/headers replaced with *** unless ?reveal=true), audit log sanitization. Unified view: tab merges DB-managed servers with config-sourced servers detected on nodes. Config servers shown as read-only rows with "config" badge — no edit/delete. Admin UI: 7-column grid with magenta status dots, transport badges, single-column create/edit modal, paste-based JSON import (mcpServers format), detail modal with per-node status. Mobile 3-column collapse, reduced-motion support, backdrop-click dismiss, focus trapping. SDKs: 7 methods on Python (async+sync) and TypeScript SDKs. Also fixes: Settings tab permission gate (admin.users → admin.settings), _ALL_PERMISSIONS list in governance.js (5 missing permissions added), _internal/mcp-reload added to APPROVE_PATHS. Docs: architecture.md (14 tabs), api-reference.md (7 endpoints), 20-mcp-architecture.puml updated with admin-driven lifecycle. 66 new tests (2232 total). * fix: address Copilot review feedback on MCP admin PR - Docs: fix "merges both sources" → "first-match-wins priority" (architecture.md) - Validation: require command for stdio, url for streamable-http transport - Validation: check args/headers/env types in import handler before storing - Schema: add transport/command/url to McpServerStatus, source to McpServerDetail - Thread safety: move all remove_server_sync mutations onto MCP event loop thread - Regenerate OpenAPI JSON snapshots for TypeScript SDK |
||
|
Patrick Buckley
|
101afd84da |
feat: database-backed settings (ConfigStore) with admin API (#59)
* feat: database-backed settings (ConfigStore) with admin API
Replace config.toml for non-bootstrap settings on the server with a
database-backed ConfigStore. ~40 settings across model, session,
tools, server, mcp, ratelimit, health, judge, and memory sections are
now managed via the admin Settings API. CLI flags for these settings
removed from the server entry point (CLI standalone tool unchanged).
Storage: system_settings table (migration 015) with composite PK
(key, node_id) for per-node overrides. ON CONFLICT upsert in both
SQLite and PostgreSQL. admin.settings permission granted to
builtin-admin role.
Settings registry (settings_registry.py): code-defined catalog of
all known settings with types, defaults, validation, descriptions.
Registry defaults aligned with previous argparse defaults.
ConfigStore (config_store.py): thread-safe in-memory cache loaded
from storage on init. Lock-free reads via dict snapshot swap.
reload() for hot-reload via internal endpoint.
Secret settings (judge.api_key) blocked from write via admin API
(403) — must be configured via config.toml or env vars.
warn_migrated_settings() logs warnings for config.toml keys that
overlap with ConfigStore-managed settings.
Console admin API: GET /v1/api/admin/settings (list with effective
values), GET .../schema (registry catalog), PUT .../{key} (update),
DELETE .../{key} (reset to default). Audit trail on mutations.
MQ: ConfigChangeEvent for cross-node cache invalidation (emission
from console deferred to bridge integration).
Python + TypeScript SDK methods. 63 new tests. Feature docs at
docs/settings.md, PlantUML diagram 24-settings-architecture.
* fix: address PR review — config-reload scope, registry defaults, doc alignment
- config-reload endpoint requires approve scope (was write)
- config-reload handler is sync def (avoids blocking event loop)
- reasoning_effort passes empty string through (removes `or "medium"`)
- ratelimit.requests_per_second changed to float (matches RateLimiter)
- session.retention_days allows 0 (disable pruning)
- ratelimit.trusted_proxies added to registry + wired in server
- admin_list_settings filters to global settings only (no node_id ambiguity)
- admin_update_setting validates "value" key presence (400 if missing)
- Console config change fans out reload to nodes directly (no MQ dep)
- Docs aligned with actual API response shapes and masking ("***")
|
||
|
Patrick Buckley
|
67f43a7ee0 |
feat: [memory] REST API endpoints + SDK methods + docs (#56)
* feat: [memory] REST API endpoints + SDK methods + docs
Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope
Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit
Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.
Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.
TypeScript SDK: matching methods + types on both clients.
Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.
Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.
Also fixes stray `total: int` on CreateChannelUserRequest.
* fix: [memory] address PR review — cross-user scope, schema types, snapshots
Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity. Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.
Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.
Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.
6 new security tests for user-scope access control.
|
||
|
Patrick Buckley
|
02d9c5c797 |
feat: workstream templates — behavioral profiles for workstream creation (#49)
* feat: workstream templates — behavioral profiles for workstream creation Workstream templates define the complete configuration for workstream creation: system prompt, model, auto-approve policy, per-tool auto-approve, temperature, reasoning effort, max tokens, agent max turns, token budget, and completion notifications. Applied once at creation time (snapshot, not live binding). Auto-versioning captures pre-update state on every edit. Schema & storage: - workstream_templates + workstream_template_versions tables (migration 011) - ws_template_id/ws_template_version columns on workstreams table - ws_template column on scheduled_tasks table - Full CRUD + versioning on SQLite and PostgreSQL backends - prompt_template_hash (SHA-256) for drift detection Runtime: - Template resolution before mgr.create() for model override - Post-creation settings application (prompt, temperature, approval, budget) - Token budget enforcement in session.send() — 80% warning, approval gate at 100% via __budget_override__ synthetic tool - WebUI.auto_approve_tools server-side per-tool auto-approve - Prompt template drift detection (hash comparison, log warning on mismatch) Integration: - ws_template field on CreateWorkstreamMessage, bridge, channel router, scheduler dispatch, MQ client - Console admin "WS Templates" tab (11th) with CRUD, version history modal - Profile dropdown on workstream creation modal - WS template dropdown on scheduler create/edit modals - Prompt template name validation on ws_template create/update - 7 console admin API endpoints + read-only summary endpoint - Full OpenAPI spec entries in console_spec.py - Python SDK (sync + async) and TypeScript SDK methods - Pydantic schemas for all request/response models Docs & diagrams: - New 21-ws-template-architecture.puml sequence diagram - Updated governance, storage, MQ protocol diagrams + PNGs - Updated architecture.md, governance.md, api-reference.md, console.md, sdk.md 48 new tests (1788 total). mypy clean. ruff clean. * fix: address PR #49 review feedback - auto_approve_tools uses approval_label (not just func_name) for consistency with tool policy evaluation - inline system_prompt from ws_template persisted as _ws_template_system_prompt in workstream_config, restored on resume (previously lost because _template_content wasn't persisted) - budget gate (__budget_override__) no longer bypassed by blanket auto_approve — requires explicit approval or tool policy allow - diagram 21 field list corrected (removed tool_search/threshold, added prompt_template_hash/notify_on_complete) * fix: address PR #49 review feedback (round 2) - Grant admin.ws_templates permission in migration 011 (tab was hidden) - Center WS template modals and fix radio button alignment - Skip template validation when ws_template overrides prompt - Guard against empty version snapshots on no-op updates - Replace setTimeout race with Promise chain in schedule ws_template select - Validate numeric fields in admin create/update handlers (400 not 500) - Add ws_template to TypeScript OpenAPI specs - Use typed Pydantic response models in SDK ws_template methods |
||
|
Patrick Buckley
|
2f7f70825b |
feat: wire prompt templates into session startup with full creation-p… (#47)
* feat: wire prompt templates into session startup with full creation-path support
Prompt templates (prompt_templates table) now have runtime effect:
- is_default=true templates auto-apply as system message content,
concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
field on POST /v1/api/workstreams/new, console creation modal dropdown,
scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession
Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion
Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.
Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.
Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.
Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.
* fix: address PR #47 review feedback
- Defer template validation until after resume_ws — a bad template name
no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
reject unknown template names with 400 instead of allowing schedules
that would silently fail at dispatch time
|
||
|
Patrick Buckley
|
e057c364b8 |
Fix console proxy regressions and add workstream task field (#21) (#21)
* Fix console proxy regressions and add workstream task field (#21) Bug fixes: - Fix collector polling unversioned /api/dashboard (404 after API versioning PR) — nodes showed red/unreachable, no workstreams - Fix SSE proxy dropping all data events — upstream sends \r\n line endings but proxy split on \n\n only; normalize before parsing - Fix workstream state stuck on idle — on_state_change() only broadcasted via SSE but never updated ws.state on the Workstream object; dashboard polling now sees correct attention/running states - Fix deep-link switchTab early return — when ?ws_id matched the only workstream, switchTab bailed (wsId === currentWsId) before establishing SSE connection; inline init instead of delegating - Fix console banner covering dashboard overlay — inject <style> offsetting .dashboard-overlay below the 32px banner Enhancements: - Add turnstone branding to console proxy banner (turnstone │ Console │ node-id) - Add initial_message field to CreateWorkstreamMessage protocol and console "New Workstream" modal (Task textarea, sent as first message) - Refactor SSE proxy to use shared httpx client with 30s read timeout instead of per-request client creation - Increase approval timeout default from 300s to 3600s (1 hour) Updated: Python SDK, TypeScript SDK, OpenAPI specs, MQ client, API schemas, MQ protocol diagram, SDK docs. * Address PR #21 review feedback (4 items) - Log unknown state strings in on_state_change instead of silently swallowing; remove unnecessary KeyError catch - Wrap initial_message POST in _handle_create_ws with error handling so workstream creation success isn't masked by send failure - Strip all \r from SSE chunks instead of replacing \r\n, fixing chunk-boundary split edge case - Add tests for initial_message wiring in directed and pool targeting * Refactor SSE proxy to use httpx-sse aconnect_sse Replace manual SSE chunk buffering/parsing with httpx_sse.aconnect_sse() which handles line endings, event types, and all SSE spec edge cases. Eliminates the \r\n chunk-boundary bug class entirely. Event types are now always forwarded (sse.event defaults to "message" per spec). |
||
|
Patrick Buckley
|
5ee539c983 |
Add Python and TypeScript client SDKs for server and console APIs (#19)
* Add Python and TypeScript client SDKs for server and console APIs Python SDK (turnstone/sdk/) with sync + async clients for both server and console APIs. Returns Pydantic models directly, streams SSE events as typed dataclasses. 27 event types with registry-based deserialization. High-level send_and_wait() for request-response patterns. TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses fetch + ReadableStream for SSE parsing. Discriminated union event types with type guards. Same API surface as Python SDK. 63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at docs/sdk.md with SDK architecture diagram. * Address PR #19 review feedback + fix lint - Fix consume_task leak in send_and_wait when send() raises (try/finally) - Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout - Add signal param to TS streamSSE for cancellation support - Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF - Fix generate-types.py sys.path (parents[3] not parents[2]) - Document token ignored when httpx_client provided - Document TS timeout units as milliseconds - Fix stale docstring in test_sdk_sse.py - Fix import sorting (ruff I001) |