mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
main
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
cc84f9d176 | fix(memory): harden project scope authorization and consistency | ||
|
Patrick Buckley
|
480a1426b3 |
Fail-closed history-commit handoff (#1005)
* fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964 |
||
|
Patrick Buckley
|
7a06f5e8bc |
refactor(session): make ModelLane the provider boundary (#979) (#989)
* refactor(session): make ModelLane the provider boundary (#979) ## Summary This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot. - Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding. - Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call. - Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references. - Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results. - Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts. ## Validation - SQLite suite: 11,188 passed, 9 skipped, 10 deselected - PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected - Live backend: 3 passed - SSE recovery: 6 passed; browser recovery harness passed all scenarios - Ruff: clean; 595 files correctly formatted - mypy: 243 source files clean - TypeScript: typecheck/build and 35 tests passed - OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte - `git diff --check` and Git LFS integrity clean Closes #979. * fix(deps): update nanoid for GHSA-2v37-7h3g-55p8 Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation. Validation: - npm ci - npm audit --audit-level=moderate: 0 vulnerabilities - TypeScript typecheck and build - TypeScript tests: 35 passed * fix(test): assert canonical model registry URLs Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation. Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy. |
||
|
Patrick Buckley
|
7c34d859b4 |
test(sdk): update stale send() vitest expectations to the path-keyed contract
Three server.test.ts / server-attachments.test.ts cases still asserted
the pre-verb-lift send shape (POST /v1/api/send with ws_id in the body).
The server route has been POST /v1/api/workstreams/{ws_id}/send (ws_id
in the path, {message} body) since that lift, and the SDK send() was
updated with it — only these expectations rotted. They fail on main
too; unrelated to approvals, folded in here to leave the TS SDK suite
green. Test-only, no SDK source change.
|
||
|
Patrick Buckley
|
110d44b07e |
refactor(tools): remove man, math, and plan_agent built-in tools
`man` and `math` duplicated capabilities already reachable through `bash`; `plan_agent` is better expressed as a `task_agent` running a planning skill, and carried a large amount of special-case machinery (plan-review gate, refinement loop, per-kind model routing). Removing all three shrinks the tool surface and cuts per-call token cost. Also removed, as dead-once-the-tools-are-gone: - the `math` sandbox executor (`turnstone.core.sandbox`) and its `[sandbox]` extra; the eval analyst now runs bash-only - the read-only `AGENT_TOOLS` sub-agent tool set and the `agent` tool-metadata key (`task_agent`/`TASK_AGENT_TOOLS` retained) - the plan-review protocol end to end: the `on_plan_review` UI hook, `resolve_plan`, `POST /v1/api/plan` + `POST /v1/api/route/plan`, the `plan_review`/`plan_resolved` SSE events, and their Python SDK / TypeScript SDK / OpenAPI / frontend / Discord+Slack bindings - the `model.plan_alias` / `model.plan_effort` settings and the registry `plan_model` / `plan_effort` routing fields TOOLS 31->28, TASK_AGENT_TOOLS 13->11; COORDINATOR_TOOLS unchanged. BREAKING CHANGE: removes the `man`, `math`, `plan_agent` tools, the plan-review SSE/HTTP/SDK surface, and the plan_* model-routing settings from the experimental 1.6 line. |
||
|
Patrick Buckley
|
edf52016ac |
refactor(core): lift list + saved verb bodies across both kinds (Stage 2 verb lift)
New ``make_list_handler(cfg)`` and ``make_saved_handler(cfg)``
factories in ``turnstone/core/session_routes.py`` replace four
pre-lift bodies (interactive ``list_workstreams`` +
``list_saved_workstreams``; coord ``coordinator_list`` +
``coordinator_saved``). Same factory + capability-flag pattern as
the merged cancel / open / events / create lifts.
Four new ``SessionEndpointConfig`` fields:
- ``list_resolve_titles: ListResolveTitles | None`` — bulk lookup
``(ws_ids) -> {ws_id: title-or-None}``. Interactive wires
``get_workstream_display_names`` (new bulk helper added on the
storage layer + memory.py); the lifted body resolves every active
row in ONE ``SELECT ... WHERE ws_id IN (...)`` instead of the
pre-lift N+1 (one SELECT per row).
- ``list_kind: WorkstreamKind | None`` — explicit kind classifier
for the saved-list storage filter. Replaces the initial draft's
``audit_action_prefix == "coordinator"`` string compare which
would have silently leaked INTERACTIVE rows for any future kind
whose audit prefix didn't match. Required when a kind mounts
list/saved; misconfig surfaces as a 500 with a clear log line.
- ``saved_state_filter: str | None`` — coord wires ``"closed"``;
interactive wires ``None``.
- ``saved_loaded_lookup: SavedLoadedLookup | None`` — coord-only
defence-in-depth filter that excludes ws_ids in the warm pool.
Behaviour changes (all observable in CHANGELOG):
- **Active-list row shape converges on always-include** ``{ws_id,
name, state, kind, parent_ws_id, user_id}``. Interactive renames
``id`` → ``ws_id``; both kinds populate every field (coord adds
kind + parent_ws_id; interactive adds user_id).
- **Top-level response key converges on ``"workstreams"``** on
both endpoints. Coord ``coordinators`` key removed — coord is a
1.5.0aN-only surface (never shipped stable) so the convergence
has no compat shim; SDK / frontend consumers swap once.
- **Storage + manager-lock work moved off the event loop on
interactive**. ``list_workstreams_with_history`` runs through
``asyncio.to_thread`` on both kinds (matches coord's pre-existing
perf-2 pattern from the saved-coordinators review); ``mgr.list_all``
+ per-row work also offloaded.
- **N+1 storage round-trips on /v1/api/workstreams eliminated**.
Pre-lift interactive resolved the alias for every active row in a
separate SELECT (up to 50 round-trips per dashboard refresh on a
saturated node). Lifted body issues one bulk SELECT.
Pydantic schemas: ``WorkstreamInfo.id`` renamed → ``ws_id``,
``WorkstreamInfo.user_id`` field added. ``CoordinatorInfo`` and
``CoordinatorListResponse`` removed (folded into the unified
``WorkstreamInfo`` / ``ListWorkstreamsResponse``). OpenAPI spec
snapshots regenerated. TS SDK types updated (``WorkstreamInfo``
interface gains ws_id + the always-include fields); TS test
mock + assertion updated to match.
``GET /v1/api/dashboard`` is intentionally NOT in this PR's scope
and still returns rows keyed on ``id``. Tracked as a separate
cleanup PR (tombstone-note added at the dashboard handler).
/review pipeline run; the four Major findings + one Minor + six
nits all addressed in the same commit:
- M1: TS SDK ``WorkstreamInfo`` interface stale (id: string) →
renamed + fields added.
- M2: TS SDK test masked the type-mismatch with stale mock → updated.
- M3: N+1 alias resolution on active list → bulk
``get_workstream_display_names`` helper + ``list_resolve_titles``
bulk cfg hook.
- M4: Missing interactive parity regression test for unified row
shape → mirror of coord's added in test_server_authz.py.
- Mi1: ``audit_action_prefix`` string-compare deriving kind →
explicit ``cfg.list_kind: WorkstreamKind`` field.
- Six nits: redundant inner asyncio import, forward-ref quotes on
Awaitable, duplicated frontend comments, dashboard ``id`` field
has no tombstone-note, empty-coord_mgr short-circuit on
``saved_loaded_lookup``.
4512 tests passing; ruff + mypy clean.
|
||
|
Patrick Buckley
|
6cbd3eb2c1 |
feat: workstream attachments at creation time + SDK + UI parity (#362)
* feat: workstream attachments at creation time + SDK + UI parity Closes the two big deferred items from PR #356: attaching files as part of the initial workstream-creation request, and full SDK coverage of the attachment surface. Server: POST /v1/api/workstreams/new now accepts multipart/form-data (meta JSON + 0..N file parts). Files are validated and saved as pending under the new ws; when initial_message is also set the create handler reserves them onto that turn before the dispatch worker fires, mirroring the /v1/api/send pattern. Validation failure rolls back the workstream via delete_workstream so we don't leak orphan rows or emit a phantom ws_created/ws_closed pair on SSE. JSON path is unchanged. Console routing: route_create accepts multipart with ?ws_id=<hex> as a query parameter (the console hashes the id before the body lands). Added /v1/api/route/workstreams/{ws_id}/attachments POST/GET/DELETE + .../{attachment_id}/content GET proxies that forward raw bytes and preserve upstream headers (Content-Disposition, X-Content-Type-Options, CSP sandbox). Python + TypeScript SDKs: AttachmentUpload type, upload_attachment, list_attachments, get_attachment_content, delete_attachment, and send(attachment_ids=...). create_workstream(attachments=...) sends multipart and pre-generates a ws_id client-side so cluster routing works. SDKs reject attachments+target_node combinations since the multipart route doesn't honor target_node. Web UI: dashboard composer refactored to a single unified create flow. Replaced the inconsistent split (Enter created+sent raw, "New Chat" opened a modal) with one rich composer carrying a textarea, paperclip + chip strip, drag-drop, paste-image, and a collapsible Options panel for model/judge_model/skill. Submit button dynamically labels Create vs Send. New-workstream modal also gained the same paperclip + chip strip + first-message field for the tab-bar + entry point. Tests: 30 new tests across server multipart create, console route multipart + attachment proxies, Python + TS SDK attachment surfaces, plus regressions for the three review-flagged bugs (Content-Type boundary preservation, attachments+target_node rejection, no phantom ws_created on validation failure). * fix: address Copilot review feedback on PR #362 - web_helpers: docstring now matches behaviour — read_multipart_create_or_400 does enforce the optional max_per_file_bytes cap as defense-in-depth. - app.js: drop the duplicated _formatAttachSize definition (one already exists earlier for pane chips); add a shared _isAttachmentAllowed helper that mirrors the server's classifier (png/jpeg/gif/webp images, text/* MIMEs, allowlisted application/* MIMEs, known text extensions) and call it from both _newWsAddFiles and _addDashboardFiles so unsupported files fail fast client-side instead of after a server roundtrip. - app.js: dashboardSubmit catch now suppresses the redundant error toast on authFetch's "auth" Error and falls back to a generic message when err.message is undefined, instead of rendering "Connection error: undefined". - SendResponse (Pydantic + TS): document and expose attached_ids, dropped_attachment_ids, priority, and msg_id so attachment-aware SDK callers can detect partial reservations and dequeue queued messages. - test_server_attachments_on_create: drop the dual `import turnstone.server` + `from turnstone.server import` style — use monkeypatch.setattr by dotted path for module-level mutation and `from … import …` for the helpers, keeping a single import style. |
||
|
Patrick Buckley
|
87a9af1075 |
fix: broadcast plan_resolved SSE so other clients dismiss in sync
Previously, resolving a plan on one client (e.g. phone) cleared the server's pending state and unblocked the worker, but emitted no event to other connected clients. Their plan-approval modal stayed stuck. resolve_plan() now enqueues a plan_resolved frame (mirroring the approval_resolved pattern in resolve_approval) before clearing _pending_plan_review, so a reconnecting client cannot receive both the replayed plan_review and the live plan_resolved. Skips the frame on the cancel-with-no-plan path. Client adds a plan_resolved handler that dismisses the modal without re-firing /v1/api/plan, restores keyboard context (skipped on touch to avoid soft-keyboard pop on mobile), labels the inline plan summary "(synced)" so remote dismissal is unambiguous, announces via the existing aria-live #toast for screen-reader parity, and falls back to an info message if plan_resolved races ahead of plan_review. Adds PlanResolvedEvent to the Python and TypeScript SDKs with deserialization and type-guard tests. |
||
|
Patrick Buckley
|
28cb3a5c51 |
fix: approval timeout UI state and content flush before tool calls (#42)
* fix: approval timeout UI state and content flush before tool calls Two bug fixes: 1. Approval timeout now shows denied state in UI — resolve_approval() emits an approval_resolved SSE event so the browser transitions from pending to denied (red border + badge). Also fixes the cancel- during-approval path. Frontend resolveInlineApproval() gains a skipPost parameter to avoid redundant POST when server-initiated. ApprovalResolvedEvent added to Python and TypeScript SDKs. 2. Content streaming flushes pending buffer before tool call deltas — _stream_response() held up to 13 trailing chars in the pending buffer (for <think> tag detection) when transitioning to tool calls. Now flushed eagerly when tool_call_deltas arrive, before clearing in_think so reasoning text is correctly categorized. * fix: address Copilot review feedback on PR #42 Patch _execute_tools in stream flush test to prevent real bash execution, simplify confusing nested comprehension, and update resolve_approval() docstring to reflect cancel/timeout call paths. |
||
|
Patrick Buckley
|
5ee539c983 |
Add Python and TypeScript client SDKs for server and console APIs (#19)
* Add Python and TypeScript client SDKs for server and console APIs Python SDK (turnstone/sdk/) with sync + async clients for both server and console APIs. Returns Pydantic models directly, streams SSE events as typed dataclasses. 27 event types with registry-based deserialization. High-level send_and_wait() for request-response patterns. TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses fetch + ReadableStream for SSE parsing. Discriminated union event types with type guards. Same API surface as Python SDK. 63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at docs/sdk.md with SDK architecture diagram. * Address PR #19 review feedback + fix lint - Fix consume_task leak in send_and_wait when send() raises (try/finally) - Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout - Add signal param to TS streamSSE for cancellation support - Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF - Fix generate-types.py sys.path (parents[3] not parents[2]) - Document token ignored when httpx_client provided - Document TS timeout units as milliseconds - Fix stale docstring in test_sdk_sse.py - Fix import sorting (ruff I001) |