mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
2e95f2ac73bcdbdd4558ffaafa263d80cd87c953
63 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
2e95f2ac73 |
test: add scope coverage for internal MCP/config reload endpoints (#75)
* test: add scope coverage for internal MCP/config reload endpoints Verify required_scope() returns "approve" for _internal endpoints across all access patterns (bare, /v1/-prefixed, console proxy with and without /v1/), plus a GET negative test confirming only POST is elevated. Closes the "internal endpoints accept read scope" item in PROGRESS.md — the endpoints were already in APPROVE_PATHS. * test: add config-reload v1/proxy scope tests per review feedback Add /v1/-prefixed and console proxy variants for config-reload to match the mcp-reload coverage, as flagged by Copilot review. |
||
|
Patrick Buckley
|
20df7b3034 |
feat: OIDC SSO authentication with PKCE, auto-provisioning, and role … (#71)
* feat: OIDC SSO authentication with PKCE, auto-provisioning, and role mapping Add OpenID Connect as a fourth authentication method, enabling single sign-on via any OIDC provider (Okta, Azure AD, Google, Keycloak). Opt-in via env vars (TURNSTONE_OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET). Security: - Authorization Code Flow with PKCE (S256) - State/nonce parameters with database-backed pending store (multi-node safe) - JWKS signature validation with async fetch + key rotation retry - Algorithm allowlist from JWKS key (not token header) prevents confusion - Identity matching exclusively by (issuer, sub) — prevents account takeover - password_enabled=false enforced server-side, not just UI - Rate limiting on both authorize and callback endpoints - OIDC users get "!oidc" password sentinel (bcrypt rejects naturally) - ID token validated for iss, aud, exp, nonce Features: - Auto-provisioning with username deduplication on first login - Claim-based role mapping with IdP demotion propagation (revokes stale roles) - "Continue with [Provider]" SSO button on login page - OIDC-only mode hides password form - Setup wizard required before OIDC login (admin bootstrap) Storage: migration 018 (oidc_identities + oidc_pending_states tables), 8 new protocol methods on both SQLite and PostgreSQL backends. 66 new tests (2273 total). * fix: address PR #71 review feedback (18 items) Bugs fixed: - OIDC success redirect now fetches permissions via new /auth/whoami endpoint before completing login (fixes permission-gating in UI) - Remove double decodeURIComponent on oidc_error (URLSearchParams already decodes; extra call throws on stray %) - Authorize rate limiter returns redirect instead of JSON 429 (endpoint reached via browser navigation, not fetch) - Lazy JWKS fetch in callback when startup discovery failed (IdP recovery without restart) - Startup exception handlers now log with exc_info=True - PostgreSQL pop_oidc_pending_state uses DELETE...RETURNING for true atomicity (eliminates TOCTOU) Behavior: - New OIDC users without role mapping get builtin-viewer by default (assigned_by="oidc-default", not revoked by role sync) Documentation fixes: - Role mapping: sync semantics (add + revoke stale), not "additive only" - PASSWORD_ENABLED=false blocks ALL password logins including admin - Algorithm: asymmetric allowlist, not per-key derivation - PlantUML diagram updated for role revocation API spec fixes: - Removed error_codes=[302] from callback (302 is success redirect) - Added /auth/whoami to both server + console specs - Regenerated TypeScript SDK OpenAPI snapshots (23 + 51 paths) * fix: address PR #71 round 2 review feedback (10 items) Rate limiting: - Authorize endpoint now calls record() after check() so the rate limiter actually counts attempts (was a no-op before) OIDC resilience: - Split startup try/except: discovery failure disables OIDC, JWKS prefetch failure leaves OIDC enabled for lazy retry on first login - JWKS unavailable message changed to "temporarily unavailable" (was misleadingly "not configured") - create_oidc_pending_state raises on collision instead of OR IGNORE (prevents silent insert drop on state collision) - SQLite pop_oidc_pending_state uses BEGIN IMMEDIATE for write lock (eliminates TOCTOU race) Frontend: - OIDC error display deferred 300ms so showLogin()'s async status fetch doesn't clear it via _switchMode → _clearError API spec: - OIDC authorize/callback endpoints now declare response_code=302 - Added AuthWhoamiResponse Pydantic model for /auth/whoami - Regenerated TypeScript SDK OpenAPI snapshots Documentation: - Diagram: JWKS "cached at startup, refreshed on-demand" (was "hourly") - Added TODO(tech-debt) comments on Host header redirect_uri sites |
||
|
Patrick Buckley
|
376da3d084 |
feat: prompt template tech debt — tests, read-only endpoints, double-… (#67)
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal Close test coverage gaps for prompt templates: - Resume with deleted template: verifies graceful degradation (template_content=None, warning logged) - Threading safety: concurrent set_template/init_system_messages with no race conditions - Factory passthrough: template kwarg propagation through WorkstreamManager.create() Add read-only template listing endpoints (read scope, no content exposed): - GET /v1/api/templates — prompt template summaries (name, category, is_default, origin) - GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model) - Available on both server and console; Python + TypeScript SDK methods added - Console creation modal switched from admin endpoint to read-scope endpoint Eliminate double-load inefficiency in workstream creation: - Template validation moved before mgr.create() (no create-then-rollback on invalid template) - template kwarg plumbed through WorkstreamManager.create() and session factory - _SessionFactory Protocol added for proper mypy typing Add workstream creation modal to server web UI: - Name, model, template dropdown, ws_template/profile dropdown - Instrument panel aesthetic: gradient top border, blur backdrop, amber accent - Focus trap, Escape/Enter keyboard handling, loading state, error display - WCAG AA contrast compliance, reduced-motion support * fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates() to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint. Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types. Regenerate openapi-server.json and openapi-console.json snapshots. Addresses Copilot review feedback on PR #67. * fix: skip template pre-validation when resuming a workstream When resume_ws is set, the request's template field is irrelevant — resume() restores the template from workstream_config. Pre-validating a stale template name would incorrectly return 400 before the resume even runs. Addresses Copilot review feedback on PR #67. |
||
|
Patrick Buckley
|
3658b77de8 |
feat: Discord content catch-up + bidirectional notification replies (… (#64)
* feat: Discord content catch-up + bidirectional notification replies (#64) Two improvements to the Discord channel adapter: 1. Fix intermittent dropped responses caused by a race between the bridge's two independent SSE connections (global SSE detects idle before per-ws SSE delivers all content tokens). The bridge now accumulates content in _ws_content_buffer and attaches it to TurnCompleteEvent.content. The Discord bot uses this as a catch-up when streaming events were missed. 2. Bidirectional notification replies — when the notify tool sends a DM, the message is tracked with the originating ws_id. Users can reply to the DM and the reply is routed to the workstream. The response is forwarded back to the DM, with the response itself tracked for multi-turn conversations. Includes user identity verification, stale notification feedback, and FIFO-capped tracking (100 entries). * fix: address Copilot review — re-insert on unlinked user, deque buffer - Re-insert _notify_ws_map entry when resolve_user returns None so the user can retry after linking (same pattern as user-mismatch re-insert) - Rename _MAX_CONTENT_BUFFER_BYTES → _MAX_CONTENT_BUFFER_CHARS (len() returns characters, not bytes) - Use deque + running total for O(1) popleft instead of list.pop(0) |
||
|
Patrick Buckley
|
19abc0cc65 |
feat: admin MCP Servers tab — database-backed MCP server management w… (#62)
* feat: admin MCP Servers tab — database-backed MCP server management with live status Add MCP Servers admin tab (14th tab, System group) for managing MCP server definitions via the database instead of static JSON config files. Storage: `mcp_servers` table (migration 016), 6 CRUD methods on both SQLite and PostgreSQL backends, `MCP_SERVER_MUTABLE` field allowlist. Config priority chain: DB rows (if any enabled) → CLI `--mcp-config` → `mcp.config_path` setting → none. Nodes auto-load from DB on startup via `load_mcp_config(storage=)`. Hot-reload: `reconcile_sync(storage)` diffs running servers against DB — adds missing, removes stale, reconnects changed. `_db_managed` set tracks DB-sourced servers so config-file servers (MCP_CONFIG env) are never removed by reconcile. Per-server `AsyncExitStack` for clean teardown. Reload pattern: console writes to DB then signals nodes via `POST /_internal/mcp-reload` (update by reference, no config payload). Console admin API: 7 endpoints under `/v1/api/admin/mcp-servers` (CRUD + reload + import), `admin.mcp` permission, secret masking (env/headers replaced with *** unless ?reveal=true), audit log sanitization. Unified view: tab merges DB-managed servers with config-sourced servers detected on nodes. Config servers shown as read-only rows with "config" badge — no edit/delete. Admin UI: 7-column grid with magenta status dots, transport badges, single-column create/edit modal, paste-based JSON import (mcpServers format), detail modal with per-node status. Mobile 3-column collapse, reduced-motion support, backdrop-click dismiss, focus trapping. SDKs: 7 methods on Python (async+sync) and TypeScript SDKs. Also fixes: Settings tab permission gate (admin.users → admin.settings), _ALL_PERMISSIONS list in governance.js (5 missing permissions added), _internal/mcp-reload added to APPROVE_PATHS. Docs: architecture.md (14 tabs), api-reference.md (7 endpoints), 20-mcp-architecture.puml updated with admin-driven lifecycle. 66 new tests (2232 total). * fix: address Copilot review feedback on MCP admin PR - Docs: fix "merges both sources" → "first-match-wins priority" (architecture.md) - Validation: require command for stdio, url for streamable-http transport - Validation: check args/headers/env types in import handler before storing - Schema: add transport/command/url to McpServerStatus, source to McpServerDetail - Thread safety: move all remove_server_sync mutations onto MCP event loop thread - Regenerate OpenAPI JSON snapshots for TypeScript SDK |
||
|
Patrick Buckley
|
8895bf07eb |
feat: admin Settings tab — form-based editor replacing "coming soon" … (#60)
* feat: admin Settings tab — form-based editor replacing "coming soon" stub Section-grouped layout with collapsible headers for all ~40 ConfigStore settings (model, session, tools, server, mcp, ratelimit, health, judge, memory). Type-appropriate inputs: CSS toggle for bools, number with min/max/step, select for choices, text for strings. Secret fields shown read-only. Source badge (storage/default), amber restart indicator. Inline save per field with dirty detection, row flash on success, reset to default via styled confirm modal. Full WCAG keyboard accessibility (Enter/Space on section headers, aria-labels, focus-visible). Mobile responsive single-column at <700px. Reduced-motion safe. * fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes Settings UI: - Help tooltips: ? button on ~25 settings with plain-English explanations and optional reference links (arXiv, Fowler, MCP spec). Click to toggle popover, Escape to dismiss, aria-expanded for accessibility. - Sections start collapsed for scannable overview. - Restart badge: hidden by default, shows when dirty, persists after save with amber glow. Positioned left of source badge. - Secret row alignment fixed (transparent border matches input box model). - Docs link in toolbar → Swagger UI Settings section. - Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard, numeric dirty detection (0.1 vs 0.10 no longer false positive). - Secret reset button enabled when source=storage (clear legacy overrides). - Space key repeat guard on section headers. - Sidebar: sticky + max-height:100vh, no longer stretches with content. Backend: - context_window default changed from 131072 to 0 (auto-detect). Fallback lowered from 131K to 32K (realistic for local models when detection fails). Session normalizes 0→32768 defensively. - Settings registry: help + reference_url fields on SettingDef, richer descriptions for model/session/tools/judge/memory settings. - Schema API includes help + reference_url. - Bootstrap system prompt: added Runtime Settings section. Docs: tab counts updated to 13 across README, architecture, console, governance. |
||
|
Patrick Buckley
|
101afd84da |
feat: database-backed settings (ConfigStore) with admin API (#59)
* feat: database-backed settings (ConfigStore) with admin API
Replace config.toml for non-bootstrap settings on the server with a
database-backed ConfigStore. ~40 settings across model, session,
tools, server, mcp, ratelimit, health, judge, and memory sections are
now managed via the admin Settings API. CLI flags for these settings
removed from the server entry point (CLI standalone tool unchanged).
Storage: system_settings table (migration 015) with composite PK
(key, node_id) for per-node overrides. ON CONFLICT upsert in both
SQLite and PostgreSQL. admin.settings permission granted to
builtin-admin role.
Settings registry (settings_registry.py): code-defined catalog of
all known settings with types, defaults, validation, descriptions.
Registry defaults aligned with previous argparse defaults.
ConfigStore (config_store.py): thread-safe in-memory cache loaded
from storage on init. Lock-free reads via dict snapshot swap.
reload() for hot-reload via internal endpoint.
Secret settings (judge.api_key) blocked from write via admin API
(403) — must be configured via config.toml or env vars.
warn_migrated_settings() logs warnings for config.toml keys that
overlap with ConfigStore-managed settings.
Console admin API: GET /v1/api/admin/settings (list with effective
values), GET .../schema (registry catalog), PUT .../{key} (update),
DELETE .../{key} (reset to default). Audit trail on mutations.
MQ: ConfigChangeEvent for cross-node cache invalidation (emission
from console deferred to bridge integration).
Python + TypeScript SDK methods. 63 new tests. Feature docs at
docs/settings.md, PlantUML diagram 24-settings-architecture.
* fix: address PR review — config-reload scope, registry defaults, doc alignment
- config-reload endpoint requires approve scope (was write)
- config-reload handler is sync def (avoids blocking event loop)
- reasoning_effort passes empty string through (removes `or "medium"`)
- ratelimit.requests_per_second changed to float (matches RateLimiter)
- session.retention_days allows 0 (disable pruning)
- ratelimit.trusted_proxies added to registry + wired in server
- admin_list_settings filters to global settings only (no node_id ambiguity)
- admin_update_setting validates "value" key presence (400 if missing)
- Console config change fans out reload to nodes directly (no MQ dep)
- Docs aligned with actual API response shapes and masking ("***")
|
||
|
Patrick Buckley
|
67f43a7ee0 |
feat: [memory] REST API endpoints + SDK methods + docs (#56)
* feat: [memory] REST API endpoints + SDK methods + docs
Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope
Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit
Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.
Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.
TypeScript SDK: matching methods + types on both clients.
Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.
Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.
Also fixes stray `total: int` on CreateChannelUserRequest.
* fix: [memory] address PR review — cross-user scope, schema types, snapshots
Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity. Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.
Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.
Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.
6 new security tests for user-scope access control.
|
||
|
Patrick Buckley
|
723cad24bb |
feat: structured memory system — typed/scoped memories with BM25 rele… (#53)
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting Replace flat key-value memories table with structured_memories (migration 014). Four memory types (user/project/feedback/reference), three scopes (global/workstream/user). Consolidate remember/recall/forget into two tools: memory (action-based: save/search/delete/list) and recall (conversation history only). BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5 memories for system message injection based on conversation context. Metacognitive prompting injects ephemeral nudges after corrections, tool denials, workstream resume, and completion signals. Scope isolation enforced: system message injection and nudge counts filtered to visible memories only (global + current workstream + authenticated user). User scope requires authentication. Content capped at 32KB. ILIKE/LIKE metacharacters escaped in both backends. 113 new tests (2053 total). * fix: CI failure + copilot review feedback - Fix time.monotonic() cooldown: use None sentinel instead of 0.0 default (monotonic clock starts at boot, not epoch — fresh CI runners have uptime < 300s so cooldown check always triggered) - Catch sa.exc.IntegrityError specifically in upsert instead of broad Exception (copilot review) - Preserve existing description/type on upsert when caller doesn't explicitly set them (copilot review) - Add last_accessed + access_count columns to schema/migration for future LRU/LFU eviction support |
||
|
Patrick Buckley
|
1295919613 |
fix: simplify conversation storage — atomic assistant rows with tool_… (#51)
* fix: simplify conversation storage — atomic assistant rows with tool_calls JSON Replace the denormalized storage model (separate rows for assistant content, tool_call, tool_result) with atomic assistant rows carrying tool_calls as a JSON column. Eliminates the 100-line heuristic reconstruct_messages function and its cross-turn merge bug. Schema: add tool_calls TEXT column to conversations (migration 013). Migration backfills existing data — merges tool_call rows into their parent assistant row as JSON, renames tool_result to tool, deletes consumed tool_call rows. Session save path: assistant content + tool_calls saved in one save_message call before tool execution (crash resilient). Tool results saved as role="tool". Extract shared storage utilities to _utils.py: row_to_dict, mutable field frozensets, reconstruct_messages. Both backends import from _utils — PostgreSQL no longer depends on _sqlite.py. Includes denied/blocked tool call badge fix on resume: _build_history detects denied results and propagates flag to parent assistant entry. Frontend uses flag for correct badge-denied rendering. Denied tools visually muted. role="status" on badges for accessibility. Net -45 lines. 8 new tests for reconstruction, all 1914 tests pass. * fix: migration 013 uses parameterized deletes and ordered downgrade - DELETE of consumed tool_call rows now uses parameterized batches (chunks of 500) instead of string interpolation - Downgrade rebuilds via temp table to preserve chronological id ordering when re-inserting tool_call rows |
||
|
Patrick Buckley
|
09ea3d164d |
feat: intent validation v1 — advisory LLM judge for tool approvals (#50) (#50)
* feat: intent validation v1 — advisory LLM judge for tool approvals (#50) Two-tier evaluation pipeline for non-auto-approved tool calls: - Heuristic tier (instant): 23 pattern-based rules across 4 severity levels (critical/high/medium/low) with first-match-wins priority - LLM judge tier (async): multi-turn evaluation with read_file/ list_directory tool access, security-hardened path blocking, forcing message on final turn, four-stage JSON parsing with retry nudge Progressive UI: heuristic verdict badge + judge spinner, LLM verdict upgrade via intent_verdict SSE event, glow on action buttons. Verdict persisted to intent_verdicts table for audit. Prometheus metrics for verdict counts and LLM latency. Enabled by default (--no-judge to opt out). 132 new tests (1938 total). Integration: session, server/WebUI, CLI, MQ bridge, console admin API, Discord channel adapter. Config via [judge] in config.toml or CLI flags. * fix: address PR #50 Copilot review feedback - Fix double JSON encoding of func_args in both heuristic and LLM verdict persistence paths — use pre-serialized string from verdict - Fix confidence 0.0 treated as falsy in channel verdict formatter - Fix timestamp format inconsistency in storage backends (isoformat vs strftime) — now uses strftime consistently - Add on_intent_verdict to eval.py NullUI (mypy fix) - Fix late verdict after approval resolved — store last decision and apply immediately to late-arriving verdicts - Add permission rollback to migration 012 downgrade - Update docs to reflect judge enabled by default - Document confidence_threshold as reserved for v2 * fix: judge per-call timeout and credential recon heuristic - Wrap create_completion() in ThreadPoolExecutor with per-call timeout to prevent indefinite hangs on slow local models. On timeout, replace the executor so subsequent batch items don't queue behind lingering API calls - Add IntentJudge.shutdown() and wire into session.close() for cleanup - Add credential-recon heuristic rule: /etc/passwd, /etc/shadow, /etc/master.passwd access flagged as HIGH/review (reconnaissance pattern even though the command itself is read-only) - 3 new tests for credential file access patterns * fix: denied/blocked tool calls show correct badge on resume - _build_history() detects denied results ("Denied by user") and blocked results ("Blocked") and propagates denied flag to parent assistant entry for frontend consumption - Frontend history replay uses denied flag for badge-denied class instead of hardcoding badge-approved for all historical tool calls - Denial feedback always prefixed with "Denied by user:" so content detection works with custom user feedback - Denied tools visually muted (opacity 0.55, muted tool name) - role="status" on all approval badge elements (accessibility) - Broadened "Blocked" prefix match (catches "Blocked by tool policy") |
||
|
Patrick Buckley
|
02d9c5c797 |
feat: workstream templates — behavioral profiles for workstream creation (#49)
* feat: workstream templates — behavioral profiles for workstream creation Workstream templates define the complete configuration for workstream creation: system prompt, model, auto-approve policy, per-tool auto-approve, temperature, reasoning effort, max tokens, agent max turns, token budget, and completion notifications. Applied once at creation time (snapshot, not live binding). Auto-versioning captures pre-update state on every edit. Schema & storage: - workstream_templates + workstream_template_versions tables (migration 011) - ws_template_id/ws_template_version columns on workstreams table - ws_template column on scheduled_tasks table - Full CRUD + versioning on SQLite and PostgreSQL backends - prompt_template_hash (SHA-256) for drift detection Runtime: - Template resolution before mgr.create() for model override - Post-creation settings application (prompt, temperature, approval, budget) - Token budget enforcement in session.send() — 80% warning, approval gate at 100% via __budget_override__ synthetic tool - WebUI.auto_approve_tools server-side per-tool auto-approve - Prompt template drift detection (hash comparison, log warning on mismatch) Integration: - ws_template field on CreateWorkstreamMessage, bridge, channel router, scheduler dispatch, MQ client - Console admin "WS Templates" tab (11th) with CRUD, version history modal - Profile dropdown on workstream creation modal - WS template dropdown on scheduler create/edit modals - Prompt template name validation on ws_template create/update - 7 console admin API endpoints + read-only summary endpoint - Full OpenAPI spec entries in console_spec.py - Python SDK (sync + async) and TypeScript SDK methods - Pydantic schemas for all request/response models Docs & diagrams: - New 21-ws-template-architecture.puml sequence diagram - Updated governance, storage, MQ protocol diagrams + PNGs - Updated architecture.md, governance.md, api-reference.md, console.md, sdk.md 48 new tests (1788 total). mypy clean. ruff clean. * fix: address PR #49 review feedback - auto_approve_tools uses approval_label (not just func_name) for consistency with tool policy evaluation - inline system_prompt from ws_template persisted as _ws_template_system_prompt in workstream_config, restored on resume (previously lost because _template_content wasn't persisted) - budget gate (__budget_override__) no longer bypassed by blanket auto_approve — requires explicit approval or tool policy allow - diagram 21 field list corrected (removed tool_search/threshold, added prompt_template_hash/notify_on_complete) * fix: address PR #49 review feedback (round 2) - Grant admin.ws_templates permission in migration 011 (tab was hidden) - Center WS template modals and fix radio button alignment - Skip template validation when ws_template overrides prompt - Guard against empty version snapshots on no-op updates - Replace setTimeout race with Promise chain in schedule ws_template select - Validate numeric fields in admin create/update handlers (400 not 500) - Add ws_template to TypeScript OpenAPI specs - Use typed Pydantic response models in SDK ws_template methods |
||
|
Patrick Buckley
|
2f7f70825b |
feat: wire prompt templates into session startup with full creation-p… (#47)
* feat: wire prompt templates into session startup with full creation-path support
Prompt templates (prompt_templates table) now have runtime effect:
- is_default=true templates auto-apply as system message content,
concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
field on POST /v1/api/workstreams/new, console creation modal dropdown,
scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession
Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion
Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.
Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.
Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.
Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.
* fix: address PR #47 review feedback
- Defer template validation until after resume_ws — a bad template name
no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
reject unknown template names with 400 instead of allowing schedules
that would silently fail at dispatch time
|
||
|
Patrick Buckley
|
8b2e2130fc |
fix: MCP resource template URI expansion via prefix matching (#46)
* fix: MCP resource template URI expansion via prefix matching
Resource templates (RFC 6570 URI patterns like `db://tables/{table}/rows/{id}`)
were discovered from MCP servers but non-functional — `read_resource_sync()`
only accepted exact URIs from `_resource_map`, which excludes templates.
Add prefix-based fallback: extract the static prefix from each template
(everything before the first `{`), store a prefix→server mapping, and
fall back to longest-prefix matching when exact URI lookup fails. MCP
servers handle URI routing internally so we just need to route the
expanded URI to the correct server.
Also surface templates in the system message catalog and `/mcp` command
so the model knows they exist and can construct expanded URIs.
* fix: address PR #46 review feedback
- Template prefix collision now keeps more specific (longer) template
URI instead of blindly overriding
- Fix _match_template docstring to accurately describe startswith
matching on static prefixes (not full template matching)
- Add missing loop.close() in integration test finally block
- Rewrite test_template_longest_prefix_wins with genuinely different
prefix lengths to avoid brittle collision-order dependency
|
||
|
Patrick Buckley
|
f81c06761d |
chore: remove dead code, add MCP integration + collector tests (#45)
* chore: remove dead code, add MCP integration + collector tests Remove unused delete_prompt_templates_by_server from protocol and both storage backends (sync uses per-template deletion). Add 10 MCP integration tests exercising full lifecycle: rebuild resources/prompts, read_resource_sync/get_prompt_sync with real asyncio loop, governance sync to real SQLite, shutdown cleanup, listener notification isolation. Add 3 console collector MCP aggregation tests: multi-node sums, absent when zero, mixed nodes with/without MCP. * fix: close event loops and SQLite backend in MCP integration tests |
||
|
Patrick Buckley
|
be165c1971 |
feat: MCP resource and prompt discovery with read_resource tool (#44)
* feat: MCP resource and prompt discovery with read_resource tool Extends MCPClientManager with resource and prompt discovery alongside existing tool support. Resources and prompts are discovered on connect, cached per-server with copy-on-write rebuilds, and refreshed via push notifications, periodic polling, or manual /mcp refresh. New read_resource built-in tool reads MCP resources by URI. Requires user approval (same as MCP tool calls) since resources are served by external MCP servers. Resource catalog injected into system message with XML delimiters. Error messages sanitized to prevent leaking server internals to the model. Prompt discovery stores prefixed names (mcp__server__prompt) and exposes get_prompt_sync() for future use_prompt tool (Chunk D). /mcp command now shows tools, resources, and prompts. Docs and diagrams updated. * feat: MCP prompt governance sync with origin tracking and readonly guards Migration 009 adds origin, mcp_server, and readonly columns to prompt_templates. MCP prompts discovered by MCPClientManager are automatically synced into the governance table as read-only templates with origin="mcp". Sync engine handles: create on connect, update on prompt refresh, delete when prompts are removed from server. Manual templates take precedence on name collision (MCP prompt skipped with warning). Admin API returns 403 on update/delete of readonly templates. Console UI shows MCP origin badge and disables edit/delete buttons. Storage backends gain get_prompt_template_by_name, list_prompt_templates_by_origin, and delete_prompt_templates_by_server methods. Also addresses PR #44 review feedback: concurrent.futures.TimeoutError handling in sync dispatch, XML-escape resource catalog descriptions, resource template entries excluded from _resource_map, URI collision warnings, needs_periodic capability-aware computation, malformed JSON primary key fallback for read_resource. * feat: use_prompt tool, prompt catalog, and PR review hardening New use_prompt built-in tool invokes MCP prompt templates by name, expanding them into messages. Requires user approval (external MCP servers). Prompt catalog injected into system message with XML delimiters (up to 30 prompts, HTML-escaped). Prompt listener registered in session for catalog rebuild on changes. Addresses PR #44 review feedback: - _init_system_messages() now uses copy-on-write (build locally, assign atomically) so background thread callbacks never see partial system messages - sync_prompts_to_storage() serialized behind _sync_lock to prevent races between set_storage() (main thread) and MCP background thread - shutdown() clears listener lists to release callback references Docs and diagrams updated for 18 built-in tools. * feat: granular tool policies for MCP resources, prompts, and tools Policy evaluation now uses approval_label (falling back to func_name) for fnmatch pattern matching, enabling fine-grained per-URI and per-server policies: - read_resource: mcp_resource__{normalized_uri} - use_prompt: mcp__{server}__{prompt} (prefixed name) - MCP tools: mcp__{server}__{tool} (was static "mcp_tool") URI normalization resolves .. path segments to prevent traversal bypasses in policy matching. Resource templates filtered from system message catalog (not directly readable). use_prompt arguments validated as dict with string coercion. TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly fields. Governance docs updated with MCP policy patterns. * feat: MCP visibility in server and console UIs Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts counts. Server UI status bar shows magenta MCP indicator with tooltip. Console cluster status bar shows MCP metrics with magenta LED dot. Console node detail view shows per-node MCP summary. Console collector aggregates MCP counts across nodes in overview. Uses var(--magenta) design token with new --magenta-glow for theme adaptation. ARIA roles on MCP status elements. Tooltips on console MCP metric labels. Node MCP summary hidden on mobile (< 700px). New diagram: 20-mcp-architecture.puml covering full MCP lifecycle (connection, discovery, refresh, governance sync, policy, UI). * fix: McpStatus in health schema, count properties, catalog name fidelity Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so typed clients see the mcp field from /health. Addresses Copilot review feedback: - resource_count/prompt_count properties avoid list allocation on /health and /metrics polls - get_tools/resources/prompts return shallow-copied dicts to prevent callers from mutating internal cache - Prompt names and arg names in system message catalog are NOT HTML-escaped (model must use exact strings in use_prompt calls); only descriptions are escaped * fix: OpenAPI spec McpStatus + diagram approval column accuracy Adds McpStatus schema and optional mcp field to HealthResponse in openapi-server.json, matching the Python schema and TypeScript types. Fixes tool pipeline diagram: math, web_fetch, web_search correctly shown as auto-approve (not "Yes" for approval). |
||
|
Patrick Buckley
|
3264fdefca |
fix: channel bidirectional routing — emit TurnCompleteEvent on all id… (#43)
* fix: channel bidirectional routing — emit TurnCompleteEvent on all idle transitions Bridge previously only emitted TurnCompleteEvent for MQ-initiated turns (those with a correlation_id in _active_sends). Server-UI-initiated turns went idle without emitting TurnCompleteEvent, so the Discord bot's StreamingMessage never finalized — content accumulated in the buffer and collided with the next Discord-triggered response. Now TurnCompleteEvent is emitted unconditionally on every idle transition. correlation_id is empty for non-MQ turns; SDK client filters by correlation_id so existing consumers are unaffected. * fix: remove unused variable flagged by ruff |
||
|
Patrick Buckley
|
28cb3a5c51 |
fix: approval timeout UI state and content flush before tool calls (#42)
* fix: approval timeout UI state and content flush before tool calls Two bug fixes: 1. Approval timeout now shows denied state in UI — resolve_approval() emits an approval_resolved SSE event so the browser transitions from pending to denied (red border + badge). Also fixes the cancel- during-approval path. Frontend resolveInlineApproval() gains a skipPost parameter to avoid redundant POST when server-initiated. ApprovalResolvedEvent added to Python and TypeScript SDKs. 2. Content streaming flushes pending buffer before tool call deltas — _stream_response() held up to 13 trailing chars in the pending buffer (for <think> tag detection) when transitioning to tool calls. Now flushed eagerly when tool_call_deltas arrive, before clearing in_think so reasoning text is correctly categorized. * fix: address Copilot review feedback on PR #42 Patch _execute_tools in stream flush test to prevent real bash execution, simplify confusing nested comprehension, and update resolve_approval() docstring to reflect cancel/timeout call paths. |
||
|
Patrick Buckley
|
a2e2ffacd8 |
feat: robust plan quality gate, iterative refinement, and amend UX (#41)
* feat: robust plan quality gate, iterative refinement, and amend UX Plan agent output from weak models often produced garbage (11-char plans that echo the prompt). Two fixes: 1. Quality validation (_validate_plan) checks length, section structure, echo detection, and refusal patterns. Fails trigger one automatic retry with a coaching message injected into the agent's existing conversation, preserving all prior exploration context. 2. Iterative feedback loop — user feedback at plan review re-runs the plan agent via _refine_plan() instead of appending text to the tool result. Up to 5 refinement rounds. The plan file path is always included in the tool result so the outer model knows where it lives. UI improvements: - Web: Reject button dynamically becomes "Amend" (amber) when feedback is typed. Key hint badges (Esc/Enter) on plan buttons. Main input disabled during review. Light-theme contrast fix via --on-color var. - CLI: Prompt shows all three actions (approve/amend/reject). - Bridge: Race condition fix — clear pending entry before HTTP POST so sequential plan reviews from the refinement loop aren't skipped. 15 new tests covering validation, retry, and refinement. * fix: address PR 41 review feedback - Escape key in plan dialog now mirrors the Amend button: if feedback is typed, Esc sends the feedback (amend); if empty, Esc rejects. Previously Esc always hard-coded "reject", discarding typed feedback. - Coaching message for plan retry now says "should include at least two of" instead of "MUST include these", matching the actual validation rule (_MIN_PLAN_SECTIONS = 2). * feat: render plan inline in chat after approval After the plan review dialog closes, the plan content is now rendered as a collapsible inline block in the chat stream — styled with a status header (approved/rejected/amending), markdown-rendered body, and feedback note when amending. Uses the same makeCollapsible pattern as tool output blocks. * fix: prevent plan approval hang when inline render fails The authFetch call that unblocks the server must fire before the cosmetic inline plan rendering. Previously _addInlinePlan ran first and any JS error (e.g. from renderMarkdown) prevented the API call, leaving the session thread blocked forever. - Move authFetch before _addInlinePlan - Wrap _addInlinePlan in try-catch - Guard against empty content - Only auto-collapse plans longer than 12 lines * fix: address PR 41 review feedback (round 2) - Max refinement rounds no longer implicitly approve: the loop now shows the final plan for explicit approve/reject before proceeding. Previously exhausting 5 rounds silently accepted the last revision. - Plan inline block: correct aria-label from "Tool output" to "Plan content" when makeCollapsible is applied. - XSS concern (not applicable): renderMarkdown is used for all assistant messages — plan content follows the same trust model. - Test loop concern (acknowledged): refinement tests verify component logic; full _execute_tools integration would require extensive mocking for marginal coverage gain. * feat: thinking spinner + inline plan hardening * fix lint |
||
|
Patrick Buckley
|
c6ba8d59b0 |
feat: bootstrap wizard — LLM-guided interactive setup for deployments
Add `turnstone-bootstrap`, a new entry point that uses any LLM (OpenAI, Anthropic, or local/vLLM) to conversationally walk users through configuring a Turnstone deployment. Generates .env files, setup.sh scripts, and optional docker-compose overrides. - Fully interactive startup (zero CLI args) with provider/model selection - Auto-detects available models on local OpenAI-compatible endpoints - 7 tools: read_file, write_file, generate_secret, check_port, validate_api_key, check_docker, finish - Path traversal protection on file read/write - Duplicate write detection (skips identical content) - Bounded retry loop (3 attempts) on LLM errors - Anthropic message conversion with consecutive-role merging |
||
|
Patrick Buckley
|
fd507c6a3c |
feat: generation cancellation — stop button, cancel API, cooperative … (#40)
* feat: generation cancellation — stop button, cancel API, cooperative cancel Add cooperative cancellation via threading.Event on ChatSession. The cancel signal is set from outside the worker thread (HTTP handler, MQ bridge, or Escape key) and checked at defined checkpoints: per streaming chunk, before tool execution, inside bash commands, and at each sub-agent turn. Core: GenerationCancelled(BaseException) exception, cancel()/_check_cancelled() methods, partial content preservation in _stream_response, clean rollback in send() with idle state emission (no re-raise). Server: POST /v1/api/cancel endpoint, CancelledEvent SSE emission, worker thread safety net. Frontend: Stop button (■ Stop) with send/stop swap via setBusy(), Escape key shortcut, cancelled event handler. Accessible: aria-label, focus-visible override, light theme contrast, non-color differentiation. MQ: CancelMessage inbound type, bridge _handle_cancel routed handler. SDK: cancel() on Python async+sync clients, CancelledEvent in Python+TypeScript event registries, isCancelledEvent type guard. OpenAPI: CancelRequest schema + endpoint spec. Docs: API reference, architecture, SDK docs updated. Diagrams: conversation turn, tool pipeline, MQ protocol, workstream states, SDK architecture. * fix: address PR #40 review feedback - setBusy() now resets stopBtn.disabled so stop button is re-enabled on next generation after a successful cancel - Gate cancel side effects (resolve_approval, resolve_plan, cancelled SSE event) on worker_thread.is_alive() to avoid spurious events when idle - Add /v1/api/cancel endpoint and CancelRequest schema to TypeScript openapi-server.json to keep it in sync with Python-generated spec |
||
|
Patrick Buckley
|
7492816ab2 |
feat: governance — RBAC, tool policies, prompt templates, usage track… (#39)
* feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging Add comprehensive governance layer for the admin console: - RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer), custom role CRUD, user-role assignment with privilege escalation prevention - Tool policies with glob pattern matching, priority-ordered evaluation (allow/deny/ask), enforced before auto-approve in WebUI.approve_tools() - Prompt templates with variable substitution, categories, default flag - Usage tracking: per-LLM-request token/tool metrics, aggregated queries (group by day/model/user), automatic 90-day pruning via scheduler - Audit logging: append-only event trail for all admin mutations, filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For aware IP extraction - require_permission() enforced on all 35+ admin endpoints (users, tokens, channels, schedules, watches, roles, orgs, policies, templates, usage, audit) - Field allowlists on storage update methods prevent mass-assignment bugs - Self-deletion guard on admin_delete_user, delete_user cascades user_roles - _row_to_dict helper eliminates ~400 lines of fragile positional row mapping - _audit_context helper deduplicates 18 instances of audit boilerplate - Migration 008: 7 new tables, 3 builtin roles, org_id on users - Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit) with permission-gated visibility, 7 modal dialogs, full keyboard accessibility - Python + TypeScript SDK methods for all governance endpoints - 120+ new tests (1554 total) * fix: address PR #39 review feedback - Rebuild serialized items after policy evaluation so denied/allowed verdicts are reflected in tool_info/approve_request SSE payloads - Make `since` query param optional in usage OpenAPI spec (handler already defaults to last 7 days) - Add response_model=StatusResponse to DELETE role/policy/template and POST/DELETE role assignment endpoints in OpenAPI spec - Add missing org_id/created/updated fields to UserRoleInfo schema - Add missing created field to AuditEventInfo schema - Show "no permissions" empty state instead of loading inaccessible tab when all admin tabs are permission-gated - Fix "13 permissions" → "15 permissions" in architecture.md and security.md - Fix import sorting in test_audit.py and test_tool_policy.py * fix: address PR #39 round 2 review feedback - Clear stale permissions from sessionStorage on config-token login (auth.js _storePermissions) - Only trust X-Forwarded-For when behind a proxy that sets X-Forwarded-Proto (conditional on is_secure_request trust model) - Thread user_id from auth into WebUI.on_status for usage events - Add created field to TS AuditEventInfo type - Return typed Pydantic models from all SDK governance methods instead of dict[str, Any] — both async and sync clients - Validate group_by param against allowed enum in admin_usage handler - Add deterministic secondary sort (event_id DESC) to list_audit_events in both SQLite and PostgreSQL backends |
||
|
Patrick Buckley
|
70d495aa5b |
fix: per-workstream SSE fan-out — multiple consumers no longer steal … (#38)
* fix: per-workstream SSE fan-out — multiple consumers no longer steal each other's tokens
After
|
||
|
Patrick Buckley
|
de64535221 |
Feat/eval improvements (#37)
* ci: add GitHub Release creation on tag push
* refactor: rename plan tool to create_plan
Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns
* feat: eval harness improvements inspired by autoresearch patterns
Major enhancements to turnstone-eval:
- Per-test timeout (--test-timeout, default 300s) and suite timeout
(--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
models and providers (--optimizer-model, --observer-model, etc.)
with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)
* fix: address Copilot review feedback
- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
|
||
|
Patrick Buckley
|
187d004033 |
feat: watch tool — periodic command polling within workstreams (#36)
* feat: watch tool — periodic command polling within workstreams
Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.
Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)
* fix: address Copilot review — condition errors, restore deadlock, docs
- Condition eval errors now deactivate the watch immediately instead
of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
|
||
|
Patrick Buckley
|
3bc3250869 |
fix: recovered workstreams invisible in console UI (#35)
* fix: recovered workstreams invisible in console UI Bridge startup recovery (_recover_workstreams) re-registered workstream ownership but never published WorkstreamCreatedEvent to the cluster channel. The collector's poll loop would pick up the workstream in its internal state, but _apply_poll never fanned out SSE events to connected browsers. Combined, this made channel-resumed workstreams invisible in the console while remaining accessible through the proxied node UI. - Bridge: emit WorkstreamCreatedEvent for each recovered workstream - Collector: diff poll results and fan out synthetic ws_created/ws_closed events for workstream additions and removals - Skip workstreams with empty IDs in poll processing - Add 4 tests for poll-diff fanout behavior - Update console data-flow diagram and architecture docs * fix: address PR review — filter empty ws IDs, stable event ordering - Filter empty-string keys from old_ids to avoid phantom ws_closed events if a previous poll inserted a workstream under key "". - Sort set diffs before iterating so ws_created/ws_closed fanout order is deterministic across poll cycles. |
||
|
Patrick Buckley
|
5f0004dc91 |
feat: add ClusterSnapshot for instant console UI state rebuild (#34)
* feat: add ClusterSnapshot for instant console UI state rebuild
The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.
Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).
Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.
SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.
* fix: address review feedback and SSE proxy reconnect bug
Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
both state and listener locks, eliminating the event gap between
snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
console schema models (ClusterNodeInfo, NodeDetailResponse,
ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
and ClusterOverviewResponse types instead of Record<string, unknown>.
SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
when no upstream data arrives, preventing the browser EventSource
from dropping idle connections. The raw byte passthrough refactor
(
|
||
|
Patrick Buckley
|
6cc1b3a5bd |
feat: add vision/image support to read_file tool (#33)
* feat: add vision/image support to read_file tool read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO) and returns base64-encoded content parts for vision-capable models. Non-vision models receive a text description instead. A new supports_vision flag on ModelCapabilities gates the feature, with config.toml [models.*.capabilities] overrides for local models (vLLM, llama.cpp, NIM). * fix: address PR review feedback - Discard _read_files on no-vision OSError path, include exception detail - Discard _read_files on oversized image error (not a successful read) - Validate capabilities type from config.toml (reject non-dict) - Clarify tool description re: vision behavior and offset/limit scope - Remove unused os import in tests, fix import sort order - Handle list content (image tool results) in eval.py tool result loop |
||
|
Patrick Buckley
|
4d1107839b |
refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing - Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy - Stream bytes verbatim to preserve server-side ping comments and event framing - Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering) - Update compose.yaml to add 'cluster' profile to the service * Refactor SSE proxy to raw byte passthrough - turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with httpx.stream() + StreamingResponse for raw byte passthrough. Server pings, events, and comments now flow through verbatim. Added per-request timeout override (read=None, pool=None) for long-lived SSE streams. - tests/test_console.py: Add 3 new tests for SSE proxy: - Ping and event preservation - Upstream error status handling - Client disconnect handling - docs/console.md: Update SSE Proxy section to reflect raw byte passthrough approach. |
||
|
Patrick Buckley
|
c79c47b940 |
Add MCP dynamic tool refresh with push notifications and periodic pol… (#31)
* Add MCP dynamic tool refresh with push notifications and periodic polling MCP tool lists now stay up-to-date without restart via three mechanisms: push notifications (ToolListChangedNotification) for servers that support it, staggered periodic polling for servers that don't, and manual /mcp refresh [server] command. MCPClientManager tracks tools per-server with copy-on-write rebuild, notifies ChatSession listeners which rebuild tool lists and ToolSearchManager (preserving expanded tools). * Address Copilot review feedback on MCP refresh PR - Fix /mcp refresh typo matching (startswith → exact token check) - Validate --mcp-refresh-interval >= 0 at parse time via shared nonneg_float in config.py (deduplicated from cli.py + server.py) - Clamp negative refresh_interval to 0 in MCPClientManager constructor - Fix periodic refresh first poll timing (was initial_delay + interval, now initial_delay then immediate first poll) - Clarify _on_mcp_tools_changed docstring re: O(n) BM25 build cost |
||
|
Patrick Buckley
|
c7586abd0a |
Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI
When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:
- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool
New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.
Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.
43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.
* Fix Copilot review feedback on tool search
- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
|
||
|
Patrick Buckley
|
fb190f8977 |
Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity Eliminate the separate session_id concept. The workstream ID (ws_id) is now the single identity used for both real-time routing and conversation persistence, removing a layer of indirection that was 1:1 in practice and buggy on resume (stale pointers, orphaned rows). Schema changes (migration 006): - Drop sessions table; add alias/title columns to workstreams - Rename conversations.session_id → ws_id - Rename session_config table → workstream_config (ws_id column) - Data migration remaps existing conversations to ws_id Storage/API renames: - register_session → register_workstream (already existed, merged) - save_message/load_messages now keyed by ws_id - resolve_session → resolve_workstream - ChatSession.session_id property → ws_id - ChatSession.resume_session() → resume() - resume_session field → resume_ws - SessionResumedEvent → WorkstreamResumedEvent - /api/sessions → /api/workstreams/saved - /sessions slash command → /workstreams - --session-retention-days → --retention-days Channel eviction recovery simplified: reuses old ws_id directly instead of get_session_id_by_ws() reverse lookup. * Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec - docs/channels.md: "resumes the session" → "resumes the workstream", "Session resumed:" → "Resumed:", "old session was pruned" → "old workstream was pruned" - docs/api-reference.md: "Each session object" → "Each saved workstream object", field descriptions updated, removed stale node_id field - sdk/typescript/openapi-server.json: fully regenerated from Python models — removes all stale session_id properties from WorkstreamInfo, DashboardWorkstream, CreateWorkstreamResponse schemas |
||
|
Patrick Buckley
|
339981a258 |
Add GPT-5.3, GPT-5.4, and pro model capabilities (#28)
* Add GPT-5.3, GPT-5.4, and pro model capabilities Add capability entries for gpt-5-pro (272k output, high-only reasoning), gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro. * Validate reasoning_effort against model capabilities _apply_model_params now falls back to caps.default_reasoning_effort when the requested value is not in caps.reasoning_effort_values. Prevents sending unsupported effort levels to models like gpt-5-pro (high only). |
||
|
Patrick Buckley
|
924b976f1f |
Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system (cron/at scheduling via console API). Includes Python SDK methods (async + sync), TypeScript SDK methods, console.md API reference, sdk.md table update, and architecture.md module map entry. |
||
|
Patrick Buckley
|
e7fe8fca9d |
Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening Implements the `notify` tool allowing the LLM to send notifications to Discord channels/users via the channel gateway. Includes fixes for 11 review findings: JWT auth on the gateway endpoint, first-healthy gateway delivery with retry+backoff, rate limiting only on success, SSRF URL scheme validation, Discord mention sanitization, SQLite ON CONFLICT upsert preserving created timestamps, advertise URL resolution for 0.0.0.0 bind, randomized service IDs, generic error messages to prevent internal state leakage, and partial direct-target validation. Service registry with heartbeat-based health filtering (migration 005). Channel gateway registers on startup, heartbeats every 30s, deregisters on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint auth (static + JWT), storage CRUD, and retry behavior. * Add notify documentation, diagrams, and review fixes Documentation: - New sequence diagram 17-notify-flow.puml showing end-to-end delivery - Updated 16-channel-architecture.puml with services table, notify HTTP path, and Notification Flow note - channels.md: Notifications section (targeting, delivery flow, service registry, security) and new config table entries - tools.md: notify tool reference, updated counts/tables (14→15 tools) - security.md: channel gateway row in service-to-service auth table - architecture.md: notification subsystem paragraph Review fixes (copilot): - _http.py: fail closed when auth unconfigured (401 instead of pass- through), strip whitespace on message/title, generic error messages for user-not-found vs no-linked-channels - session.py: parse gateway response JSON and require at least one result with status=="sent" before counting as success - _postgresql.py: use index_elements instead of constraint for upsert |
||
|
Patrick Buckley
|
42b9f89988 |
Add scheduled task system with cron/at scheduling (#26)
* Add scheduled task system with cron/at scheduling, admin API, and console UI Console-integrated background scheduler dispatches workstreams on recurring cron expressions or one-shot ISO8601 timestamps. Four target modes: auto (best node by headroom), pool (shared queue), all (fan-out), or specific node. Redis distributed lock with unique owner + Lua conditional release prevents duplicate dispatch in multi-console deployments. Storage: scheduled_tasks + scheduled_task_runs tables (migration 004), 9 protocol methods on both SQLite and PostgreSQL backends, field allowlist on updates, run history auto-pruned at 90 days. API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter validation, ISO8601 future-time checks, field length bounds, schedule count cap (200), and OpenAPI spec entries with Pydantic models. UI: Schedules tab in admin panel with create/edit/delete modals, run history modal, cron/at type toggle, target mode select, status dots for accessibility, responsive grid, keyboard navigation, and focus management. Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches logged at WARNING with created_by attribution, user_id propagated in CreateWorkstreamMessage for audit trail. 46 tests across storage, scheduler engine, and API endpoints. * Add croniter to test extras for CI compatibility CI installs [test] extras but not [console], so croniter was missing when schedule API tests import console/server.py validation functions. * Address Copilot review: timezone validation, focus trap, enabled flag - Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z) - UI appends +00:00 to datetime-local values for explicit UTC - Fix datetime-local normalization: check length before appending seconds - Add textarea to modal focus trap selector (prevents focus escape) - Fix _normalize_task_dict not called in update response - Persist enabled=false on create (storage defaults to enabled=1) - Validate at_time is still in future when re-enabling a one-shot task - broker._redis coupling acknowledged as tracked tech debt |
||
|
Patrick Buckley
|
872e1770e6 |
Feature/code dedup (#25)
* Add JWT auth security hardening (6 fixes) - Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h - Login brute-force protection: LoginRateLimiter (5 attempts/5min per key) - JWT aud/iss claims: create_jwt/validate_jwt support audience validation - Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh - CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var - JWT secret strength: warning on secrets shorter than 32 chars - Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing * Refactor duplicated code into shared utilities and fix 3 UI bugs Code deduplication (~235 net lines removed): - Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py - Create core/web_helpers.py (require_storage_or_503, read_json_or_400, parse_cors_origins, cors_middleware) - Extract add_redis_args/broker_from_args to mq/broker.py - Extract add_log_args/configure_logging_from_args to core/log.py - Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper, unused required_role(), duplicate detect_model() wrapper Bug fixes: - Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server nodes (use ServiceTokenManager with JWT_AUD_SERVER instead) - Fix login form autofill: wrap inputs in <form>, add name attributes, set type=submit on button - Fix SSE reconnecting flash: add onopen handler to clear status immediately on connection (not waiting for first message) - Fix chat scroll: add min-height:0 to flex containers, overflow:hidden on body to constrain viewport height * Address CI typecheck failure and Copilot review feedback - Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary) - Bridge SSE loops: use event_hooks for auth header refresh on reconnect instead of static headers that go stale after token rotation - Login form: remove javascript:void(0) action (CSP anti-pattern) - Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals in middleware builder calls to prevent drift |
||
|
Patrick Buckley
|
a6e929b0a0 |
Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24) Bidirectional channel adapter framework connecting external messaging platforms to turnstone workstreams via Redis MQ. Discord ships as the first adapter; the protocol supports future Slack/Teams integrations. Channel framework: - ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping - AsyncRedisBroker with single dispatch loop and per-channel ordered workers - channel_routes table (migration 003) for persistent route storage - 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD) - Unified turnstone-channel gateway entry point, loads adapters by config - Message chunking, approval formatting, plan review formatting Discord adapter: - discord.py v2.4+ bot with thread-per-@mention model - Slash commands: /link (modal), /unlink, /ask, /status, /close - Persistent button views for tool approval and plan review - Streaming responses via edit-in-place (1.5s interval) - Stale route detection and atomic session resume via resume_session field - SessionResumedEvent confirmation back to channel - Auto-approve support (blanket + per-tool list) Atomic session resume: - resume_session field on CreateWorkstreamMessage for single-request resume - Server resumes session during POST /v1/api/workstreams/new atomically - Bridge emits SessionResumedEvent to per-workstream channel - WorkstreamCreatedEvent extended with resumed/session_id/message_count - Server UI dashboardResumeSession simplified to single request - Pruned sessions fall back gracefully to fresh start Service auth: - Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET - Bridge: approve scope (1 week). Console collector: read. Proxy: write. Console admin: - Channels tab with per-user view, force-link modal, unlink - 3 admin API endpoints for channel user management - Styled confirm modals replacing browser confirm() dialogs Bug fixes: - AsyncRedisBroker: replaced per-channel listener tasks with single dispatch loop + per-channel queue workers (fixes message stealing race) - Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates - Bridge: _active_sends tracked for initial messages (fixes missing TurnCompleteEvent and unfinalized streaming messages) - Bridge: HTTP calls moved outside lock scope in approval handlers - Bridge: _handle_send cleans up _active_sends on HTTP/server errors - Formatter: reads server SSE format (func_name/preview) with fallback Docs, SDK, tests: - docs/channels.md setup guide, architecture diagram 16 - Updated api-reference.md, architecture.md, console.md, docker.md - Python SDK: resume_session param on create_workstream (async + sync) - TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces - OpenAPI schema: resume_session request, resumed/message_count response - 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing, 18 discord, 12 resume flow) — 1120 total passing * Fix CI lint/typecheck failures and address Copilot review feedback (#24) Lint: fix import ordering, remove unused imports, use contextlib.suppress. Mypy: explicit postgresql dialect import, add discord module overrides for optional-dependency CI environments. Copilot: fix double-escaping in admin confirm modals, return resolved session_id from server resume response, fix channel_routes diagram schema, use atomic setdefault for routing locks, add post-insert race guard in admin channel create, support SSE format in auto-approve check, update identity linking note in architecture diagram. * Fix remaining mypy call-arg errors for discord.py optional dependency Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class definitions that fail when discord.py is not installed in CI. |
||
|
Patrick Buckley
|
047680d669 |
Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23) JWT-based authentication with three token types: config-file (hmac, backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs (HS256, 24h expiry). Username:password login via bcrypt. Hierarchical scopes: read < write < approve. New tables: users (username, password_hash), api_tokens (token_hash, scopes, expires), channel_users (future channel integrations). user_id column added to sessions and workstreams for attribution. Console owns admin CRUD (6 endpoints under /api/admin/). Server validates JWTs locally with shared signing secret. Public /api/auth/setup endpoint for first-time admin creation (atomic, only works with zero users). turnstone-admin CLI for user/token management. Admin console UI: Users and Tokens tabs with full CRUD modals, scope badges, token show-once with clipboard copy, keyboard accessibility (focus traps, Escape, arrow key tabs, ARIA roles). Login UI redesigned: username:password primary, token toggle for legacy, setup wizard auto-detected via /api/auth/status. Python + TypeScript SDKs updated with login(username, password), authStatus(), setup(). New docs/security.md + diagram 15-auth-architecture.puml. All existing docs updated. OpenAPI specs include all new endpoints. 64 new tests (1023 total). Dependencies: PyJWT, bcrypt. * Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review Address Copilot review feedback: escape double quotes in escapeHtml() to prevent XSS in HTML attributes, add JWT validation fallback so config tokens containing dots still work, add user_id to AuthLoginResponse schema, return created field from admin_create_user, and correct five documentation files to match actual API behavior. |
||
|
Patrick Buckley
|
0fd0ad3b2d |
Add structured logging with structlog and context propagation
Replace ad-hoc logging.basicConfig() calls across all 6 entry points with a centralized configure_logging() function backed by structlog. JSON output when stderr is not a TTY (production/Docker), colored console output otherwise. - New turnstone/core/log.py: configure_logging(), get_logger(), contextvars for node_id/ws_id/user_id/request_id auto-injected into every log event - All entry points (server, bridge, console, sim, cli, migrate) call configure_logging() with --log-level and --log-format CLI flags - Server operational print() calls replaced with structured log.info() - LogContextMiddleware sets request_id + ws_id per HTTP request with token-based reset to prevent context leaking across requests - Bridge _run_in_context() helper propagates ctx_node_id to child threads - Env var overrides: TURNSTONE_LOG_LEVEL, TURNSTONE_LOG_FORMAT - 18 new tests (959 total passing) |
||
|
Patrick Buckley
|
a20a058c59 |
Add cluster-scale schema, fix console proxy UX, harden SDK sync runner (#22)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner Schema redesign for multi-node deployments: - New `workstreams` table with node_id, state, lifecycle tracking - Add node_id + ws_id columns to sessions table with indexes - Full UUID (32 hex) for session_id and ws_id (was truncated 12/8) - Server generates and owns node_id, bridge retrieves via /health - Bridge retries with exponential backoff, fatal on auth errors - WorkstreamManager persists workstreams and state changes to storage - /health endpoint exposes node_id for bridge discovery Console proxy UX fixes: - Remove duplicate turnstone branding from proxy banner - Same-tab navigation for Open Node UI and workstream deep links SDK _SyncRunner fix: - Sentinel pattern for StopAsyncIteration across thread boundary Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir). * Address PR #22 review feedback - Fix CLI session_factory signature (ws_id param) — CI typecheck failure - First-phase eviction in create() now calls _cleanup_ui + record_eviction - close() persists "closed" state to storage via update_workstream_state - Fix noqa comment in test to pragma: no cover |
||
|
Patrick Buckley
|
e057c364b8 |
Fix console proxy regressions and add workstream task field (#21) (#21)
* Fix console proxy regressions and add workstream task field (#21) Bug fixes: - Fix collector polling unversioned /api/dashboard (404 after API versioning PR) — nodes showed red/unreachable, no workstreams - Fix SSE proxy dropping all data events — upstream sends \r\n line endings but proxy split on \n\n only; normalize before parsing - Fix workstream state stuck on idle — on_state_change() only broadcasted via SSE but never updated ws.state on the Workstream object; dashboard polling now sees correct attention/running states - Fix deep-link switchTab early return — when ?ws_id matched the only workstream, switchTab bailed (wsId === currentWsId) before establishing SSE connection; inline init instead of delegating - Fix console banner covering dashboard overlay — inject <style> offsetting .dashboard-overlay below the 32px banner Enhancements: - Add turnstone branding to console proxy banner (turnstone │ Console │ node-id) - Add initial_message field to CreateWorkstreamMessage protocol and console "New Workstream" modal (Task textarea, sent as first message) - Refactor SSE proxy to use shared httpx client with 30s read timeout instead of per-request client creation - Increase approval timeout default from 300s to 3600s (1 hour) Updated: Python SDK, TypeScript SDK, OpenAPI specs, MQ client, API schemas, MQ protocol diagram, SDK docs. * Address PR #21 review feedback (4 items) - Log unknown state strings in on_state_change instead of silently swallowing; remove unnecessary KeyError catch - Wrap initial_message POST in _handle_create_ws with error handling so workstream creation success isn't masked by send failure - Strip all \r from SSE chunks instead of replacing \r\n, fixing chunk-boundary split edge case - Add tests for initial_message wiring in directed and pool targeting * Refactor SSE proxy to use httpx-sse aconnect_sse Replace manual SSE chunk buffering/parsing with httpx_sse.aconnect_sse() which handles line endings, event types, and all SSE spec edge cases. Eliminates the \r\n chunk-boundary bug class entirely. Event types are now always forwarded (sse.event defaults to "message" per spec). |
||
|
Patrick Buckley
|
2b58c127b1 |
Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)
* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic migrations, singleton registry. memory.py reduced to thin facade. Session.py open_db() calls replaced with generic KV methods. [database] config section with env var support. Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with postgres extras and migration entrypoint, Helm chart with bitnami subcharts, Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB. 39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated. * Address PR #20 review feedback (16 items) - Backends only call create_all() when Alembic migrations are disabled - Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL constructed via env expansion with secret reference instead of ConfigMap - Migration errors fail fast for PostgreSQL (only non-fatal for SQLite) - save_memory/delete_memory wrapped in exception handling like other facade fns - pool_size passed through from config/env to init_storage() in cli + server - Terraform: DB URL moved to Secrets Manager, auth enabled flag set, optional TLS listeners with certificate_arn, Redis transit encryption on - Docker entrypoint no longer suppresses migration output - Diagram fixes: removed StaticPool claim, removed non-existent migration ref - compose.yaml/README: clarified production profile requires DB env vars |
||
|
Patrick Buckley
|
5ee539c983 |
Add Python and TypeScript client SDKs for server and console APIs (#19)
* Add Python and TypeScript client SDKs for server and console APIs Python SDK (turnstone/sdk/) with sync + async clients for both server and console APIs. Returns Pydantic models directly, streams SSE events as typed dataclasses. 27 event types with registry-based deserialization. High-level send_and_wait() for request-response patterns. TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses fetch + ReadableStream for SSE parsing. Discriminated union event types with type guards. Same API surface as Python SDK. 63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at docs/sdk.md with SDK architecture diagram. * Address PR #19 review feedback + fix lint - Fix consume_task leak in send_and_wait when send() raises (try/finally) - Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout - Add signal param to TS streamSSE for cancellation support - Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF - Fix generate-types.py sys.path (parents[3] not parents[2]) - Document token ignored when httpx_client provided - Document TS timeout units as milliseconds - Fix stale docstring in test_sdk_sse.py - Fix import sorting (ruff I001) |
||
|
Patrick Buckley
|
62a4ceac96 |
Dev/api versioning openapi (#18)
* Add API versioning under /v1/ prefix with OpenAPI 3.1 spec
All API endpoints move to /v1/api/* (clean break, no unversioned
aliases). Non-API routes (/, /health, /metrics, /static, /shared,
/node proxy) stay unversioned.
New turnstone/api/ package:
- Pydantic v2 models for all request/response schemas (server +
console) used for OpenAPI spec generation
- Programmatic OpenAPI 3.1 spec builder with EndpointSpec catalog
- /openapi.json serves machine-readable spec, /docs serves Swagger UI
Route changes:
- Both servers use Mount("/v1", routes=[...API routes...])
- Auth middleware strips /v1/ prefix before path classification
(PUBLIC_PATHS/WRITE_PATHS stay unversioned internally)
- Console proxy handles /node/{id}/v1/api/ upstream forwarding
- Bridge and CLI HTTP clients updated to /v1/api/ paths
- /openapi.json and /docs added to PUBLIC_PATHS and rate limiter
EXEMPT_PATHS
Security fix from review: required_role() now correctly handles
/node/{id}/v1/api/{path} proxy routes (previously the v1 segment
caused write-path detection to fail, allowing read-only token
escalation).
42 new tests (830 total). All frontend JS, docs, and diagrams updated.
* Fix mypy type errors in turnstone/api/ package
- Add generic type params to dict fields in console_schemas.py
- Add return type annotations to docs.py handler factories
- Move type-only imports (BaseModel, Callable, Awaitable) into
TYPE_CHECKING blocks to satisfy TC002/TC003 ruff rules
* Address PR #18 review feedback + fix mypy errors
Review fixes:
- Add pydantic>=2.0 as explicit dependency in pyproject.toml
(was only transitively available via openai/mcp)
- Auto-detect path parameters from {param} segments in OpenAPI
spec builder (fixes missing required path params)
- Use startswith() with concrete prefix for proxy version
detection instead of fragile substring check
- Make Swagger UI base URL configurable via swagger_ui_base_url
parameter for air-gapped deployments
Mypy fixes:
- Add generic type params to dict fields in console_schemas
- Add return type annotations to docs.py handler factories
- Move type-only imports into TYPE_CHECKING blocks
|
||
|
Patrick Buckley
|
29c00c0cdf |
Extract shared frontend design system into turnstone/shared_static/ (#17)
* Extract shared frontend design system into turnstone/shared_static/ The server UI and console UI had ~60% CSS overlap and significant JS duplication. Extract shared assets into a new turnstone/shared_static/ package mounted at /shared/ in both servers: - base.css: design tokens, reset, typography, login/toast/kb overlays, dashboard table, state dots, health bar, scrollbar, reduced motion - auth.js: authFetch, login overlay with focus trap, logout (hooks for page-specific post-login/logout callbacks) - theme.js: dark/light toggle with system preference detection - toast.js: notification queue with configurable timeout - utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount - kb.js: keyboard shortcuts overlay with configurable content, focus management, and focus restore on dismiss Console proxy updated: JS shim injection moved from proxy_static (app.js prepend) to proxy_index (inline <script> in HTML) so it runs before any external scripts. New /shared/ path rewriting and proxy_shared_static route added. ~1540 lines removed from page-specific files, 775 lines in shared package. 13 new tests (788 total). * Fix /shared/ auth and remove __init__.py from shared_static Address PR #17 review feedback: 1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS loads before authentication (required for login overlay to render) 2. Remove turnstone/shared_static/__init__.py to prevent exposing Python package internals (__init__.py, __pycache__) via the StaticFiles mount. Not needed for packaging since pyproject.toml uses explicit glob includes. 3 new auth tests for /shared/ public path access. |
||
|
Patrick Buckley
|
b6e0f0fcca |
Add node version tracking and drift detection to console dashboard (#16)
* Add node version tracking and drift detection to console dashboard Surface the version field from each node's /health endpoint in the console dashboard. Collector extracts version into get_overview() (version_drift + versions fields), promotes it to top-level in get_nodes(), and adds get_version_info() for per-node detail. Console /health endpoint includes drift fields. Frontend adds a VER column to the 7-column node table grid, shows per-node version strings, tracks versions per group with "mixed" + yellow drift badge when nodes disagree, and displays a DRIFT warning or single version in the status bar. Column hidden on mobile (<700px). ARIA labels include version info for accessibility. 10 new tests (745 total). Docs and diagram updated. * Fix drift tooltip text: show 'Versions detected' not 'Nodes running' |
||
|
Patrick Buckley
|
206e37e73e |
Fix circuit breaker, rate limiter, and Anthropic web search correctne… (#15)
* Fix circuit breaker, rate limiter, and Anthropic web search correctness (#15) Three tech debt items addressing correctness and security gaps: Circuit breaker HALF_OPEN single-request permit: - Rename should_allow_request property to acquire_request_permit() method to make the side-effecting, non-idempotent nature explicit - Add _half_open_permit flag: exactly one probe request in HALF_OPEN, subsequent callers blocked until probe completes - Explicitly reset permit on all state transitions (record_success, record_failure) for clean state machine invariants - Session uses BaseException catch to ensure record_failure always fires, preventing permanent circuit deadlock on probe crash Rate limiter X-Forwarded-For support: - Add resolve_client_ip() with rightmost-untrusted XFF parsing - Configurable trusted_proxies via --ratelimit-trusted-proxies CLI flag and [ratelimit] trusted_proxies config (comma-separated CIDRs) - IPv4-mapped IPv6 normalization (::ffff:x.x.x.x → IPv4) for dual-stack - Clientless requests (request.client is None) pass through instead of sharing a single "unknown" bucket - Log warning for invalid CIDR entries in trusted_proxies config - Show trusted proxies in startup log when enabled Anthropic web search multi-turn encrypted content: - Capture raw provider content blocks during streaming via _block_to_dict() using model_dump(exclude_none=True) to avoid Anthropic API rejection - Accumulate thinking_delta into raw_blocks (was silently empty on replay) - Store _provider_content on assistant messages, pass through verbatim in _convert_messages() so encrypted_content/encrypted_index survive turns - Persist to SQLite via new provider_data column (auto-migrated) - Add thinking/signature to _block_to_dict fallback attribute list 23 new tests (735 total), ruff + mypy clean. * Fix Copilot PR #15 review issues: provider data, circuit breaker, IP normalization - Persist assistant message when provider_data exists even if text content is empty — prevents losing Anthropic web search encrypted content needed for multi-turn replay (session.py) - Re-raise KeyboardInterrupt/SystemExit immediately after recording failure instead of attempting fallback models (session.py) - Consume HALF_OPEN permit for the transition caller — prevents two concurrent probe requests when only one should be allowed (healthcheck.py) - Normalize IPv4-mapped IPv6 addresses consistently in resolve_client_ip() — prevents duplicate rate-limit buckets for ::ffff:x.x.x.x vs x.x.x.x (ratelimit.py) |
||
|
Patrick Buckley
|
6c5441435b |
Add console workstream creation + server reverse proxy (#14)
* Add console workstream creation + server reverse proxy (#14) Enable the console dashboard to create workstreams and proxy server UIs, so users only need network access to the console port. Workstream creation via MQ: - POST /api/cluster/workstreams/new with three targeting modes: specific node (directed queue), auto (best node by capacity), or general pool (shared queue, any bridge picks up) - Console pushes CreateWorkstreamMessage to Redis; bridge handles the rest (server creation, ownership registration, SSE events) Reverse proxy for server UIs: - /node/{node_id}/ serves the server's HTML with static path rewriting and a console-return banner injected after <body> - JS proxy shim prepended to app.js overrides fetch() and EventSource() to route root-relative URLs through /node/{id}/api/... - SSE streams proxied via httpx.AsyncClient(timeout=None) with per- connection clients for long-lived streams - GET/POST API requests forwarded with body and auth token Security: - Proxy write paths checked against WRITE_PATHS to prevent read-token escalation (read tokens cannot POST /api/send through proxy) - html.escape() on node_id in banner HTML to prevent XSS - String length limits on name/model inputs Frontend: - "+ new" button in header opens creation modal with node dropdown (Auto / General pool / specific nodes with capacity display) - Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling - Workstream rows and node links deep-link via proxy paths - Custom select arrow, Instrument Panel modal styling Documentation: - docs/console.md rewritten with proxy and creation API docs - docs/architecture.md console section updated - PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered - README.md updated 28 new tests (741 total), ruff + mypy clean. * Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness - Normalize trailing slashes in required_role() to prevent write-role bypass via /api/send/ or /node/{id}/api/send/ (auth.py) - Validate node_id format in proxy handlers (alphanumeric, dot, dash, underscore only) to prevent injection vectors - Use json.dumps() for JS proxy shim prefix to prevent script injection - URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static) - Check upstream status in _proxy_sse() — emit error event on non-200 instead of keeping a dead SSE connection open - Check upstream status in proxy_index() — propagate non-2xx errors - Forward query string in _proxy_post() (consistency with _proxy_get) - Handle JSON null values in create_workstream() — treat null as empty, reject non-string types with 400 - Fix docs/diagram LPUSH → RPUSH to match actual broker implementation |
||
|
Patrick Buckley
|
f02972c11d |
Add provider-native web search with Tavily fallback (#13)
* Add provider-native web search with Tavily fallback Replace client-side Tavily web search with provider-native implementations: - Anthropic: inject web_search_20250305 server-side tool, handle server_tool_use / web_search_tool_result streaming blocks, emit info_delta for search status display - OpenAI: inject web_search_options for gpt-5-search-api, format url_citation annotations as footnote sources - Local/vLLM: preserve existing Tavily-based web_search tool as fallback Add supports_web_search to ModelCapabilities and info_delta to StreamChunk. Remove end-of-life GPT-4o model entries from capability tables. Update docs, diagrams, and README. 88 provider tests (32 new). * Fix Copilot PR #13 review: capture streaming url_citation annotations Accumulate url_citation annotations during OpenAI streaming and emit formatted citations as a final info_delta chunk after the stream ends. Previously annotations were only captured in non-streaming mode, so search model users in the interactive path never saw citation sources. |