1818 Commits

Author SHA1 Message Date
Patrick Buckley de64535221 Feat/eval improvements (#37)
* ci: add GitHub Release creation on tag push

* refactor: rename plan tool to create_plan

Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns

* feat: eval harness improvements inspired by autoresearch patterns

Major enhancements to turnstone-eval:

- Per-test timeout (--test-timeout, default 300s) and suite timeout
  (--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
  models and providers (--optimizer-model, --observer-model, etc.)
  with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
  failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)

* fix: address Copilot review feedback

- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
2026-03-10 13:06:39 -07:00
Patrick Buckley 187d004033 feat: watch tool — periodic command polling within workstreams (#36)
* feat: watch tool — periodic command polling within workstreams

Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.

Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)

* fix: address Copilot review — condition errors, restore deadlock, docs

- Condition eval errors now deactivate the watch immediately instead
  of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
  approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
  detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
2026-03-10 08:18:28 -07:00
Patrick Buckley 7ea150fa71 Bump version to 0.5.2 v0.5.2 2026-03-09 13:42:28 -07:00
Patrick Buckley 4d665a5f62 fix: SSE reconnect loop — remove _sse_generation single-consumer lock
The _sse_generation mechanism assumed one SSE consumer per workstream,
but the bridge also maintains an SSE connection to each workstream.
When a new client connected (browser, proxy, or test), it incremented
the generation counter, killing the bridge's connection. The bridge
reconnected, killing the new client's connection — creating a
mutual-kill cascade that closed every SSE connection after one ping
cycle (5s).

Fix: remove _sse_generation entirely. sse-starlette handles disconnect
detection via its own ASGI task. Also remove the redundant
request.is_disconnected() check which raced with sse-starlette's
disconnect listener in Starlette 0.52.

Root cause confirmed via raw socket test: the server was sending
a zero-length chunked terminator (0\r\n\r\n) at exactly 5s,
cleanly ending the HTTP response body.
2026-03-09 13:40:36 -07:00
Patrick Buckley 3bc3250869 fix: recovered workstreams invisible in console UI (#35)
* fix: recovered workstreams invisible in console UI

Bridge startup recovery (_recover_workstreams) re-registered workstream
ownership but never published WorkstreamCreatedEvent to the cluster
channel. The collector's poll loop would pick up the workstream in its
internal state, but _apply_poll never fanned out SSE events to connected
browsers. Combined, this made channel-resumed workstreams invisible in
the console while remaining accessible through the proxied node UI.

- Bridge: emit WorkstreamCreatedEvent for each recovered workstream
- Collector: diff poll results and fan out synthetic ws_created/ws_closed
  events for workstream additions and removals
- Skip workstreams with empty IDs in poll processing
- Add 4 tests for poll-diff fanout behavior
- Update console data-flow diagram and architecture docs

* fix: address PR review — filter empty ws IDs, stable event ordering

- Filter empty-string keys from old_ids to avoid phantom ws_closed
  events if a previous poll inserted a workstream under key "".
- Sort set diffs before iterating so ws_created/ws_closed fanout
  order is deterministic across poll cycles.
2026-03-09 13:39:47 -07:00
Patrick Buckley db937486cf Bump version to 0.5.1 v0.5.1 2026-03-09 01:48:18 -07:00
Patrick Buckley 554257ac4d fix: SSE proxy Firefox reconnect — Connection: keep-alive header 2026-03-09 01:46:35 -07:00
Patrick Buckley 5f0004dc91 feat: add ClusterSnapshot for instant console UI state rebuild (#34)
* feat: add ClusterSnapshot for instant console UI state rebuild

The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.

Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).

Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.

SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.

* fix: address review feedback and SSE proxy reconnect bug

Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
  both state and listener locks, eliminating the event gap between
  snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
  to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
  console schema models (ClusterNodeInfo, NodeDetailResponse,
  ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
  and ClusterOverviewResponse types instead of Record<string, unknown>.

SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
  when no upstream data arrives, preventing the browser EventSource
  from dropping idle connections. The raw byte passthrough refactor
  (4d11078) removed the proxy's independent keepalive — this restores
  it without reverting to EventSourceResponse.
2026-03-09 01:13:35 -07:00
Patrick Buckley 6cc1b3a5bd feat: add vision/image support to read_file tool (#33)
* feat: add vision/image support to read_file tool

read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO)
and returns base64-encoded content parts for vision-capable models.
Non-vision models receive a text description instead. A new
supports_vision flag on ModelCapabilities gates the feature, with
config.toml [models.*.capabilities] overrides for local models
(vLLM, llama.cpp, NIM).

* fix: address PR review feedback

- Discard _read_files on no-vision OSError path, include exception detail
- Discard _read_files on oversized image error (not a successful read)
- Validate capabilities type from config.toml (reject non-dict)
- Clarify tool description re: vision behavior and offset/limit scope
- Remove unused os import in tests, fix import sort order
- Handle list content (image tool results) in eval.py tool result loop
2026-03-08 23:43:42 -07:00
Patrick Buckley cc9afe94cd get title in collector for console 2026-03-08 22:32:52 -07:00
Patrick Buckley 136b75fdef Bump version to 0.5.0 v0.5.0 2026-03-08 04:47:10 -07:00
Patrick Buckley 4d1107839b refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing

- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service

* Refactor SSE proxy to raw byte passthrough

- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
  httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
  events, and comments now flow through verbatim. Added per-request timeout
  override (read=None, pool=None) for long-lived SSE streams.

- tests/test_console.py: Add 3 new tests for SSE proxy:
  - Ping and event preservation
  - Upstream error status handling
  - Client disconnect handling

- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
  approach.
2026-03-08 04:46:34 -07:00
Patrick Buckley 7d66bc2159 Bump version to 0.4.6 v0.4.6 2026-03-08 03:44:22 -07:00
Patrick Buckley 165cbb2d29 Bump version to 0.4.5 v0.4.5 2026-03-08 03:29:44 -07:00
Patrick Buckley c79c47b940 Add MCP dynamic tool refresh with push notifications and periodic pol… (#31)
* Add MCP dynamic tool refresh with push notifications and periodic polling

MCP tool lists now stay up-to-date without restart via three mechanisms:
push notifications (ToolListChangedNotification) for servers that support
it, staggered periodic polling for servers that don't, and manual
/mcp refresh [server] command. MCPClientManager tracks tools per-server
with copy-on-write rebuild, notifies ChatSession listeners which rebuild
tool lists and ToolSearchManager (preserving expanded tools).

* Address Copilot review feedback on MCP refresh PR

- Fix /mcp refresh typo matching (startswith → exact token check)
- Validate --mcp-refresh-interval >= 0 at parse time via shared
  nonneg_float in config.py (deduplicated from cli.py + server.py)
- Clamp negative refresh_interval to 0 in MCPClientManager constructor
- Fix periodic refresh first poll timing (was initial_delay + interval,
  now initial_delay then immediate first poll)
- Clarify _on_mcp_tools_changed docstring re: O(n) BM25 build cost
2026-03-08 03:28:38 -07:00
Patrick Buckley 660c273e8e remove old demo.svg 2026-03-08 01:47:34 -08:00
Patrick Buckley c7586abd0a Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI

When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:

- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool

New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.

Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.

43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.

* Fix Copilot review feedback on tool search

- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
2026-03-08 01:43:38 -08:00
Patrick Buckley 14d57176ce Bump version to 0.4.4 v0.4.4 2026-03-07 15:43:56 -08:00
Patrick Buckley 96084ca5f3 Fix PostgreSQL migration race condition with advisory lock
Multiple containers starting simultaneously race on Alembic migrations
against shared PostgreSQL. Use pg_advisory_lock so they wait in line.
Also update SQLite bootstrap to detect post-migration databases.
2026-03-07 15:41:29 -08:00
Patrick Buckley 8b92302247 Bump version to 0.4.3 v0.4.3 2026-03-07 13:11:42 -08:00
Patrick Buckley e195ca54a6 new high level arch abstract 2026-03-07 13:01:19 -08:00
Patrick Buckley 50277cd4de Add 10-node cluster profile to Docker Compose 2026-03-07 12:54:41 -08:00
Patrick Buckley fb190f8977 Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity

Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).

Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id

Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days

Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.

* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec

- docs/channels.md: "resumes the session" → "resumes the workstream",
  "Session resumed:" → "Resumed:", "old session was pruned" → "old
  workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
  object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
  models — removes all stale session_id properties from WorkstreamInfo,
  DashboardWorkstream, CreateWorkstreamResponse schemas
2026-03-07 12:49:07 -08:00
Patrick Buckley 25b5e32089 Bump version to 0.4.2 v0.4.2 2026-03-05 20:18:46 -08:00
Patrick Buckley 339981a258 Add GPT-5.3, GPT-5.4, and pro model capabilities (#28)
* Add GPT-5.3, GPT-5.4, and pro model capabilities

Add capability entries for gpt-5-pro (272k output, high-only reasoning),
gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro.

* Validate reasoning_effort against model capabilities

_apply_model_params now falls back to caps.default_reasoning_effort when
the requested value is not in caps.reasoning_effort_values. Prevents
sending unsupported effort levels to models like gpt-5-pro (high only).
2026-03-05 20:17:03 -08:00
Patrick Buckley 06de9ff83b Bump version to 0.4.1 v0.4.1 2026-03-05 18:13:42 -08:00
Patrick Buckley 924b976f1f Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
2026-03-05 18:11:36 -08:00
Patrick Buckley d5db817391 Fix channel gateway Docker networking and console proxy approval scope
Two runtime bugs:

1. Channel gateway advertised http://127.0.0.1:8091 which is
   unreachable from other Docker containers. Add
   TURNSTONE_CHANNEL_ADVERTISE_URL env var override for Docker/K8s
   environments, set to http://channel:8091 in compose.yaml, and
   pass --http-host=0.0.0.0 so the gateway listens on all interfaces.

2. Console proxy service JWT had only "write" scope but the approval
   endpoint requires "approve". Tool approval buttons in the server
   web UI silently failed when accessed through the console proxy.
   Changed proxy token scopes to read+write+approve.
2026-03-05 18:01:52 -08:00
Patrick Buckley fc8ceb4c72 Update README: 3-node cluster diagram, remove directory tree
Replace single-node Mermaid diagram with a 3-node cluster layout
showing bridge+server pairs per node, shared Redis MQ, console, and
channel gateway. Remove the verbose directory tree listing.
2026-03-05 17:44:33 -08:00
Patrick Buckley 07234dec4d Replace ASCII architecture diagram with Mermaid in README
GitHub renders Mermaid natively as an interactive SVG. The new diagram
shows all client entry points (CLI, browser, SDK, Discord), the full
cluster topology including the channel gateway and notify path, and
the LLM provider layer.
2026-03-05 17:34:42 -08:00
Patrick Buckley dd4cc0b30d Add PROGRESS.md and .coverage to .gitignore 2026-03-05 17:31:55 -08:00
Patrick Buckley e7fe8fca9d Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening

Implements the `notify` tool allowing the LLM to send notifications to
Discord channels/users via the channel gateway. Includes fixes for 11
review findings: JWT auth on the gateway endpoint, first-healthy gateway
delivery with retry+backoff, rate limiting only on success, SSRF URL
scheme validation, Discord mention sanitization, SQLite ON CONFLICT
upsert preserving created timestamps, advertise URL resolution for
0.0.0.0 bind, randomized service IDs, generic error messages to prevent
internal state leakage, and partial direct-target validation.

Service registry with heartbeat-based health filtering (migration 005).
Channel gateway registers on startup, heartbeats every 30s, deregisters
on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint
auth (static + JWT), storage CRUD, and retry behavior.

* Add notify documentation, diagrams, and review fixes

Documentation:
- New sequence diagram 17-notify-flow.puml showing end-to-end delivery
- Updated 16-channel-architecture.puml with services table, notify HTTP
  path, and Notification Flow note
- channels.md: Notifications section (targeting, delivery flow, service
  registry, security) and new config table entries
- tools.md: notify tool reference, updated counts/tables (14→15 tools)
- security.md: channel gateway row in service-to-service auth table
- architecture.md: notification subsystem paragraph

Review fixes (copilot):
- _http.py: fail closed when auth unconfigured (401 instead of pass-
  through), strip whitespace on message/title, generic error messages
  for user-not-found vs no-linked-channels
- session.py: parse gateway response JSON and require at least one
  result with status=="sent" before counting as success
- _postgresql.py: use index_elements instead of constraint for upsert
2026-03-05 17:24:00 -08:00
Patrick Buckley 42b9f89988 Add scheduled task system with cron/at scheduling (#26)
* Add scheduled task system with cron/at scheduling, admin API, and console UI

Console-integrated background scheduler dispatches workstreams on recurring
cron expressions or one-shot ISO8601 timestamps. Four target modes: auto
(best node by headroom), pool (shared queue), all (fan-out), or specific
node. Redis distributed lock with unique owner + Lua conditional release
prevents duplicate dispatch in multi-console deployments.

Storage: scheduled_tasks + scheduled_task_runs tables (migration 004),
9 protocol methods on both SQLite and PostgreSQL backends, field allowlist
on updates, run history auto-pruned at 90 days.

API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter
validation, ISO8601 future-time checks, field length bounds, schedule
count cap (200), and OpenAPI spec entries with Pydantic models.

UI: Schedules tab in admin panel with create/edit/delete modals, run
history modal, cron/at type toggle, target mode select, status dots for
accessibility, responsive grid, keyboard navigation, and focus management.

Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches
logged at WARNING with created_by attribution, user_id propagated in
CreateWorkstreamMessage for audit trail.

46 tests across storage, scheduler engine, and API endpoints.

* Add croniter to test extras for CI compatibility

CI installs [test] extras but not [console], so croniter was missing
when schedule API tests import console/server.py validation functions.

* Address Copilot review: timezone validation, focus trap, enabled flag

- Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z)
- UI appends +00:00 to datetime-local values for explicit UTC
- Fix datetime-local normalization: check length before appending seconds
- Add textarea to modal focus trap selector (prevents focus escape)
- Fix _normalize_task_dict not called in update response
- Persist enabled=false on create (storage defaults to enabled=1)
- Validate at_time is still in future when re-enabling a one-shot task
- broker._redis coupling acknowledged as tracked tech debt
2026-03-05 16:05:00 -08:00
Patrick Buckley 77c0a7736b Bump version to 0.4.0 and update security docs
- Version bump in __init__.py, pyproject.toml, api-reference.md
- security.md: document JWT aud/iss claims, login rate limiting,
  secure cookie defaults (24h, Secure flag), CORS restriction,
  service JWT auto-rotation, secret strength validation, and
  proxy auth forwarding via service tokens (not user JWT forwarding)
v0.4.0
2026-03-04 20:45:00 -08:00
Patrick Buckley 872e1770e6 Feature/code dedup (#25)
* Add JWT auth security hardening (6 fixes)

- Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h
- Login brute-force protection: LoginRateLimiter (5 attempts/5min per key)
- JWT aud/iss claims: create_jwt/validate_jwt support audience validation
- Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh
- CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var
- JWT secret strength: warning on secrets shorter than 32 chars
- Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing

* Refactor duplicated code into shared utilities and fix 3 UI bugs

Code deduplication (~235 net lines removed):
- Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py
- Create core/web_helpers.py (require_storage_or_503, read_json_or_400,
  parse_cors_origins, cors_middleware)
- Extract add_redis_args/broker_from_args to mq/broker.py
- Extract add_log_args/configure_logging_from_args to core/log.py
- Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper,
  unused required_role(), duplicate detect_model() wrapper

Bug fixes:
- Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server
  nodes (use ServiceTokenManager with JWT_AUD_SERVER instead)
- Fix login form autofill: wrap inputs in <form>, add name attributes,
  set type=submit on button
- Fix SSE reconnecting flash: add onopen handler to clear status
  immediately on connection (not waiting for first message)
- Fix chat scroll: add min-height:0 to flex containers, overflow:hidden
  on body to constrain viewport height

* Address CI typecheck failure and Copilot review feedback

- Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary)
- Bridge SSE loops: use event_hooks for auth header refresh on reconnect
  instead of static headers that go stale after token rotation
- Login form: remove javascript:void(0) action (CSP anti-pattern)
- Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals
  in middleware builder calls to prevent drift
2026-03-04 20:35:21 -08:00
Patrick Buckley a6e929b0a0 Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24)

Bidirectional channel adapter framework connecting external messaging
platforms to turnstone workstreams via Redis MQ. Discord ships as the
first adapter; the protocol supports future Slack/Teams integrations.

Channel framework:
- ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping
- AsyncRedisBroker with single dispatch loop and per-channel ordered workers
- channel_routes table (migration 003) for persistent route storage
- 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD)
- Unified turnstone-channel gateway entry point, loads adapters by config
- Message chunking, approval formatting, plan review formatting

Discord adapter:
- discord.py v2.4+ bot with thread-per-@mention model
- Slash commands: /link (modal), /unlink, /ask, /status, /close
- Persistent button views for tool approval and plan review
- Streaming responses via edit-in-place (1.5s interval)
- Stale route detection and atomic session resume via resume_session field
- SessionResumedEvent confirmation back to channel
- Auto-approve support (blanket + per-tool list)

Atomic session resume:
- resume_session field on CreateWorkstreamMessage for single-request resume
- Server resumes session during POST /v1/api/workstreams/new atomically
- Bridge emits SessionResumedEvent to per-workstream channel
- WorkstreamCreatedEvent extended with resumed/session_id/message_count
- Server UI dashboardResumeSession simplified to single request
- Pruned sessions fall back gracefully to fresh start

Service auth:
- Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET
- Bridge: approve scope (1 week). Console collector: read. Proxy: write.

Console admin:
- Channels tab with per-user view, force-link modal, unlink
- 3 admin API endpoints for channel user management
- Styled confirm modals replacing browser confirm() dialogs

Bug fixes:
- AsyncRedisBroker: replaced per-channel listener tasks with single
  dispatch loop + per-channel queue workers (fixes message stealing race)
- Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates
- Bridge: _active_sends tracked for initial messages (fixes missing
  TurnCompleteEvent and unfinalized streaming messages)
- Bridge: HTTP calls moved outside lock scope in approval handlers
- Bridge: _handle_send cleans up _active_sends on HTTP/server errors
- Formatter: reads server SSE format (func_name/preview) with fallback

Docs, SDK, tests:
- docs/channels.md setup guide, architecture diagram 16
- Updated api-reference.md, architecture.md, console.md, docker.md
- Python SDK: resume_session param on create_workstream (async + sync)
- TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces
- OpenAPI schema: resume_session request, resumed/message_count response
- 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing,
  18 discord, 12 resume flow) — 1120 total passing

* Fix CI lint/typecheck failures and address Copilot review feedback (#24)

Lint: fix import ordering, remove unused imports, use contextlib.suppress.
Mypy: explicit postgresql dialect import, add discord module overrides for
optional-dependency CI environments.
Copilot: fix double-escaping in admin confirm modals, return resolved
session_id from server resume response, fix channel_routes diagram schema,
use atomic setdefault for routing locks, add post-insert race guard in
admin channel create, support SSE format in auto-approve check, update
identity linking note in architecture diagram.

* Fix remaining mypy call-arg errors for discord.py optional dependency

Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class
definitions that fail when discord.py is not installed in CI.
2026-03-04 13:02:58 -08:00
Patrick Buckley 047680d669 Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23)

JWT-based authentication with three token types: config-file (hmac,
backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs
(HS256, 24h expiry). Username:password login via bcrypt. Hierarchical
scopes: read < write < approve.

New tables: users (username, password_hash), api_tokens (token_hash,
scopes, expires), channel_users (future channel integrations). user_id
column added to sessions and workstreams for attribution.

Console owns admin CRUD (6 endpoints under /api/admin/). Server
validates JWTs locally with shared signing secret. Public /api/auth/setup
endpoint for first-time admin creation (atomic, only works with zero
users). turnstone-admin CLI for user/token management.

Admin console UI: Users and Tokens tabs with full CRUD modals, scope
badges, token show-once with clipboard copy, keyboard accessibility
(focus traps, Escape, arrow key tabs, ARIA roles).

Login UI redesigned: username:password primary, token toggle for legacy,
setup wizard auto-detected via /api/auth/status. Python + TypeScript
SDKs updated with login(username, password), authStatus(), setup().

New docs/security.md + diagram 15-auth-architecture.puml. All existing
docs updated. OpenAPI specs include all new endpoints. 64 new tests
(1023 total). Dependencies: PyJWT, bcrypt.

* Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review

Address Copilot review feedback: escape double quotes in escapeHtml()
to prevent XSS in HTML attributes, add JWT validation fallback so
config tokens containing dots still work, add user_id to
AuthLoginResponse schema, return created field from admin_create_user,
and correct five documentation files to match actual API behavior.
2026-03-04 09:12:18 -08:00
Patrick Buckley 0fd0ad3b2d Add structured logging with structlog and context propagation
Replace ad-hoc logging.basicConfig() calls across all 6 entry points with
a centralized configure_logging() function backed by structlog. JSON output
when stderr is not a TTY (production/Docker), colored console output otherwise.

- New turnstone/core/log.py: configure_logging(), get_logger(), contextvars
  for node_id/ws_id/user_id/request_id auto-injected into every log event
- All entry points (server, bridge, console, sim, cli, migrate) call
  configure_logging() with --log-level and --log-format CLI flags
- Server operational print() calls replaced with structured log.info()
- LogContextMiddleware sets request_id + ws_id per HTTP request with
  token-based reset to prevent context leaking across requests
- Bridge _run_in_context() helper propagates ctx_node_id to child threads
- Env var overrides: TURNSTONE_LOG_LEVEL, TURNSTONE_LOG_FORMAT
- 18 new tests (959 total passing)
2026-03-04 06:47:30 -08:00
Patrick Buckley f3dba836dd Add Git LFS requirement note to README, Diagram PNGs are stored in LFS; git-lfs must be installed for cloning them. 2026-03-04 06:14:27 -08:00
Patrick Buckley 7adda343fc Update docs and diagrams for cluster-scale schema changes
- StorageBackend protocol: document 5 new workstream methods (26 total)
- Session ID: 12-char hex → 32-char full UUID in API reference
- /health endpoint: add node_id field to response docs
- sessions table: document node_id and ws_id columns
- Bridge node_id: document server-owned identity with /health retrieval
- Regenerate storage architecture PNG from updated PlantUML
2026-03-04 06:12:33 -08:00
Patrick Buckley a20a058c59 Add cluster-scale schema, fix console proxy UX, harden SDK sync runner (#22)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner

Schema redesign for multi-node deployments:
- New `workstreams` table with node_id, state, lifecycle tracking
- Add node_id + ws_id columns to sessions table with indexes
- Full UUID (32 hex) for session_id and ws_id (was truncated 12/8)
- Server generates and owns node_id, bridge retrieves via /health
- Bridge retries with exponential backoff, fatal on auth errors
- WorkstreamManager persists workstreams and state changes to storage
- /health endpoint exposes node_id for bridge discovery

Console proxy UX fixes:
- Remove duplicate turnstone branding from proxy banner
- Same-tab navigation for Open Node UI and workstream deep links

SDK _SyncRunner fix:
- Sentinel pattern for StopAsyncIteration across thread boundary

Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir).

* Address PR #22 review feedback

- Fix CLI session_factory signature (ws_id param) — CI typecheck failure
- First-phase eviction in create() now calls _cleanup_ui + record_eviction
- close() persists "closed" state to storage via update_workstream_state
- Fix noqa comment in test to pragma: no cover
2026-03-04 06:04:59 -08:00
Patrick Buckley 498f23c19e Bump version to 0.3.5 v0.3.5 2026-03-04 00:21:28 -08:00
Patrick Buckley e057c364b8 Fix console proxy regressions and add workstream task field (#21) (#21)
* Fix console proxy regressions and add workstream task field (#21)

Bug fixes:
- Fix collector polling unversioned /api/dashboard (404 after API
  versioning PR) — nodes showed red/unreachable, no workstreams
- Fix SSE proxy dropping all data events — upstream sends \r\n line
  endings but proxy split on \n\n only; normalize before parsing
- Fix workstream state stuck on idle — on_state_change() only
  broadcasted via SSE but never updated ws.state on the Workstream
  object; dashboard polling now sees correct attention/running states
- Fix deep-link switchTab early return — when ?ws_id matched the
  only workstream, switchTab bailed (wsId === currentWsId) before
  establishing SSE connection; inline init instead of delegating
- Fix console banner covering dashboard overlay — inject <style>
  offsetting .dashboard-overlay below the 32px banner

Enhancements:
- Add turnstone branding to console proxy banner (turnstone │ Console │ node-id)
- Add initial_message field to CreateWorkstreamMessage protocol and
  console "New Workstream" modal (Task textarea, sent as first message)
- Refactor SSE proxy to use shared httpx client with 30s read timeout
  instead of per-request client creation
- Increase approval timeout default from 300s to 3600s (1 hour)

Updated: Python SDK, TypeScript SDK, OpenAPI specs, MQ client,
API schemas, MQ protocol diagram, SDK docs.

* Address PR #21 review feedback (4 items)

- Log unknown state strings in on_state_change instead of silently
  swallowing; remove unnecessary KeyError catch
- Wrap initial_message POST in _handle_create_ws with error handling
  so workstream creation success isn't masked by send failure
- Strip all \r from SSE chunks instead of replacing \r\n, fixing
  chunk-boundary split edge case
- Add tests for initial_message wiring in directed and pool targeting

* Refactor SSE proxy to use httpx-sse aconnect_sse

Replace manual SSE chunk buffering/parsing with httpx_sse.aconnect_sse()
which handles line endings, event types, and all SSE spec edge cases.
Eliminates the \r\n chunk-boundary bug class entirely. Event types are
now always forwarded (sse.event defaults to "message" per spec).
2026-03-04 00:18:48 -08:00
Patrick Buckley e785a94539 Fix Alembic migration auth failure with PostgreSQL
str(engine.url) in SQLAlchemy 2.x masks the password as '***',
causing SCRAM-SHA-256 authentication to fail when Alembic creates
its own engine from the config URL.
2026-03-03 23:09:31 -08:00
Patrick Buckley 2b58c127b1 Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)
* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging

Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core
schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic
migrations, singleton registry. memory.py reduced to thin facade. Session.py
open_db() calls replaced with generic KV methods. [database] config section
with env var support.

Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with
postgres extras and migration entrypoint, Helm chart with bitnami subcharts,
Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB.

39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated.

* Address PR #20 review feedback (16 items)

- Backends only call create_all() when Alembic migrations are disabled
- Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL
  constructed via env expansion with secret reference instead of ConfigMap
- Migration errors fail fast for PostgreSQL (only non-fatal for SQLite)
- save_memory/delete_memory wrapped in exception handling like other facade fns
- pool_size passed through from config/env to init_storage() in cli + server
- Terraform: DB URL moved to Secrets Manager, auth enabled flag set,
  optional TLS listeners with certificate_arn, Redis transit encryption on
- Docker entrypoint no longer suppresses migration output
- Diagram fixes: removed StaticPool claim, removed non-existent migration ref
- compose.yaml/README: clarified production profile requires DB env vars
2026-03-03 22:57:34 -08:00
Patrick Buckley 5ee539c983 Add Python and TypeScript client SDKs for server and console APIs (#19)
* Add Python and TypeScript client SDKs for server and console APIs

Python SDK (turnstone/sdk/) with sync + async clients for both server
and console APIs. Returns Pydantic models directly, streams SSE events
as typed dataclasses. 27 event types with registry-based deserialization.
High-level send_and_wait() for request-response patterns.

TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses
fetch + ReadableStream for SSE parsing. Discriminated union event types
with type guards. Same API surface as Python SDK.

63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at
docs/sdk.md with SDK architecture diagram.

* Address PR #19 review feedback + fix lint

- Fix consume_task leak in send_and_wait when send() raises (try/finally)
- Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout
- Add signal param to TS streamSSE for cancellation support
- Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF
- Fix generate-types.py sys.path (parents[3] not parents[2])
- Document token ignored when httpx_client provided
- Document TS timeout units as milliseconds
- Fix stale docstring in test_sdk_sse.py
- Fix import sorting (ruff I001)
2026-03-03 21:22:24 -08:00
Patrick Buckley 62a4ceac96 Dev/api versioning openapi (#18)
* Add API versioning under /v1/ prefix with OpenAPI 3.1 spec

All API endpoints move to /v1/api/* (clean break, no unversioned
aliases). Non-API routes (/, /health, /metrics, /static, /shared,
/node proxy) stay unversioned.

New turnstone/api/ package:
- Pydantic v2 models for all request/response schemas (server +
  console) used for OpenAPI spec generation
- Programmatic OpenAPI 3.1 spec builder with EndpointSpec catalog
- /openapi.json serves machine-readable spec, /docs serves Swagger UI

Route changes:
- Both servers use Mount("/v1", routes=[...API routes...])
- Auth middleware strips /v1/ prefix before path classification
  (PUBLIC_PATHS/WRITE_PATHS stay unversioned internally)
- Console proxy handles /node/{id}/v1/api/ upstream forwarding
- Bridge and CLI HTTP clients updated to /v1/api/ paths
- /openapi.json and /docs added to PUBLIC_PATHS and rate limiter
  EXEMPT_PATHS

Security fix from review: required_role() now correctly handles
/node/{id}/v1/api/{path} proxy routes (previously the v1 segment
caused write-path detection to fail, allowing read-only token
escalation).

42 new tests (830 total). All frontend JS, docs, and diagrams updated.

* Fix mypy type errors in turnstone/api/ package

- Add generic type params to dict fields in console_schemas.py
- Add return type annotations to docs.py handler factories
- Move type-only imports (BaseModel, Callable, Awaitable) into
  TYPE_CHECKING blocks to satisfy TC002/TC003 ruff rules

* Address PR #18 review feedback + fix mypy errors

Review fixes:
- Add pydantic>=2.0 as explicit dependency in pyproject.toml
  (was only transitively available via openai/mcp)
- Auto-detect path parameters from {param} segments in OpenAPI
  spec builder (fixes missing required path params)
- Use startswith() with concrete prefix for proxy version
  detection instead of fragile substring check
- Make Swagger UI base URL configurable via swagger_ui_base_url
  parameter for air-gapped deployments

Mypy fixes:
- Add generic type params to dict fields in console_schemas
- Add return type annotations to docs.py handler factories
- Move type-only imports into TYPE_CHECKING blocks
2026-03-03 20:28:49 -08:00
Patrick Buckley 29c00c0cdf Extract shared frontend design system into turnstone/shared_static/ (#17)
* Extract shared frontend design system into turnstone/shared_static/

The server UI and console UI had ~60% CSS overlap and significant JS
duplication. Extract shared assets into a new turnstone/shared_static/
package mounted at /shared/ in both servers:

- base.css: design tokens, reset, typography, login/toast/kb overlays,
  dashboard table, state dots, health bar, scrollbar, reduced motion
- auth.js: authFetch, login overlay with focus trap, logout (hooks for
  page-specific post-login/logout callbacks)
- theme.js: dark/light toggle with system preference detection
- toast.js: notification queue with configurable timeout
- utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount
- kb.js: keyboard shortcuts overlay with configurable content, focus
  management, and focus restore on dismiss

Console proxy updated: JS shim injection moved from proxy_static (app.js
prepend) to proxy_index (inline <script> in HTML) so it runs before any
external scripts. New /shared/ path rewriting and proxy_shared_static
route added. ~1540 lines removed from page-specific files, 775 lines in
shared package. 13 new tests (788 total).

* Fix /shared/ auth and remove __init__.py from shared_static

Address PR #17 review feedback:

1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS
   loads before authentication (required for login overlay to render)

2. Remove turnstone/shared_static/__init__.py to prevent exposing
   Python package internals (__init__.py, __pycache__) via the
   StaticFiles mount. Not needed for packaging since pyproject.toml
   uses explicit glob includes.

3 new auth tests for /shared/ public path access.
2026-03-03 20:11:49 -08:00
Patrick Buckley b6e0f0fcca Add node version tracking and drift detection to console dashboard (#16)
* Add node version tracking and drift detection to console dashboard

Surface the version field from each node's /health endpoint in the
console dashboard. Collector extracts version into get_overview()
(version_drift + versions fields), promotes it to top-level in
get_nodes(), and adds get_version_info() for per-node detail. Console
/health endpoint includes drift fields.

Frontend adds a VER column to the 7-column node table grid, shows
per-node version strings, tracks versions per group with "mixed" +
yellow drift badge when nodes disagree, and displays a DRIFT warning
or single version in the status bar. Column hidden on mobile (<700px).
ARIA labels include version info for accessibility.

10 new tests (745 total). Docs and diagram updated.

* Fix drift tooltip text: show 'Versions detected' not 'Nodes running'
2026-03-03 18:57:30 -08:00
Patrick Buckley 206e37e73e Fix circuit breaker, rate limiter, and Anthropic web search correctne… (#15)
* Fix circuit breaker, rate limiter, and Anthropic web search correctness (#15)

Three tech debt items addressing correctness and security gaps:

Circuit breaker HALF_OPEN single-request permit:
- Rename should_allow_request property to acquire_request_permit() method
  to make the side-effecting, non-idempotent nature explicit
- Add _half_open_permit flag: exactly one probe request in HALF_OPEN,
  subsequent callers blocked until probe completes
- Explicitly reset permit on all state transitions (record_success,
  record_failure) for clean state machine invariants
- Session uses BaseException catch to ensure record_failure always fires,
  preventing permanent circuit deadlock on probe crash

Rate limiter X-Forwarded-For support:
- Add resolve_client_ip() with rightmost-untrusted XFF parsing
- Configurable trusted_proxies via --ratelimit-trusted-proxies CLI flag
  and [ratelimit] trusted_proxies config (comma-separated CIDRs)
- IPv4-mapped IPv6 normalization (::ffff:x.x.x.x → IPv4) for dual-stack
- Clientless requests (request.client is None) pass through instead of
  sharing a single "unknown" bucket
- Log warning for invalid CIDR entries in trusted_proxies config
- Show trusted proxies in startup log when enabled

Anthropic web search multi-turn encrypted content:
- Capture raw provider content blocks during streaming via _block_to_dict()
  using model_dump(exclude_none=True) to avoid Anthropic API rejection
- Accumulate thinking_delta into raw_blocks (was silently empty on replay)
- Store _provider_content on assistant messages, pass through verbatim in
  _convert_messages() so encrypted_content/encrypted_index survive turns
- Persist to SQLite via new provider_data column (auto-migrated)
- Add thinking/signature to _block_to_dict fallback attribute list

23 new tests (735 total), ruff + mypy clean.

* Fix Copilot PR #15 review issues: provider data, circuit breaker, IP normalization

- Persist assistant message when provider_data exists even if text
  content is empty — prevents losing Anthropic web search encrypted
  content needed for multi-turn replay (session.py)
- Re-raise KeyboardInterrupt/SystemExit immediately after recording
  failure instead of attempting fallback models (session.py)
- Consume HALF_OPEN permit for the transition caller — prevents two
  concurrent probe requests when only one should be allowed
  (healthcheck.py)
- Normalize IPv4-mapped IPv6 addresses consistently in
  resolve_client_ip() — prevents duplicate rate-limit buckets for
  ::ffff:x.x.x.x vs x.x.x.x (ratelimit.py)
2026-03-03 18:28:39 -08:00