Compare commits

..

66 Commits

Author SHA1 Message Date
Patrick Buckley f1f448277f chore: bump version to 0.5.6
Prompt template runtime wiring, security hardening, scheduler/channel/MQ
template support, migration 010.
2026-03-12 16:58:20 -07:00
Patrick Buckley 2f7f70825b feat: wire prompt templates into session startup with full creation-p… (#47)
* feat: wire prompt templates into session startup with full creation-path support

Prompt templates (prompt_templates table) now have runtime effect:

- is_default=true templates auto-apply as system message content,
  concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
  field on POST /v1/api/workstreams/new, console creation modal dropdown,
  scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
  regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession

Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
  server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion

Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.

Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.

Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.

Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.

* fix: address PR #47 review feedback

- Defer template validation until after resume_ws — a bad template name
  no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
  openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
  reject unknown template names with 400 instead of allowing schedules
  that would silently fail at dispatch time
2026-03-12 16:57:31 -07:00
Patrick Buckley 4866c9873c feat: ddgCluster compose profile with DuckDuckGo Search MCP sidecar (#48)
Add ddgCluster profile extending the 10-node cluster with a DuckDuckGo
Search MCP sidecar. All cluster nodes connect via streamable-http and
gain duckduckgo_web_search + duckduckgo_fetch_content tools. No API
key required.

Key implementation details learned during testing:
- MCP SDK DNS rebinding protection must be disabled for Docker
  internal networking (Host header uses container names)
- FastMCP server binds to 127.0.0.1 by default; must set
  mcp.settings.host='0.0.0.0' for cross-container access
- DDG CLI lacks --host/--port flags; settings configured via Python
  entry point that patches FastMCP.settings directly
- Safe search disabled by default

Also adds MCP_CONFIG env var support to all server commands (shell
conditional, no-op when empty) and moves default server/bridge to
production profile for cleaner profile separation.
2026-03-12 16:51:53 -07:00
Patrick Buckley 8b2e2130fc fix: MCP resource template URI expansion via prefix matching (#46)
* fix: MCP resource template URI expansion via prefix matching

Resource templates (RFC 6570 URI patterns like `db://tables/{table}/rows/{id}`)
were discovered from MCP servers but non-functional — `read_resource_sync()`
only accepted exact URIs from `_resource_map`, which excludes templates.

Add prefix-based fallback: extract the static prefix from each template
(everything before the first `{`), store a prefix→server mapping, and
fall back to longest-prefix matching when exact URI lookup fails. MCP
servers handle URI routing internally so we just need to route the
expanded URI to the correct server.

Also surface templates in the system message catalog and `/mcp` command
so the model knows they exist and can construct expanded URIs.

* fix: address PR #46 review feedback

- Template prefix collision now keeps more specific (longer) template
  URI instead of blindly overriding
- Fix _match_template docstring to accurately describe startswith
  matching on static prefixes (not full template matching)
- Add missing loop.close() in integration test finally block
- Rewrite test_template_longest_prefix_wins with genuinely different
  prefix lengths to avoid brittle collision-order dependency
2026-03-12 15:46:00 -07:00
Patrick Buckley f81c06761d chore: remove dead code, add MCP integration + collector tests (#45)
* chore: remove dead code, add MCP integration + collector tests

Remove unused delete_prompt_templates_by_server from protocol and
both storage backends (sync uses per-template deletion).

Add 10 MCP integration tests exercising full lifecycle: rebuild
resources/prompts, read_resource_sync/get_prompt_sync with real
asyncio loop, governance sync to real SQLite, shutdown cleanup,
listener notification isolation.

Add 3 console collector MCP aggregation tests: multi-node sums,
absent when zero, mixed nodes with/without MCP.

* fix: close event loops and SQLite backend in MCP integration tests
2026-03-12 15:16:51 -07:00
Patrick Buckley be165c1971 feat: MCP resource and prompt discovery with read_resource tool (#44)
* feat: MCP resource and prompt discovery with read_resource tool

Extends MCPClientManager with resource and prompt discovery alongside
existing tool support. Resources and prompts are discovered on connect,
cached per-server with copy-on-write rebuilds, and refreshed via push
notifications, periodic polling, or manual /mcp refresh.

New read_resource built-in tool reads MCP resources by URI. Requires
user approval (same as MCP tool calls) since resources are served by
external MCP servers. Resource catalog injected into system message
with XML delimiters. Error messages sanitized to prevent leaking
server internals to the model.

Prompt discovery stores prefixed names (mcp__server__prompt) and
exposes get_prompt_sync() for future use_prompt tool (Chunk D).

/mcp command now shows tools, resources, and prompts. Docs and
diagrams updated.

* feat: MCP prompt governance sync with origin tracking and readonly guards

Migration 009 adds origin, mcp_server, and readonly columns to
prompt_templates. MCP prompts discovered by MCPClientManager are
automatically synced into the governance table as read-only templates
with origin="mcp".

Sync engine handles: create on connect, update on prompt refresh,
delete when prompts are removed from server. Manual templates take
precedence on name collision (MCP prompt skipped with warning).

Admin API returns 403 on update/delete of readonly templates. Console
UI shows MCP origin badge and disables edit/delete buttons. Storage
backends gain get_prompt_template_by_name, list_prompt_templates_by_origin,
and delete_prompt_templates_by_server methods.

Also addresses PR #44 review feedback: concurrent.futures.TimeoutError
handling in sync dispatch, XML-escape resource catalog descriptions,
resource template entries excluded from _resource_map, URI collision
warnings, needs_periodic capability-aware computation, malformed JSON
primary key fallback for read_resource.

* feat: use_prompt tool, prompt catalog, and PR review hardening

New use_prompt built-in tool invokes MCP prompt templates by name,
expanding them into messages. Requires user approval (external MCP
servers). Prompt catalog injected into system message with XML
delimiters (up to 30 prompts, HTML-escaped).

Prompt listener registered in session for catalog rebuild on changes.

Addresses PR #44 review feedback:
- _init_system_messages() now uses copy-on-write (build locally,
  assign atomically) so background thread callbacks never see
  partial system messages
- sync_prompts_to_storage() serialized behind _sync_lock to prevent
  races between set_storage() (main thread) and MCP background thread
- shutdown() clears listener lists to release callback references

Docs and diagrams updated for 18 built-in tools.

* feat: granular tool policies for MCP resources, prompts, and tools

Policy evaluation now uses approval_label (falling back to func_name)
for fnmatch pattern matching, enabling fine-grained per-URI and
per-server policies:
- read_resource: mcp_resource__{normalized_uri}
- use_prompt: mcp__{server}__{prompt} (prefixed name)
- MCP tools: mcp__{server}__{tool} (was static "mcp_tool")

URI normalization resolves .. path segments to prevent traversal
bypasses in policy matching. Resource templates filtered from system
message catalog (not directly readable). use_prompt arguments
validated as dict with string coercion.

TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly
fields. Governance docs updated with MCP policy patterns.

* feat: MCP visibility in server and console UIs

Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts
counts. Server UI status bar shows magenta MCP indicator with tooltip.
Console cluster status bar shows MCP metrics with magenta LED dot.
Console node detail view shows per-node MCP summary. Console collector
aggregates MCP counts across nodes in overview.

Uses var(--magenta) design token with new --magenta-glow for theme
adaptation. ARIA roles on MCP status elements. Tooltips on console
MCP metric labels. Node MCP summary hidden on mobile (< 700px).

New diagram: 20-mcp-architecture.puml covering full MCP lifecycle
(connection, discovery, refresh, governance sync, policy, UI).

* fix: McpStatus in health schema, count properties, catalog name fidelity

Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so
typed clients see the mcp field from /health.

Addresses Copilot review feedback:
- resource_count/prompt_count properties avoid list allocation on
  /health and /metrics polls
- get_tools/resources/prompts return shallow-copied dicts to prevent
  callers from mutating internal cache
- Prompt names and arg names in system message catalog are NOT
  HTML-escaped (model must use exact strings in use_prompt calls);
  only descriptions are escaped

* fix: OpenAPI spec McpStatus + diagram approval column accuracy

Adds McpStatus schema and optional mcp field to HealthResponse in
openapi-server.json, matching the Python schema and TypeScript types.

Fixes tool pipeline diagram: math, web_fetch, web_search correctly
shown as auto-approve (not "Yes" for approval).
2026-03-12 14:49:58 -07:00
Patrick Buckley 3264fdefca fix: channel bidirectional routing — emit TurnCompleteEvent on all id… (#43)
* fix: channel bidirectional routing — emit TurnCompleteEvent on all idle transitions

Bridge previously only emitted TurnCompleteEvent for MQ-initiated turns
(those with a correlation_id in _active_sends). Server-UI-initiated turns
went idle without emitting TurnCompleteEvent, so the Discord bot's
StreamingMessage never finalized — content accumulated in the buffer and
collided with the next Discord-triggered response.

Now TurnCompleteEvent is emitted unconditionally on every idle transition.
correlation_id is empty for non-MQ turns; SDK client filters by
correlation_id so existing consumers are unaffected.

* fix: remove unused variable flagged by ruff
2026-03-12 11:57:15 -07:00
Patrick Buckley 28cb3a5c51 fix: approval timeout UI state and content flush before tool calls (#42)
* fix: approval timeout UI state and content flush before tool calls

Two bug fixes:

1. Approval timeout now shows denied state in UI — resolve_approval()
   emits an approval_resolved SSE event so the browser transitions
   from pending to denied (red border + badge). Also fixes the cancel-
   during-approval path. Frontend resolveInlineApproval() gains a
   skipPost parameter to avoid redundant POST when server-initiated.
   ApprovalResolvedEvent added to Python and TypeScript SDKs.

2. Content streaming flushes pending buffer before tool call deltas —
   _stream_response() held up to 13 trailing chars in the pending
   buffer (for <think> tag detection) when transitioning to tool calls.
   Now flushed eagerly when tool_call_deltas arrive, before clearing
   in_think so reasoning text is correctly categorized.

* fix: address Copilot review feedback on PR #42

Patch _execute_tools in stream flush test to prevent real bash execution,
simplify confusing nested comprehension, and update resolve_approval()
docstring to reflect cancel/timeout call paths.
2026-03-12 11:43:06 -07:00
Patrick Buckley 8b11e0a6f9 fix: bridge retries node_id fetch indefinitely with capped backoff 2026-03-12 11:12:03 -07:00
Patrick Buckley 648ba477e1 refactor: list_user_roles uses _row_to_dict instead of positional row mapping 2026-03-11 21:14:58 -07:00
Patrick Buckley 7960784786 fix: usage events now record per-request tool_calls delta, not cumulative total 2026-03-11 21:12:03 -07:00
Patrick Buckley e06554d1ec feat: add channel admin endpoints to console OpenAPI spec 2026-03-11 21:08:28 -07:00
Patrick Buckley 8eb8722346 Bump version to 0.5.5 2026-03-11 20:24:12 -07:00
Patrick Buckley a2e2ffacd8 feat: robust plan quality gate, iterative refinement, and amend UX (#41)
* feat: robust plan quality gate, iterative refinement, and amend UX

Plan agent output from weak models often produced garbage (11-char plans
that echo the prompt). Two fixes:

1. Quality validation (_validate_plan) checks length, section structure,
   echo detection, and refusal patterns. Fails trigger one automatic
   retry with a coaching message injected into the agent's existing
   conversation, preserving all prior exploration context.

2. Iterative feedback loop — user feedback at plan review re-runs the
   plan agent via _refine_plan() instead of appending text to the tool
   result. Up to 5 refinement rounds. The plan file path is always
   included in the tool result so the outer model knows where it lives.

UI improvements:
- Web: Reject button dynamically becomes "Amend" (amber) when feedback
  is typed. Key hint badges (Esc/Enter) on plan buttons. Main input
  disabled during review. Light-theme contrast fix via --on-color var.
- CLI: Prompt shows all three actions (approve/amend/reject).
- Bridge: Race condition fix — clear pending entry before HTTP POST so
  sequential plan reviews from the refinement loop aren't skipped.

15 new tests covering validation, retry, and refinement.

* fix: address PR 41 review feedback

- Escape key in plan dialog now mirrors the Amend button: if feedback is
  typed, Esc sends the feedback (amend); if empty, Esc rejects. Previously
  Esc always hard-coded "reject", discarding typed feedback.

- Coaching message for plan retry now says "should include at least two of"
  instead of "MUST include these", matching the actual validation rule
  (_MIN_PLAN_SECTIONS = 2).

* feat: render plan inline in chat after approval

After the plan review dialog closes, the plan content is now rendered
as a collapsible inline block in the chat stream — styled with a
status header (approved/rejected/amending), markdown-rendered body,
and feedback note when amending. Uses the same makeCollapsible pattern
as tool output blocks.

* fix: prevent plan approval hang when inline render fails

The authFetch call that unblocks the server must fire before the
cosmetic inline plan rendering. Previously _addInlinePlan ran first
and any JS error (e.g. from renderMarkdown) prevented the API call,
leaving the session thread blocked forever.

- Move authFetch before _addInlinePlan
- Wrap _addInlinePlan in try-catch
- Guard against empty content
- Only auto-collapse plans longer than 12 lines

* fix: address PR 41 review feedback (round 2)

- Max refinement rounds no longer implicitly approve: the loop now
  shows the final plan for explicit approve/reject before proceeding.
  Previously exhausting 5 rounds silently accepted the last revision.

- Plan inline block: correct aria-label from "Tool output" to
  "Plan content" when makeCollapsible is applied.

- XSS concern (not applicable): renderMarkdown is used for all
  assistant messages — plan content follows the same trust model.

- Test loop concern (acknowledged): refinement tests verify component
  logic; full _execute_tools integration would require extensive
  mocking for marginal coverage gain.

* feat: thinking spinner + inline plan hardening

* fix lint
2026-03-11 20:22:41 -07:00
Patrick Buckley c6ba8d59b0 feat: bootstrap wizard — LLM-guided interactive setup for deployments
Add `turnstone-bootstrap`, a new entry point that uses any LLM (OpenAI,
Anthropic, or local/vLLM) to conversationally walk users through
configuring a Turnstone deployment. Generates .env files, setup.sh
scripts, and optional docker-compose overrides.

- Fully interactive startup (zero CLI args) with provider/model selection
- Auto-detects available models on local OpenAI-compatible endpoints
- 7 tools: read_file, write_file, generate_secret, check_port,
  validate_api_key, check_docker, finish
- Path traversal protection on file read/write
- Duplicate write detection (skips identical content)
- Bounded retry loop (3 attempts) on LLM errors
- Anthropic message conversion with consecutive-role merging
2026-03-11 01:58:39 -07:00
Patrick Buckley 087f5b49f6 Bump version to 0.5.4 2026-03-10 20:47:40 -07:00
Patrick Buckley fd507c6a3c feat: generation cancellation — stop button, cancel API, cooperative … (#40)
* feat: generation cancellation — stop button, cancel API, cooperative cancel

Add cooperative cancellation via threading.Event on ChatSession. The cancel
signal is set from outside the worker thread (HTTP handler, MQ bridge, or
Escape key) and checked at defined checkpoints: per streaming chunk, before
tool execution, inside bash commands, and at each sub-agent turn.

Core: GenerationCancelled(BaseException) exception, cancel()/_check_cancelled()
methods, partial content preservation in _stream_response, clean rollback in
send() with idle state emission (no re-raise).

Server: POST /v1/api/cancel endpoint, CancelledEvent SSE emission, worker
thread safety net.

Frontend: Stop button (■ Stop) with send/stop swap via setBusy(), Escape key
shortcut, cancelled event handler. Accessible: aria-label, focus-visible
override, light theme contrast, non-color differentiation.

MQ: CancelMessage inbound type, bridge _handle_cancel routed handler.

SDK: cancel() on Python async+sync clients, CancelledEvent in Python+TypeScript
event registries, isCancelledEvent type guard.

OpenAPI: CancelRequest schema + endpoint spec.

Docs: API reference, architecture, SDK docs updated. Diagrams: conversation
turn, tool pipeline, MQ protocol, workstream states, SDK architecture.

* fix: address PR #40 review feedback

- setBusy() now resets stopBtn.disabled so stop button is re-enabled on
  next generation after a successful cancel
- Gate cancel side effects (resolve_approval, resolve_plan, cancelled SSE
  event) on worker_thread.is_alive() to avoid spurious events when idle
- Add /v1/api/cancel endpoint and CancelRequest schema to TypeScript
  openapi-server.json to keep it in sync with Python-generated spec
2026-03-10 20:43:52 -07:00
Patrick Buckley 562c3c8ab7 docs: add governance section and missing diagrams to README 2026-03-10 19:50:07 -07:00
Patrick Buckley 4773535bb8 docs: add governance architecture diagram PNG 2026-03-10 19:41:35 -07:00
Patrick Buckley 7492816ab2 feat: governance — RBAC, tool policies, prompt templates, usage track… (#39)
* feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging

Add comprehensive governance layer for the admin console:

- RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer),
  custom role CRUD, user-role assignment with privilege escalation prevention
- Tool policies with glob pattern matching, priority-ordered evaluation
  (allow/deny/ask), enforced before auto-approve in WebUI.approve_tools()
- Prompt templates with variable substitution, categories, default flag
- Usage tracking: per-LLM-request token/tool metrics, aggregated queries
  (group by day/model/user), automatic 90-day pruning via scheduler
- Audit logging: append-only event trail for all admin mutations,
  filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For
  aware IP extraction
- require_permission() enforced on all 35+ admin endpoints (users, tokens,
  channels, schedules, watches, roles, orgs, policies, templates, usage, audit)
- Field allowlists on storage update methods prevent mass-assignment bugs
- Self-deletion guard on admin_delete_user, delete_user cascades user_roles
- _row_to_dict helper eliminates ~400 lines of fragile positional row mapping
- _audit_context helper deduplicates 18 instances of audit boilerplate
- Migration 008: 7 new tables, 3 builtin roles, org_id on users
- Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit)
  with permission-gated visibility, 7 modal dialogs, full keyboard accessibility
- Python + TypeScript SDK methods for all governance endpoints
- 120+ new tests (1554 total)

* fix: address PR #39 review feedback

- Rebuild serialized items after policy evaluation so denied/allowed
  verdicts are reflected in tool_info/approve_request SSE payloads
- Make `since` query param optional in usage OpenAPI spec (handler
  already defaults to last 7 days)
- Add response_model=StatusResponse to DELETE role/policy/template
  and POST/DELETE role assignment endpoints in OpenAPI spec
- Add missing org_id/created/updated fields to UserRoleInfo schema
- Add missing created field to AuditEventInfo schema
- Show "no permissions" empty state instead of loading inaccessible
  tab when all admin tabs are permission-gated
- Fix "13 permissions" → "15 permissions" in architecture.md and
  security.md
- Fix import sorting in test_audit.py and test_tool_policy.py

* fix: address PR #39 round 2 review feedback

- Clear stale permissions from sessionStorage on config-token login
  (auth.js _storePermissions)
- Only trust X-Forwarded-For when behind a proxy that sets
  X-Forwarded-Proto (conditional on is_secure_request trust model)
- Thread user_id from auth into WebUI.on_status for usage events
- Add created field to TS AuditEventInfo type
- Return typed Pydantic models from all SDK governance methods instead
  of dict[str, Any] — both async and sync clients
- Validate group_by param against allowed enum in admin_usage handler
- Add deterministic secondary sort (event_id DESC) to
  list_audit_events in both SQLite and PostgreSQL backends
2026-03-10 19:32:37 -07:00
Patrick Buckley d6ba1d5e25 fix: mypy no-any-return in agent context overflow handler 2026-03-10 14:11:09 -07:00
Patrick Buckley 41d1b27d34 Bump version to 0.5.3 2026-03-10 13:46:50 -07:00
Patrick Buckley 8bc284c60e fix: agent context overflow — truncate tool output, catch context errors
Agent tool outputs are now truncated to 16k chars to prevent search
results (14M+ chars observed) from blowing past the model's context
limit. On context-exceeded API errors, the agent returns its last
content instead of crashing.
2026-03-10 13:43:01 -07:00
Patrick Buckley a322d6b1d1 fix: sub-agent context — clean plan, merged task, no Qwen template error
Plan agent: own identity only (no base system prompt needed).
Task agent: base system prompt merged with task identity into a single
system message (needs tool patterns for tool execution).
Neither agent receives conversation history.

Fixes Jinja template error on Qwen models that reject system messages
appearing after non-system messages.
2026-03-10 13:36:23 -07:00
Patrick Buckley 70d495aa5b fix: per-workstream SSE fan-out — multiple consumers no longer steal … (#38)
* fix: per-workstream SSE fan-out — multiple consumers no longer steal each other's tokens

After 4d665a5 removed the single-consumer SSE lock, the shared
_event_queue let concurrent consumers (browser, bridge, console proxy)
race on Queue.get(), each receiving ~1/N of content tokens and producing
garbled streaming text.

Replace the single queue with per-client fan-out: each SSE connection
registers its own bounded queue (maxsize=500) on WebUI._listeners, and
_enqueue() copies every event to all registered queues. On eviction or
close, a ws_closed sentinel is injected so SSE generators exit promptly.

* fix: address CI failures and Copilot review feedback

- Handle ws_closed sentinel in events_sse generator (break on close)
- Guarantee sentinel delivery by evicting one item when queue is full
- Clear listeners list after injecting sentinels on cleanup
- Fix test_slow_consumer to fill only slow queue directly
- Fix ruff SIM117 (nested with), unused import, mypy unused-ignore
2026-03-10 13:25:16 -07:00
Patrick Buckley de64535221 Feat/eval improvements (#37)
* ci: add GitHub Release creation on tag push

* refactor: rename plan tool to create_plan

Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns

* feat: eval harness improvements inspired by autoresearch patterns

Major enhancements to turnstone-eval:

- Per-test timeout (--test-timeout, default 300s) and suite timeout
  (--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
  models and providers (--optimizer-model, --observer-model, etc.)
  with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
  failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)

* fix: address Copilot review feedback

- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
2026-03-10 13:06:39 -07:00
Patrick Buckley 187d004033 feat: watch tool — periodic command polling within workstreams (#36)
* feat: watch tool — periodic command polling within workstreams

Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.

Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)

* fix: address Copilot review — condition errors, restore deadlock, docs

- Condition eval errors now deactivate the watch immediately instead
  of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
  approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
  detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
2026-03-10 08:18:28 -07:00
Patrick Buckley 7ea150fa71 Bump version to 0.5.2 2026-03-09 13:42:28 -07:00
Patrick Buckley 4d665a5f62 fix: SSE reconnect loop — remove _sse_generation single-consumer lock
The _sse_generation mechanism assumed one SSE consumer per workstream,
but the bridge also maintains an SSE connection to each workstream.
When a new client connected (browser, proxy, or test), it incremented
the generation counter, killing the bridge's connection. The bridge
reconnected, killing the new client's connection — creating a
mutual-kill cascade that closed every SSE connection after one ping
cycle (5s).

Fix: remove _sse_generation entirely. sse-starlette handles disconnect
detection via its own ASGI task. Also remove the redundant
request.is_disconnected() check which raced with sse-starlette's
disconnect listener in Starlette 0.52.

Root cause confirmed via raw socket test: the server was sending
a zero-length chunked terminator (0\r\n\r\n) at exactly 5s,
cleanly ending the HTTP response body.
2026-03-09 13:40:36 -07:00
Patrick Buckley 3bc3250869 fix: recovered workstreams invisible in console UI (#35)
* fix: recovered workstreams invisible in console UI

Bridge startup recovery (_recover_workstreams) re-registered workstream
ownership but never published WorkstreamCreatedEvent to the cluster
channel. The collector's poll loop would pick up the workstream in its
internal state, but _apply_poll never fanned out SSE events to connected
browsers. Combined, this made channel-resumed workstreams invisible in
the console while remaining accessible through the proxied node UI.

- Bridge: emit WorkstreamCreatedEvent for each recovered workstream
- Collector: diff poll results and fan out synthetic ws_created/ws_closed
  events for workstream additions and removals
- Skip workstreams with empty IDs in poll processing
- Add 4 tests for poll-diff fanout behavior
- Update console data-flow diagram and architecture docs

* fix: address PR review — filter empty ws IDs, stable event ordering

- Filter empty-string keys from old_ids to avoid phantom ws_closed
  events if a previous poll inserted a workstream under key "".
- Sort set diffs before iterating so ws_created/ws_closed fanout
  order is deterministic across poll cycles.
2026-03-09 13:39:47 -07:00
Patrick Buckley db937486cf Bump version to 0.5.1 2026-03-09 01:48:18 -07:00
Patrick Buckley 554257ac4d fix: SSE proxy Firefox reconnect — Connection: keep-alive header 2026-03-09 01:46:35 -07:00
Patrick Buckley 5f0004dc91 feat: add ClusterSnapshot for instant console UI state rebuild (#34)
* feat: add ClusterSnapshot for instant console UI state rebuild

The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.

Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).

Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.

SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.

* fix: address review feedback and SSE proxy reconnect bug

Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
  both state and listener locks, eliminating the event gap between
  snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
  to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
  console schema models (ClusterNodeInfo, NodeDetailResponse,
  ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
  and ClusterOverviewResponse types instead of Record<string, unknown>.

SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
  when no upstream data arrives, preventing the browser EventSource
  from dropping idle connections. The raw byte passthrough refactor
  (4d11078) removed the proxy's independent keepalive — this restores
  it without reverting to EventSourceResponse.
2026-03-09 01:13:35 -07:00
Patrick Buckley 6cc1b3a5bd feat: add vision/image support to read_file tool (#33)
* feat: add vision/image support to read_file tool

read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO)
and returns base64-encoded content parts for vision-capable models.
Non-vision models receive a text description instead. A new
supports_vision flag on ModelCapabilities gates the feature, with
config.toml [models.*.capabilities] overrides for local models
(vLLM, llama.cpp, NIM).

* fix: address PR review feedback

- Discard _read_files on no-vision OSError path, include exception detail
- Discard _read_files on oversized image error (not a successful read)
- Validate capabilities type from config.toml (reject non-dict)
- Clarify tool description re: vision behavior and offset/limit scope
- Remove unused os import in tests, fix import sort order
- Handle list content (image tool results) in eval.py tool result loop
2026-03-08 23:43:42 -07:00
Patrick Buckley cc9afe94cd get title in collector for console 2026-03-08 22:32:52 -07:00
Patrick Buckley 136b75fdef Bump version to 0.5.0 2026-03-08 04:47:10 -07:00
Patrick Buckley 4d1107839b refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing

- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service

* Refactor SSE proxy to raw byte passthrough

- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
  httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
  events, and comments now flow through verbatim. Added per-request timeout
  override (read=None, pool=None) for long-lived SSE streams.

- tests/test_console.py: Add 3 new tests for SSE proxy:
  - Ping and event preservation
  - Upstream error status handling
  - Client disconnect handling

- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
  approach.
2026-03-08 04:46:34 -07:00
Patrick Buckley 7d66bc2159 Bump version to 0.4.6 2026-03-08 03:44:22 -07:00
Patrick Buckley 165cbb2d29 Bump version to 0.4.5 2026-03-08 03:29:44 -07:00
Patrick Buckley c79c47b940 Add MCP dynamic tool refresh with push notifications and periodic pol… (#31)
* Add MCP dynamic tool refresh with push notifications and periodic polling

MCP tool lists now stay up-to-date without restart via three mechanisms:
push notifications (ToolListChangedNotification) for servers that support
it, staggered periodic polling for servers that don't, and manual
/mcp refresh [server] command. MCPClientManager tracks tools per-server
with copy-on-write rebuild, notifies ChatSession listeners which rebuild
tool lists and ToolSearchManager (preserving expanded tools).

* Address Copilot review feedback on MCP refresh PR

- Fix /mcp refresh typo matching (startswith → exact token check)
- Validate --mcp-refresh-interval >= 0 at parse time via shared
  nonneg_float in config.py (deduplicated from cli.py + server.py)
- Clamp negative refresh_interval to 0 in MCPClientManager constructor
- Fix periodic refresh first poll timing (was initial_delay + interval,
  now initial_delay then immediate first poll)
- Clarify _on_mcp_tools_changed docstring re: O(n) BM25 build cost
2026-03-08 03:28:38 -07:00
Patrick Buckley 660c273e8e remove old demo.svg 2026-03-08 01:47:34 -08:00
Patrick Buckley c7586abd0a Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI

When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:

- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool

New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.

Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.

43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.

* Fix Copilot review feedback on tool search

- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
2026-03-08 01:43:38 -08:00
Patrick Buckley 14d57176ce Bump version to 0.4.4 2026-03-07 15:43:56 -08:00
Patrick Buckley 96084ca5f3 Fix PostgreSQL migration race condition with advisory lock
Multiple containers starting simultaneously race on Alembic migrations
against shared PostgreSQL. Use pg_advisory_lock so they wait in line.
Also update SQLite bootstrap to detect post-migration databases.
2026-03-07 15:41:29 -08:00
Patrick Buckley 8b92302247 Bump version to 0.4.3 2026-03-07 13:11:42 -08:00
Patrick Buckley e195ca54a6 new high level arch abstract 2026-03-07 13:01:19 -08:00
Patrick Buckley 50277cd4de Add 10-node cluster profile to Docker Compose 2026-03-07 12:54:41 -08:00
Patrick Buckley fb190f8977 Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity

Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).

Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id

Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days

Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.

* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec

- docs/channels.md: "resumes the session" → "resumes the workstream",
  "Session resumed:" → "Resumed:", "old session was pruned" → "old
  workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
  object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
  models — removes all stale session_id properties from WorkstreamInfo,
  DashboardWorkstream, CreateWorkstreamResponse schemas
2026-03-07 12:49:07 -08:00
Patrick Buckley 25b5e32089 Bump version to 0.4.2 2026-03-05 20:18:46 -08:00
Patrick Buckley 339981a258 Add GPT-5.3, GPT-5.4, and pro model capabilities (#28)
* Add GPT-5.3, GPT-5.4, and pro model capabilities

Add capability entries for gpt-5-pro (272k output, high-only reasoning),
gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro.

* Validate reasoning_effort against model capabilities

_apply_model_params now falls back to caps.default_reasoning_effort when
the requested value is not in caps.reasoning_effort_values. Prevents
sending unsupported effort levels to models like gpt-5-pro (high only).
2026-03-05 20:17:03 -08:00
Patrick Buckley 06de9ff83b Bump version to 0.4.1 2026-03-05 18:13:42 -08:00
Patrick Buckley 924b976f1f Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
2026-03-05 18:11:36 -08:00
Patrick Buckley d5db817391 Fix channel gateway Docker networking and console proxy approval scope
Two runtime bugs:

1. Channel gateway advertised http://127.0.0.1:8091 which is
   unreachable from other Docker containers. Add
   TURNSTONE_CHANNEL_ADVERTISE_URL env var override for Docker/K8s
   environments, set to http://channel:8091 in compose.yaml, and
   pass --http-host=0.0.0.0 so the gateway listens on all interfaces.

2. Console proxy service JWT had only "write" scope but the approval
   endpoint requires "approve". Tool approval buttons in the server
   web UI silently failed when accessed through the console proxy.
   Changed proxy token scopes to read+write+approve.
2026-03-05 18:01:52 -08:00
Patrick Buckley fc8ceb4c72 Update README: 3-node cluster diagram, remove directory tree
Replace single-node Mermaid diagram with a 3-node cluster layout
showing bridge+server pairs per node, shared Redis MQ, console, and
channel gateway. Remove the verbose directory tree listing.
2026-03-05 17:44:33 -08:00
Patrick Buckley 07234dec4d Replace ASCII architecture diagram with Mermaid in README
GitHub renders Mermaid natively as an interactive SVG. The new diagram
shows all client entry points (CLI, browser, SDK, Discord), the full
cluster topology including the channel gateway and notify path, and
the LLM provider layer.
2026-03-05 17:34:42 -08:00
Patrick Buckley dd4cc0b30d Add PROGRESS.md and .coverage to .gitignore 2026-03-05 17:31:55 -08:00
Patrick Buckley e7fe8fca9d Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening

Implements the `notify` tool allowing the LLM to send notifications to
Discord channels/users via the channel gateway. Includes fixes for 11
review findings: JWT auth on the gateway endpoint, first-healthy gateway
delivery with retry+backoff, rate limiting only on success, SSRF URL
scheme validation, Discord mention sanitization, SQLite ON CONFLICT
upsert preserving created timestamps, advertise URL resolution for
0.0.0.0 bind, randomized service IDs, generic error messages to prevent
internal state leakage, and partial direct-target validation.

Service registry with heartbeat-based health filtering (migration 005).
Channel gateway registers on startup, heartbeats every 30s, deregisters
on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint
auth (static + JWT), storage CRUD, and retry behavior.

* Add notify documentation, diagrams, and review fixes

Documentation:
- New sequence diagram 17-notify-flow.puml showing end-to-end delivery
- Updated 16-channel-architecture.puml with services table, notify HTTP
  path, and Notification Flow note
- channels.md: Notifications section (targeting, delivery flow, service
  registry, security) and new config table entries
- tools.md: notify tool reference, updated counts/tables (14→15 tools)
- security.md: channel gateway row in service-to-service auth table
- architecture.md: notification subsystem paragraph

Review fixes (copilot):
- _http.py: fail closed when auth unconfigured (401 instead of pass-
  through), strip whitespace on message/title, generic error messages
  for user-not-found vs no-linked-channels
- session.py: parse gateway response JSON and require at least one
  result with status=="sent" before counting as success
- _postgresql.py: use index_elements instead of constraint for upsert
2026-03-05 17:24:00 -08:00
Patrick Buckley 42b9f89988 Add scheduled task system with cron/at scheduling (#26)
* Add scheduled task system with cron/at scheduling, admin API, and console UI

Console-integrated background scheduler dispatches workstreams on recurring
cron expressions or one-shot ISO8601 timestamps. Four target modes: auto
(best node by headroom), pool (shared queue), all (fan-out), or specific
node. Redis distributed lock with unique owner + Lua conditional release
prevents duplicate dispatch in multi-console deployments.

Storage: scheduled_tasks + scheduled_task_runs tables (migration 004),
9 protocol methods on both SQLite and PostgreSQL backends, field allowlist
on updates, run history auto-pruned at 90 days.

API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter
validation, ISO8601 future-time checks, field length bounds, schedule
count cap (200), and OpenAPI spec entries with Pydantic models.

UI: Schedules tab in admin panel with create/edit/delete modals, run
history modal, cron/at type toggle, target mode select, status dots for
accessibility, responsive grid, keyboard navigation, and focus management.

Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches
logged at WARNING with created_by attribution, user_id propagated in
CreateWorkstreamMessage for audit trail.

46 tests across storage, scheduler engine, and API endpoints.

* Add croniter to test extras for CI compatibility

CI installs [test] extras but not [console], so croniter was missing
when schedule API tests import console/server.py validation functions.

* Address Copilot review: timezone validation, focus trap, enabled flag

- Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z)
- UI appends +00:00 to datetime-local values for explicit UTC
- Fix datetime-local normalization: check length before appending seconds
- Add textarea to modal focus trap selector (prevents focus escape)
- Fix _normalize_task_dict not called in update response
- Persist enabled=false on create (storage defaults to enabled=1)
- Validate at_time is still in future when re-enabling a one-shot task
- broker._redis coupling acknowledged as tracked tech debt
2026-03-05 16:05:00 -08:00
Patrick Buckley 77c0a7736b Bump version to 0.4.0 and update security docs
- Version bump in __init__.py, pyproject.toml, api-reference.md
- security.md: document JWT aud/iss claims, login rate limiting,
  secure cookie defaults (24h, Secure flag), CORS restriction,
  service JWT auto-rotation, secret strength validation, and
  proxy auth forwarding via service tokens (not user JWT forwarding)
2026-03-04 20:45:00 -08:00
Patrick Buckley 872e1770e6 Feature/code dedup (#25)
* Add JWT auth security hardening (6 fixes)

- Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h
- Login brute-force protection: LoginRateLimiter (5 attempts/5min per key)
- JWT aud/iss claims: create_jwt/validate_jwt support audience validation
- Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh
- CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var
- JWT secret strength: warning on secrets shorter than 32 chars
- Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing

* Refactor duplicated code into shared utilities and fix 3 UI bugs

Code deduplication (~235 net lines removed):
- Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py
- Create core/web_helpers.py (require_storage_or_503, read_json_or_400,
  parse_cors_origins, cors_middleware)
- Extract add_redis_args/broker_from_args to mq/broker.py
- Extract add_log_args/configure_logging_from_args to core/log.py
- Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper,
  unused required_role(), duplicate detect_model() wrapper

Bug fixes:
- Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server
  nodes (use ServiceTokenManager with JWT_AUD_SERVER instead)
- Fix login form autofill: wrap inputs in <form>, add name attributes,
  set type=submit on button
- Fix SSE reconnecting flash: add onopen handler to clear status
  immediately on connection (not waiting for first message)
- Fix chat scroll: add min-height:0 to flex containers, overflow:hidden
  on body to constrain viewport height

* Address CI typecheck failure and Copilot review feedback

- Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary)
- Bridge SSE loops: use event_hooks for auth header refresh on reconnect
  instead of static headers that go stale after token rotation
- Login form: remove javascript:void(0) action (CSP anti-pattern)
- Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals
  in middleware builder calls to prevent drift
2026-03-04 20:35:21 -08:00
Patrick Buckley a6e929b0a0 Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24)

Bidirectional channel adapter framework connecting external messaging
platforms to turnstone workstreams via Redis MQ. Discord ships as the
first adapter; the protocol supports future Slack/Teams integrations.

Channel framework:
- ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping
- AsyncRedisBroker with single dispatch loop and per-channel ordered workers
- channel_routes table (migration 003) for persistent route storage
- 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD)
- Unified turnstone-channel gateway entry point, loads adapters by config
- Message chunking, approval formatting, plan review formatting

Discord adapter:
- discord.py v2.4+ bot with thread-per-@mention model
- Slash commands: /link (modal), /unlink, /ask, /status, /close
- Persistent button views for tool approval and plan review
- Streaming responses via edit-in-place (1.5s interval)
- Stale route detection and atomic session resume via resume_session field
- SessionResumedEvent confirmation back to channel
- Auto-approve support (blanket + per-tool list)

Atomic session resume:
- resume_session field on CreateWorkstreamMessage for single-request resume
- Server resumes session during POST /v1/api/workstreams/new atomically
- Bridge emits SessionResumedEvent to per-workstream channel
- WorkstreamCreatedEvent extended with resumed/session_id/message_count
- Server UI dashboardResumeSession simplified to single request
- Pruned sessions fall back gracefully to fresh start

Service auth:
- Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET
- Bridge: approve scope (1 week). Console collector: read. Proxy: write.

Console admin:
- Channels tab with per-user view, force-link modal, unlink
- 3 admin API endpoints for channel user management
- Styled confirm modals replacing browser confirm() dialogs

Bug fixes:
- AsyncRedisBroker: replaced per-channel listener tasks with single
  dispatch loop + per-channel queue workers (fixes message stealing race)
- Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates
- Bridge: _active_sends tracked for initial messages (fixes missing
  TurnCompleteEvent and unfinalized streaming messages)
- Bridge: HTTP calls moved outside lock scope in approval handlers
- Bridge: _handle_send cleans up _active_sends on HTTP/server errors
- Formatter: reads server SSE format (func_name/preview) with fallback

Docs, SDK, tests:
- docs/channels.md setup guide, architecture diagram 16
- Updated api-reference.md, architecture.md, console.md, docker.md
- Python SDK: resume_session param on create_workstream (async + sync)
- TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces
- OpenAPI schema: resume_session request, resumed/message_count response
- 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing,
  18 discord, 12 resume flow) — 1120 total passing

* Fix CI lint/typecheck failures and address Copilot review feedback (#24)

Lint: fix import ordering, remove unused imports, use contextlib.suppress.
Mypy: explicit postgresql dialect import, add discord module overrides for
optional-dependency CI environments.
Copilot: fix double-escaping in admin confirm modals, return resolved
session_id from server resume response, fix channel_routes diagram schema,
use atomic setdefault for routing locks, add post-insert race guard in
admin channel create, support SSE format in auto-approve check, update
identity linking note in architecture diagram.

* Fix remaining mypy call-arg errors for discord.py optional dependency

Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class
definitions that fail when discord.py is not installed in CI.
2026-03-04 13:02:58 -08:00
Patrick Buckley 047680d669 Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23)

JWT-based authentication with three token types: config-file (hmac,
backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs
(HS256, 24h expiry). Username:password login via bcrypt. Hierarchical
scopes: read < write < approve.

New tables: users (username, password_hash), api_tokens (token_hash,
scopes, expires), channel_users (future channel integrations). user_id
column added to sessions and workstreams for attribution.

Console owns admin CRUD (6 endpoints under /api/admin/). Server
validates JWTs locally with shared signing secret. Public /api/auth/setup
endpoint for first-time admin creation (atomic, only works with zero
users). turnstone-admin CLI for user/token management.

Admin console UI: Users and Tokens tabs with full CRUD modals, scope
badges, token show-once with clipboard copy, keyboard accessibility
(focus traps, Escape, arrow key tabs, ARIA roles).

Login UI redesigned: username:password primary, token toggle for legacy,
setup wizard auto-detected via /api/auth/status. Python + TypeScript
SDKs updated with login(username, password), authStatus(), setup().

New docs/security.md + diagram 15-auth-architecture.puml. All existing
docs updated. OpenAPI specs include all new endpoints. 64 new tests
(1023 total). Dependencies: PyJWT, bcrypt.

* Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review

Address Copilot review feedback: escape double quotes in escapeHtml()
to prevent XSS in HTML attributes, add JWT validation fallback so
config tokens containing dots still work, add user_id to
AuthLoginResponse schema, return created field from admin_create_user,
and correct five documentation files to match actual API behavior.
2026-03-04 09:12:18 -08:00
Patrick Buckley 0fd0ad3b2d Add structured logging with structlog and context propagation
Replace ad-hoc logging.basicConfig() calls across all 6 entry points with
a centralized configure_logging() function backed by structlog. JSON output
when stderr is not a TTY (production/Docker), colored console output otherwise.

- New turnstone/core/log.py: configure_logging(), get_logger(), contextvars
  for node_id/ws_id/user_id/request_id auto-injected into every log event
- All entry points (server, bridge, console, sim, cli, migrate) call
  configure_logging() with --log-level and --log-format CLI flags
- Server operational print() calls replaced with structured log.info()
- LogContextMiddleware sets request_id + ws_id per HTTP request with
  token-based reset to prevent context leaking across requests
- Bridge _run_in_context() helper propagates ctx_node_id to child threads
- Env var overrides: TURNSTONE_LOG_LEVEL, TURNSTONE_LOG_FORMAT
- 18 new tests (959 total passing)
2026-03-04 06:47:30 -08:00
Patrick Buckley f3dba836dd Add Git LFS requirement note to README, Diagram PNGs are stored in LFS; git-lfs must be installed for cloning them. 2026-03-04 06:14:27 -08:00
Patrick Buckley 7adda343fc Update docs and diagrams for cluster-scale schema changes
- StorageBackend protocol: document 5 new workstream methods (26 total)
- Session ID: 12-char hex → 32-char full UUID in API reference
- /health endpoint: add node_id field to response docs
- sessions table: document node_id and ws_id columns
- Bridge node_id: document server-owned identity with /health retrieval
- Regenerate storage architecture PNG from updated PlantUML
2026-03-04 06:12:33 -08:00
Patrick Buckley a20a058c59 Add cluster-scale schema, fix console proxy UX, harden SDK sync runner (#22)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner

Schema redesign for multi-node deployments:
- New `workstreams` table with node_id, state, lifecycle tracking
- Add node_id + ws_id columns to sessions table with indexes
- Full UUID (32 hex) for session_id and ws_id (was truncated 12/8)
- Server generates and owns node_id, bridge retrieves via /health
- Bridge retries with exponential backoff, fatal on auth errors
- WorkstreamManager persists workstreams and state changes to storage
- /health endpoint exposes node_id for bridge discovery

Console proxy UX fixes:
- Remove duplicate turnstone branding from proxy banner
- Same-tab navigation for Open Node UI and workstream deep links

SDK _SyncRunner fix:
- Sentinel pattern for StopAsyncIteration across thread boundary

Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir).

* Address PR #22 review feedback

- Fix CLI session_factory signature (ws_id param) — CI typecheck failure
- First-phase eviction in create() now calls _cleanup_ui + record_eviction
- close() persists "closed" state to storage via update_workstream_state
- Fix noqa comment in test to pragma: no cover
2026-03-04 06:04:59 -08:00
204 changed files with 41294 additions and 2700 deletions
+1
View File
@@ -22,6 +22,7 @@ OPENAI_API_KEY=sk-...
# -- Authentication ------------------------------------------------------------
# TURNSTONE_AUTH_ENABLED=true
# TURNSTONE_AUTH_TOKEN=your-secret-token
# TURNSTONE_JWT_SECRET=python -c "import secrets; print(secrets.token_hex(32))"
# -- Ports ---------------------------------------------------------------------
# SERVER_PORT=8080
+8
View File
@@ -5,6 +5,7 @@ on:
tags: ["v*"]
permissions:
contents: write
id-token: write
jobs:
@@ -19,3 +20,10 @@ jobs:
- run: pip install build
- run: python -m build
- uses: pypa/gh-action-pypi-publish@release/v1
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
generate_release_notes: true
draft: false
prerelease: ${{ contains(github.ref, '-') }}
+2
View File
@@ -17,3 +17,5 @@ venv/
.plan.md
.plan-*.md
.hypothesis/
PROGRESS.md
.coverage
+1 -1
View File
@@ -34,7 +34,7 @@ RUN useradd --create-home --shell /bin/bash turnstone
# Install the wheel with all optional extras
COPY --from=builder /build/wheels/*.whl /tmp/wheels/
RUN pip install --no-cache-dir "$(ls /tmp/wheels/*.whl)[mq,console,sim,postgres]" \
RUN pip install --no-cache-dir "$(ls /tmp/wheels/*.whl)[mq,console,sim,postgres,discord]" \
&& rm -rf /tmp/wheels
# Health check script (stdlib only, no pip deps needed)
+92
View File
@@ -0,0 +1,92 @@
# Bootstrap Wizard
Interactive, AI-guided setup for Turnstone deployments. Instead of manually
editing `.env` files and reading deployment docs, the wizard walks you through
every decision conversationally and generates all the config files for you.
## Quick Start
```bash
turnstone-bootstrap
```
That's it — no flags, no arguments. The wizard prompts for everything.
## How It Works
1. **Pick a model** — Choose OpenAI, Anthropic, or a local/vLLM endpoint to
power the wizard. Local endpoints auto-detect available models.
2. **Answer questions** — The AI walks you through deployment mode, LLM
provider, database, authentication, ports, and optional features.
3. **Review generated files** — Each file is previewed before writing. You
confirm or reject every write.
4. **Start the stack** — The wizard prints the exact `docker compose` command
and a `setup.sh` script to create your first admin user, roles, and policies.
## What Gets Generated
| File | Purpose |
|------|---------|
| `.env` | All environment variables for `compose.yaml` |
| `setup.sh` | Post-start script: creates admin user, roles, tool policies, prompt templates via the API |
| `docker-compose.override.yaml` | Only if customizations beyond env vars are needed |
## Requirements
- **Python 3.11+** with turnstone installed (`pip install turnstone`)
- **An LLM API key** — for the wizard itself (OpenAI, Anthropic, or a local
model). This can differ from the LLM your deployment will use.
- **Docker & Docker Compose** — needed to run the stack. The wizard detects
whether Docker is installed and gives platform-specific install instructions
if it's missing. You can still generate config files without Docker.
## Deployment Modes
The wizard supports two deployment modes:
- **Single-node production** (`docker compose --profile production up`) —
1 server + bridge + console + PostgreSQL + Redis. Good for most use cases.
- **Multi-node cluster** (`docker compose --profile cluster up`) —
10-node server/bridge fleet + PostgreSQL + Redis. For high-throughput or
HA deployments.
## Example Session
```
$ turnstone-bootstrap
Turnstone Bootstrap Wizard v0.5.4
────────────────────────────────────────────────
Which provider for this wizard?
[1] OpenAI
[2] Anthropic
[3] OpenAI-compatible (local/vLLM)
> 3
Base URL [http://localhost:8000/v1]:
API key (press Enter for 'none'):
Querying http://localhost:8000/v1 for available models...
Found model: Qwen/Qwen3-32B
Connected to Qwen/Qwen3-32B. Handing off to AI assistant...
> (AI walks you through the rest interactively)
```
## Tips
- **Re-run safely** — running the wizard again detects your existing `.env`
and offers to update it rather than overwriting.
- **Duplicate writes are skipped** — if the LLM tries to write the same file
twice with identical content, it's silently ignored.
- **Type `quit` to exit** at any time during the conversation.
- **Ctrl+C** is handled gracefully — press once to interrupt, twice to exit.
## See Also
- [Docker Deployment](docker.md) — manual compose setup and profiles
- [Security](security.md) — auth architecture and token types
- [Governance](governance.md) — roles, policies, and templates
+36 -75
View File
@@ -11,21 +11,18 @@ Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone)
## What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. It runs as:
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. Native deferred tool loading for Anthropic and OpenAI APIs reduces token overhead and improves tool selection accuracy when MCP servers expose many tools; local models (vLLM, llama.cpp) get a transparent client-side BM25 fallback. It runs as:
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
- **Multi-node clusters** — generic work load-balances across nodes, directed work routes to a specific server
- **Cluster dashboard** — real-time view of all nodes and workstreams, workstream creation with node targeting, reverse proxy for server UIs (only the console port needs network access)
- **Governance & compliance** — role-based access control, tool policies, usage tracking, and append-only audit logs
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
```
External System → Message Queue → Bridge (per node) → Turnstone Server → LLM + Tools
Pub/Sub → Progress Events → External System
turnstone-console → Cluster Dashboard (browser)
```
<p align="center">
<img src="docs/diagrams/architecture-overview.svg" alt="Turnstone system architecture — data flow from clients through gateways, Redis MQ, cluster nodes, to LLM providers" width="960"/>
</p>
## Quickstart
@@ -111,69 +108,7 @@ All frontends connect to any OpenAI-compatible API (vLLM, NVIDIA NIM/NGC, llama.
## Architecture
```
turnstone/
├── core/ # UI-agnostic engine
│ ├── session.py # ChatSession — multi-turn loop, tool dispatch, agents
│ ├── providers/ # LLM provider adapters (OpenAI, Anthropic)
│ │ ├── _protocol.py # LLMProvider protocol, ModelCapabilities, StreamChunk
│ │ ├── _openai.py # OpenAI-compatible (OpenAI, vLLM, llama.cpp)
│ │ └── _anthropic.py # Anthropic Messages API (native streaming, thinking)
│ ├── tools.py # Tool definitions (auto-loaded from JSON)
│ ├── workstream.py # WorkstreamManager — parallel independent sessions
│ ├── mcp_client.py # MCP client manager (external tool servers)
│ ├── model_registry.py # ModelRegistry — named models, fallback routing, per-workstream selection
│ ├── config.py # Unified TOML config (~/.config/turnstone/config.toml)
│ ├── memory.py # Persistence facade (delegates to storage/)
│ ├── storage/ # Pluggable storage backend (SQLite + PostgreSQL)
│ ├── metrics.py # Prometheus-compatible metrics collector
│ ├── healthcheck.py # Backend health monitor + circuit breaker
│ ├── ratelimit.py # Per-IP token-bucket rate limiter
│ ├── edit.py # File editing (fuzzy match, indentation)
│ ├── safety.py # Path validation, sandbox checks
│ ├── sandbox.py # Command sandboxing
│ └── web.py # Web fetch/search helpers
├── mq/ # Message queue integration
│ ├── protocol.py # Typed message dataclasses (JSON serialization)
│ ├── broker.py # Abstract MessageBroker + RedisBroker
│ ├── bridge.py # Bridge service (queue ↔ HTTP API, multi-node routing)
│ └── client.py # TurnstoneClient — Python API for external systems
├── console/ # Cluster dashboard
│ ├── collector.py # ClusterCollector — aggregates all nodes via Redis + HTTP
│ ├── server.py # Dashboard Starlette/ASGI server + SSE
│ └── static/ # Cluster dashboard web UI
├── tools/ # Tool schemas (one JSON file per tool)
├── ui/ # Frontend assets and terminal rendering
│ └── static/ # Web UI (HTML, CSS, JS)
├── sim/ # Cluster simulator
│ ├── cluster.py # SimCluster — orchestrates N nodes + dispatchers
│ ├── node.py # SimNode + SimWorkstream — protocol-compatible node
│ ├── engine.py # LLM + tool execution simulation
│ ├── scenario.py # 5 workload scenarios (steady, burst, node_failure, …)
│ ├── metrics.py # Latency, throughput, utilization collection
│ └── cli.py # CLI entry point (turnstone-sim)
├── cli.py # Terminal frontend (+ /cluster commands for console)
├── server.py # Web frontend (Starlette/ASGI + SSE)
└── eval.py # Evaluation and prompt optimization harness
├── api/ # OpenAPI spec generation (Pydantic v2 models)
├── sdk/ # Client SDKs (sync + async, Python)
docs/
├── architecture.md # System architecture and threading model
├── api-reference.md # Web server API and SSE event reference
├── sdk.md # Client SDK reference (Python + TypeScript)
├── console.md # Cluster dashboard service (turnstone-console)
├── docker.md # Docker Compose deployment and configuration
├── simulator.md # Cluster simulator usage and scenarios
├── tools.md # Tool schemas, execution pipeline, approval flow
├── eval.md # Evaluation harness internals
└── diagrams/ # UML architecture diagrams (PlantUML sources + PNGs)
└── png/ # Pre-rendered diagram images
deploy/
├── helm/turnstone/ # Helm chart for Kubernetes
└── terraform/ # Terraform modules (AWS ECS/Fargate)
```
### Architecture Diagrams
### Diagrams
Detailed UML diagrams are available in [`docs/diagrams/`](docs/diagrams/):
@@ -193,6 +128,23 @@ Detailed UML diagrams are available in [`docs/diagrams/`](docs/diagrams/):
| [Deployment](docs/diagrams/png/12-deployment.png) | Docker Compose service topology |
| [SDK Architecture](docs/diagrams/png/13-sdk-architecture.png) | Python + TypeScript client libraries |
| [Storage Architecture](docs/diagrams/png/14-storage-architecture.png) | Pluggable database backends (SQLite + PostgreSQL) |
| [Auth Architecture](docs/diagrams/png/15-auth-architecture.png) | JWT, scopes, token types, login flows |
| [Channel Architecture](docs/diagrams/png/16-channel-architecture.png) | Discord/Slack adapter protocol and routing |
| [Notify Flow](docs/diagrams/png/17-notify-flow.png) | Channel notification dispatch |
| [Watch Architecture](docs/diagrams/png/18-watch-architecture.png) | Periodic command polling daemon |
| [Governance Architecture](docs/diagrams/png/19-governance-architecture.png) | RBAC, policies, audit, usage enforcement flow |
### Governance
Turnstone includes a built-in governance layer for enterprise deployments — manage who can do what, which tools run unattended, and where every token goes.
- **RBAC** — 15 granular permissions, 3 built-in roles (admin / operator / viewer), custom roles, privilege escalation prevention
- **Tool policies** — glob-pattern rules (`allow` / `deny` / `ask`) with priority ordering; automate approvals or lock down dangerous tools
- **Prompt templates** — reusable system messages with `{{variable}}` substitution and categories
- **Usage tracking** — per-request token and tool metrics, aggregation by day / model / user, automatic 90-day pruning
- **Audit logging** — append-only event trail for all admin mutations, IP-aware, 365-day retention
All governance features are managed through the console admin panel (10 tabs) and the full REST API. See [docs/governance.md](docs/governance.md) for setup and configuration.
## Multi-node routing
@@ -217,12 +169,12 @@ Bridges BLPOP from their per-node queue (priority) then the shared queue. Direct
## Tools
14 built-in tools, 2 agent tools, plus external tools via MCP:
16 built-in tools, 2 agent tools, plus external tools via MCP:
| Tool | Description | Auto-approved |
|------|-------------|:---:|
| `bash` | Execute shell commands | |
| `read_file` | Read file contents | yes |
| `read_file` | Read file contents (text or images with vision models) | yes |
| `write_file` | Write/create files | |
| `edit_file` | Fuzzy-match file editing | |
| `search` | Search files by name/content | yes |
@@ -233,13 +185,17 @@ Bridges BLPOP from their per-node queue (priority) then the shared queue. Direct
| `remember` | Save persistent facts | yes |
| `recall` | Search memories and history | yes |
| `forget` | Remove a memory | yes |
| `notify` | Send notifications to linked channels | yes |
| `watch` | Periodic command polling with conditions | |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
| `mcp__*` | External tools from MCP servers | |
When the total tool count exceeds a configurable threshold (default 20), MCP tools are automatically deferred using native `defer_loading` on Anthropic and OpenAI APIs, or a transparent client-side BM25 search for local models. The LLM discovers deferred tools on demand via a `tool_search` capability — no configuration needed beyond `--tool-search auto` (the default).
### MCP Tool Servers
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions.
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions. Tool lists stay fresh via push notifications (`tools.listChanged`), periodic polling for servers without push, and manual `/mcp refresh`.
Configure via `config.toml` or `--mcp-config`:
@@ -259,7 +215,7 @@ turnstone --mcp-config ~/.config/turnstone/mcp.json
turnstone-server --mcp-config ~/.config/turnstone/mcp.json
```
Use `/mcp` in the REPL to list connected tools. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
Use `/mcp` in the REPL to list connected tools, `/mcp refresh` to re-fetch tool lists from servers. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
### Multi-Model and Multi-Provider Support
@@ -314,6 +270,9 @@ agent_model = "" # model alias for plan/task sub-agents
[tools]
timeout = 30
skip_permissions = false
search = "auto" # "auto" (enable when >threshold tools), "on", "off"
search_threshold = 20 # min tools before tool search activates
search_max_results = 5 # max tools returned per search query
[server]
host = "0.0.0.0"
@@ -354,6 +313,7 @@ path = ".turnstone.db" # SQLite file path (relative to working directory)
[mcp]
config_path = "" # path to MCP JSON config file (alternative to TOML sections)
refresh_interval = 14400 # periodic refresh for servers without push notifications (seconds, 0 to disable)
[mcp.servers.example] # one section per MCP server
command = "npx"
@@ -412,6 +372,7 @@ Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
- Redis (for message queue bridge — `pip install turnstone[mq]`)
- Anthropic provider (optional — `pip install turnstone[anthropic]`)
- PostgreSQL (optional, for production — `pip install turnstone[postgres]`)
- [Git LFS](https://git-lfs.com/) (for cloning — diagram PNGs are stored in LFS)
## License
+315 -3
View File
@@ -2,11 +2,12 @@
# Turnstone Docker Compose Stack
#
# Usage:
# Default (SQLite): docker compose up
# Infra only: docker compose up
# Single node: docker compose --profile production up
# Production (PG): DB_BACKEND=postgresql docker compose --profile production up
# (or set DB_BACKEND=postgresql in .env)
# 10-node cluster: docker compose --profile cluster up
# Cluster + DDG: docker compose --profile ddgCluster up
# With simulator: docker compose --profile sim up
# Scale bridges: docker compose up --scale bridge=3
# =============================================================================
name: turnstone
@@ -28,6 +29,8 @@ services:
image: postgres:17-alpine
profiles:
- production
- cluster
- ddgCluster
environment:
POSTGRES_DB: turnstone
POSTGRES_USER: ${POSTGRES_USER:-turnstone}
@@ -87,6 +90,8 @@ services:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
command:
- sh
- -c
@@ -98,10 +103,12 @@ services:
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
ports:
- "${SERVER_PORT:-8080}:8080"
volumes:
- turnstone-data:/data
- ./docker/mcp-ddg.json:/etc/turnstone/mcp-ddg.json:ro
environment:
- LLM_BASE_URL=${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
- OPENAI_API_KEY=${OPENAI_API_KEY:-dummy}
@@ -109,9 +116,12 @@ services:
- SKIP_PERMISSIONS=${SKIP_PERMISSIONS:-}
- TURNSTONE_AUTH_ENABLED=${TURNSTONE_AUTH_ENABLED:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- MODEL=${MODEL:-}
- MCP_CONFIG=${MCP_CONFIG:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
- TURNSTONE_NODE_ID=${TURNSTONE_NODE_ID:-}
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -122,6 +132,9 @@ services:
postgres:
condition: service_healthy
required: false
ddg-search:
condition: service_healthy
required: false
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
@@ -138,6 +151,8 @@ services:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
command:
- turnstone-bridge
- --server-url=http://server:8080
@@ -148,6 +163,7 @@ services:
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
networks:
- turnstone-net
depends_on:
@@ -177,6 +193,9 @@ services:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_AUTH_ENABLED=${TURNSTONE_AUTH_ENABLED:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
networks:
- turnstone-net
depends_on:
@@ -190,6 +209,78 @@ services:
start_period: 10s
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-channel — Channel gateway (Discord, Slack, etc.)
# Requires TURNSTONE_DISCORD_TOKEN to enable Discord adapter
# -------------------------------------------------------------------
channel:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
- cluster
- ddgCluster
command:
- sh
- -c
- >-
turnstone-channel
--redis-host=redis
--redis-port=6379
--http-host=0.0.0.0
$${TURNSTONE_DISCORD_GUILD:+--discord-guild $$TURNSTONE_DISCORD_GUILD}
environment:
- TURNSTONE_DISCORD_TOKEN=${TURNSTONE_DISCORD_TOKEN:-}
- TURNSTONE_DISCORD_GUILD=${TURNSTONE_DISCORD_GUILD:-0}
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
- TURNSTONE_CHANNEL_ADVERTISE_URL=http://channel:8091
networks:
- turnstone-net
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
required: false
restart: unless-stopped
# -------------------------------------------------------------------
# ddg-search — DuckDuckGo Search MCP server (HTTP transport)
# Provides web search + content fetch tools to turnstone via MCP.
# No API key required.
#
# Start with: MCP_CONFIG=/etc/turnstone/mcp-ddg.json \
# docker compose --profile ddgCluster up
# -------------------------------------------------------------------
ddg-search:
image: python:3.13-slim
profiles:
- ddgCluster
command:
- sh
- -c
- >-
pip install --no-cache-dir duckduckgo-mcp-server &&
python -c "from mcp.server.transport_security import TransportSecuritySettings; import duckduckgo_mcp_server.server as s; s.safe_search=s.SafeSearchMode.OFF; s.mcp.settings.host='0.0.0.0'; s.mcp.settings.port=3000; s.mcp.settings.transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False); s.mcp.run(transport='streamable-http')"
networks:
- turnstone-net
healthcheck:
test: ["CMD-SHELL", "python -c \"import socket; s=socket.create_connection(('0.0.0.0',3000),2); s.close()\""]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
deploy:
resources:
limits:
memory: 256M
cpus: '0.25'
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-sim — Multi-node cluster simulator (no LLM needed)
# Start with: docker compose --profile sim up
@@ -229,3 +320,224 @@ services:
redis:
condition: service_healthy
restart: "no"
# ===================================================================
# 10-node cluster (profile: cluster)
#
# Each node is a server + bridge pair. All share the same PostgreSQL
# and Redis instances. Access via console at :8090.
#
# Start: docker compose --profile cluster up
# ===================================================================
# -- cluster servers ------------------------------------------------
server-1: &cluster-server
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster, ddgCluster]
command: &cluster-server-cmd
- sh
- -c
- >-
turnstone-server
--host 0.0.0.0
--port 8080
--base-url "$${LLM_BASE_URL}"
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
volumes:
- turnstone-data:/data
- ./docker/mcp-ddg.json:/etc/turnstone/mcp-ddg.json:ro
environment: &cluster-server-env
LLM_BASE_URL: ${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
OPENAI_API_KEY: ${OPENAI_API_KEY:-dummy}
TAVILY_API_KEY: ${TAVILY_API_KEY:-}
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_AUTH_ENABLED: ${TURNSTONE_AUTH_ENABLED:-}
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
MODEL: ${MODEL:-}
MCP_CONFIG: ${MCP_CONFIG:-}
TURNSTONE_DB_BACKEND: ${DB_BACKEND:-postgresql}
TURNSTONE_DB_URL: ${DATABASE_URL:-postgresql://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:?}@postgres:5432/turnstone}
TURNSTONE_NODE_ID: node-1
extra_hosts: ["host.docker.internal:host-gateway"]
networks: [turnstone-net]
depends_on:
redis: { condition: service_healthy }
postgres: { condition: service_healthy }
ddg-search: { condition: service_healthy, required: false }
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
deploy:
resources:
limits: { memory: 384M, cpus: '0.5' }
restart: unless-stopped
server-2:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-2 }
server-3:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-3 }
server-4:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-4 }
server-5:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-5 }
server-6:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-6 }
server-7:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-7 }
server-8:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-8 }
server-9:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-9 }
server-10:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-10 }
# -- cluster bridges ------------------------------------------------
bridge-1: &cluster-bridge
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster, ddgCluster]
command:
- turnstone-bridge
- --server-url=http://server-1:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
environment: &cluster-bridge-env
REDIS_PASSWORD: ${REDIS_PASSWORD:-}
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
networks: [turnstone-net]
depends_on:
server-1: { condition: service_healthy }
redis: { condition: service_healthy }
deploy:
resources:
limits: { memory: 256M, cpus: '0.25' }
restart: unless-stopped
bridge-2:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-2:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-2: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-3:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-3:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-3: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-4:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-4:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-4: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-5:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-5:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-5: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-6:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-6:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-6: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-7:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-7:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-7: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-8:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-8:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-8: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-9:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-9:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-9: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-10:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-10:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-10: { condition: service_healthy }
redis: { condition: service_healthy }
-221
View File
@@ -1,221 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" font-family="ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace" font-size="13">
<style>
@keyframes pulse-green { 0%,100% { opacity:0.5 } 50% { opacity:1 } }
@keyframes pulse-yellow { 0%,100% { opacity:0.4 } 50% { opacity:1 } }
@keyframes pulse-blue { 0%,100% { opacity:0.3 } 50% { opacity:1 } }
@keyframes fadein { from { opacity:0 } to { opacity:1 } }
.pg { animation: pulse-green 2s infinite }
.py { animation: pulse-yellow 1.8s infinite }
.pb { animation: pulse-blue 2.2s infinite }
.f1 { animation: fadein 0.4s 0.2s both }
.f2 { animation: fadein 0.4s 0.4s both }
.f3 { animation: fadein 0.4s 0.6s both }
.f4 { animation: fadein 0.4s 0.8s both }
.f5 { animation: fadein 0.4s 1.0s both }
.f6 { animation: fadein 0.4s 1.3s both }
.f7 { animation: fadein 0.4s 1.5s both }
.f8 { animation: fadein 0.4s 1.7s both }
.f9 { animation: fadein 0.4s 1.9s both }
.f10 { animation: fadein 0.4s 2.1s both }
.f11 { animation: fadein 0.4s 2.3s both }
.f12 { animation: fadein 0.4s 2.5s both }
</style>
<!-- Window chrome -->
<rect rx="10" width="860" height="520" fill="#1a1b26"/>
<rect width="860" height="36" rx="10" fill="#16161e"/>
<rect y="26" width="860" height="10" fill="#16161e"/>
<circle cx="20" cy="18" r="6" fill="#f7768e"/>
<circle cx="40" cy="18" r="6" fill="#e0af68"/>
<circle cx="60" cy="18" r="6" fill="#9ece6a"/>
<text x="430" y="22" text-anchor="middle" fill="#565f89" font-size="12">turnstone — console</text>
<!-- Header -->
<rect y="36" width="860" height="30" fill="#24283b"/>
<rect y="66" width="860" height="1" fill="#3b4261"/>
<text x="16" y="56" fill="#7aa2f7" font-size="14" font-weight="bold">turnstone console</text>
<text x="200" y="56" fill="#565f89" font-size="12">6 nodes · 10 workstreams</text>
<!-- ====== State cards ====== -->
<g transform="translate(16, 78)" class="f1" opacity="0">
<!-- RUN card -->
<rect x="0" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="0" y="0" width="156" height="3" rx="6" fill="#9ece6a"/>
<text x="78" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">3</text>
<text x="78" y="50" text-anchor="middle" fill="#565f89" font-size="10">▸ RUN</text>
<!-- THINK card -->
<rect x="168" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="168" y="0" width="156" height="3" rx="6" fill="#7aa2f7"/>
<text x="246" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">2</text>
<text x="246" y="50" text-anchor="middle" fill="#565f89" font-size="10">◌ THINK</text>
<!-- ATTN card -->
<rect x="336" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="336" y="0" width="156" height="3" rx="6" fill="#e0af68"/>
<text x="414" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">1</text>
<text x="414" y="50" text-anchor="middle" fill="#565f89" font-size="10">◆ ATTN</text>
<!-- ERR card -->
<rect x="504" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="504" y="0" width="156" height="3" rx="6" fill="#f7768e"/>
<text x="582" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">0</text>
<text x="582" y="50" text-anchor="middle" fill="#565f89" font-size="10">✖ ERR</text>
<!-- IDLE card -->
<rect x="672" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="672" y="0" width="156" height="3" rx="6" fill="#565f89"/>
<text x="750" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">4</text>
<text x="750" y="50" text-anchor="middle" fill="#565f89" font-size="10">· IDLE</text>
</g>
<!-- Aggregate bar -->
<text x="16" y="160" fill="#565f89" font-size="11" class="f2" opacity="0">197k tokens · 42 tool calls</text>
<!-- ====== NODES section ====== -->
<text x="16" y="182" fill="#7aa2f7" font-size="12" font-weight="bold" class="f3" opacity="0">NODES</text>
<!-- Node column headers -->
<g transform="translate(0, 190)" class="f4" opacity="0">
<rect width="860" height="20" fill="#24283b"/>
<rect y="20" width="860" height="1" fill="#3b4261"/>
<text y="14" fill="#565f89" font-size="10" letter-spacing="0.5">
<tspan x="36">NODE</tspan>
<tspan x="560">WS</tspan>
<tspan x="610">RUN</tspan>
<tspan x="660">ATTN</tspan>
<tspan x="710">TOKENS</tspan>
<tspan x="790">LOAD</tspan>
</text>
</g>
<!-- Node rows -->
<g transform="translate(0, 214)">
<!-- Node 1: db-west-04 — 3 ws, 1 running, has-running bar -->
<g class="f5" opacity="0">
<rect y="0" width="860" height="38" fill="#1a1b26"/>
<rect y="0" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="19" r="4" fill="#9ece6a"/>
<text x="36" y="23" fill="#a9b1d6" font-size="12" font-weight="bold">db-west-04</text>
<text x="566" y="23" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="23" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="23" fill="#565f89" font-size="11">0</text>
<text x="710" y="23" fill="#565f89" font-size="11">57.6k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="15" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="15" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="23" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 2: api-east-01 — 3 ws, 1 attention, has-attention bar -->
<g class="f6" opacity="0">
<rect y="40" width="860" height="38" fill="#24283b"/>
<rect y="40" width="3" height="38" fill="#e0af68"/>
<circle cx="22" cy="59" r="4" fill="#9ece6a"/>
<text x="36" y="63" fill="#a9b1d6" font-size="12" font-weight="bold">api-east-01</text>
<text x="566" y="63" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="63" fill="#565f89" font-size="11">0</text>
<text x="666" y="63" fill="#a9b1d6" font-size="11">1</text>
<text x="710" y="63" fill="#565f89" font-size="11">109k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="55" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="55" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="63" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 3: sre-node-03 — 2 ws, 1 running, has-running bar -->
<g class="f7" opacity="0">
<rect y="80" width="860" height="38" fill="#1a1b26"/>
<rect y="80" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="99" r="4" fill="#9ece6a"/>
<text x="36" y="103" fill="#a9b1d6" font-size="12" font-weight="bold">sre-node-03</text>
<text x="566" y="103" fill="#a9b1d6" font-size="11">2</text>
<text x="616" y="103" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="103" fill="#565f89" font-size="11">0</text>
<text x="710" y="103" fill="#565f89" font-size="11">64.4k</text>
<!-- Load bar: 2/10 = 20% -->
<rect x="770" y="95" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="95" width="12" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="103" fill="#565f89" font-size="11">20%</text>
</g>
<!-- Node 4: analytics-02 — 1 ws, thinking, has-thinking bar -->
<g class="f8" opacity="0">
<rect y="120" width="860" height="38" fill="#24283b"/>
<rect y="120" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="139" r="4" fill="#9ece6a"/>
<text x="36" y="143" fill="#a9b1d6" font-size="12" font-weight="bold">analytics-02</text>
<text x="566" y="143" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="143" fill="#565f89" font-size="11">0</text>
<text x="666" y="143" fill="#565f89" font-size="11">0</text>
<text x="710" y="143" fill="#565f89" font-size="11">18.3k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="135" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="135" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="143" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 5: data-ops-05 — 1 ws, thinking, has-thinking bar -->
<g class="f9" opacity="0">
<rect y="160" width="860" height="38" fill="#1a1b26"/>
<rect y="160" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="179" r="4" fill="#9ece6a"/>
<text x="36" y="183" fill="#a9b1d6" font-size="12" font-weight="bold">data-ops-05</text>
<text x="566" y="183" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="183" fill="#565f89" font-size="11">0</text>
<text x="666" y="183" fill="#565f89" font-size="11">0</text>
<text x="710" y="183" fill="#565f89" font-size="11">8.7k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="175" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="175" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="183" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 6: ml-gpu-07 — 0 ws, empty, no bar -->
<g class="f10" opacity="0">
<rect y="200" width="860" height="38" fill="#24283b"/>
<rect y="200" width="3" height="38" fill="transparent"/>
<circle cx="22" cy="219" r="4" fill="#9ece6a"/>
<text x="36" y="223" fill="#a9b1d6" font-size="12" font-weight="bold">ml-gpu-07</text>
<text x="566" y="223" fill="#565f89" font-size="11">0</text>
<text x="616" y="223" fill="#565f89" font-size="11">0</text>
<text x="666" y="223" fill="#565f89" font-size="11">0</text>
<text x="710" y="223" fill="#565f89" font-size="11">0</text>
<!-- Load bar: 0/10 = 0% (empty track) -->
<rect x="770" y="215" width="60" height="6" rx="3" fill="#292e42"/>
<text x="842" y="223" fill="#565f89" font-size="11">0%</text>
</g>
</g>
<!-- ====== Footer ====== -->
<g transform="translate(0, 468)" class="f12" opacity="0">
<rect width="860" height="1" fill="#3b4261"/>
<rect y="1" width="860" height="24" fill="#16161e"/>
<circle cx="20" cy="14" r="3" fill="#9ece6a"/>
<text x="28" y="18" fill="#565f89" font-size="10">db-west-04</text>
<circle cx="120" cy="14" r="3" fill="#9ece6a"/>
<text x="128" y="18" fill="#565f89" font-size="10">api-east-01</text>
<circle cx="225" cy="14" r="3" fill="#9ece6a"/>
<text x="233" y="18" fill="#565f89" font-size="10">sre-node-03</text>
<circle cx="335" cy="14" r="3" fill="#9ece6a"/>
<text x="343" y="18" fill="#565f89" font-size="10">analytics-02</text>
<circle cx="450" cy="14" r="3" fill="#9ece6a"/>
<text x="458" y="18" fill="#565f89" font-size="10">data-ops-05</text>
<circle cx="560" cy="14" r="3" fill="#9ece6a"/>
<text x="568" y="18" fill="#565f89" font-size="10">ml-gpu-07</text>
<text x="680" y="18" fill="#3b4261" font-size="10">258k tokens · 42 calls · 12m</text>
</g>
<!-- Bottom edge -->
<rect y="493" width="860" height="27" fill="#16161e"/>
<rect y="510" width="860" height="10" rx="10" fill="#16161e"/>
</svg>

Before

Width:  |  Height:  |  Size: 11 KiB

+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"ddg": {
"url": "http://ddg-search:3000/mcp"
}
}
}
+315 -28
View File
@@ -56,6 +56,170 @@ console.log(result.content);
---
## Authentication
When auth is enabled (`[auth].enabled = true` or `TURNSTONE_AUTH_ENABLED=1`), all API endpoints except public paths require a valid token.
### Sending Credentials
Include a token in one of two ways:
- **Bearer header**: `Authorization: Bearer <token>`
- **Cookie**: `turnstone_auth=<token>` (set automatically by the login endpoint)
The server accepts three token types:
| Type | Format | Example |
|------|--------|---------|
| JWT | Base64 segments separated by dots | `eyJhbG...` |
| API token | `ts_` prefix + 64 hex chars | `ts_a1b2c3d4...` |
| Config token | Arbitrary string from `config.toml` | `my-secret-token` |
JWTs are the recommended credential for browser sessions. API tokens are suitable for programmatic access and CI/CD. Config tokens are a simple option for single-node deployments.
### `POST /v1/api/auth/login`
Authenticate with credentials and receive a JWT. Accepts two credential formats:
**Username + password:**
```json
{"username": "alice", "password": "hunter2"}
```
**API token:**
```json
{"token": "ts_a1b2c3d4e5f6..."}
```
**Response (success):** `200`
```json
{
"status": "ok",
"role": "full",
"scopes": "approve,read,write",
"jwt": "eyJhbGciOiJIUzI1NiIs...",
"user_id": "u_abc123"
}
```
The response also sets a `turnstone_auth` HttpOnly cookie containing the JWT.
**Response (failure):** `401`
```json
{"error": "Invalid credentials"}
```
---
### `POST /v1/api/auth/logout`
Clears the `turnstone_auth` cookie. No request body required.
**Response:** `200`
```json
{"status": "ok"}
```
The response includes a `Set-Cookie` header that expires the auth cookie.
---
### `GET /v1/api/auth/status`
Returns the current authentication state. Works with or without a valid token.
**Response (authenticated):** `200`
```json
{
"authenticated": true,
"user_id": "u_abc123",
"scopes": ["approve", "read", "write"],
"source": "jwt"
}
```
**Response (not authenticated):** `200`
```json
{
"authenticated": false,
"user_id": null,
"scopes": [],
"source": null
}
```
**Response (auth disabled):** `200`
```json
{
"authenticated": false,
"auth_enabled": false
}
```
---
### `POST /v1/api/auth/setup`
Creates the first admin user when no users exist in the database. This is a
public endpoint (no authentication required) that only succeeds when auth is
enabled and the user database is empty. Both the server and console expose
this endpoint.
**Request body:**
```json
{
"username": "admin",
"display_name": "Admin",
"password": "strongpass"
}
```
| Field | Type | Required | Validation |
|----------------|--------|----------|-----------------------------|
| `username` | string | yes | 1-64 ASCII characters |
| `display_name` | string | yes | Non-empty |
| `password` | string | yes | Minimum 8 characters |
**Response (success):** `200`
```json
{
"status": "ok",
"user_id": "u_abc123",
"username": "admin",
"role": "full",
"scopes": "approve,read,write",
"jwt": "eyJhbGciOiJIUzI1NiIs..."
}
```
The response also sets a `turnstone_auth` HttpOnly cookie containing the JWT.
**Response (already set up):** `409`
```json
{"error": "Setup already completed"}
```
Returned when one or more users already exist in the database.
**Response (auth disabled):** `400`
```json
{"error": "Auth is not enabled"}
```
---
## Endpoints
### `GET /`
@@ -284,6 +448,14 @@ after `/clear` or `/new` commands).
{"type": "clear_ui"}
```
**`cancelled`** -- the generation was cancelled by the user (via the Stop
button or `POST /v1/api/cancel`). The client should finalize any in-progress
assistant message with whatever partial content was streamed.
```json
{"type": "cancelled"}
```
#### Keepalive
The server sends an SSE comment every 5 seconds when no events are pending:
@@ -296,13 +468,13 @@ The server sends an SSE comment every 5 seconds when no events are pending:
This prevents proxies and browsers from closing the connection due to
inactivity.
#### Generation mechanism
#### Multi-consumer fan-out
Each new SSE connection to a workstream increments an internal
`_sse_generation` counter. The previous SSE handler detects the generation
mismatch and exits its event loop, ensuring only one active SSE connection per
workstream at a time. The event queue is drained of stale events before the new
connection begins streaming.
Each SSE connection to a workstream receives its own delivery queue. Events
produced by the worker thread are fanned out to all registered listener queues,
so multiple consumers (browser, bridge, console proxy, SDK) can connect
simultaneously and each receives every event. On reconnect the client receives
a full history replay, so no catch-up mechanism is needed.
---
@@ -351,8 +523,8 @@ Returns a list of all active workstreams.
```json
{
"workstreams": [
{"id": "abc123", "name": "default", "state": "idle", "session_id": "a1b2c3d4e5f6"},
{"id": "def456", "name": "hacker-news", "state": "thinking", "session_id": "c5d6e7f8a9b0"}
{"id": "abc123", "name": "default", "state": "idle"},
{"id": "def456", "name": "hacker-news", "state": "thinking"}
]
}
```
@@ -364,22 +536,21 @@ Each workstream object:
| `id` | string | Unique workstream routing identifier |
| `name` | string | Display name (alias if set, otherwise `ws-xxxx`) |
| `state` | string | Current state (see state values above) |
| `session_id` | string/null | Session ID of the workstream's `ChatSession`, used for deduplication against `/v1/api/sessions` |
---
### `GET /v1/api/sessions`
### `GET /v1/api/workstreams/saved`
Returns a list of saved sessions from the database, ordered by most recently
Returns a list of saved workstreams from the database, ordered by most recently
updated.
**Response:**
```json
{
"sessions": [
"workstreams": [
{
"session_id": "a1b2c3d4e5f6",
"ws_id": "a1b2c3d4e5f6",
"alias": "refactor",
"title": "JWT Authentication Refactor",
"created": "2026-03-01 10:00:00",
@@ -390,16 +561,16 @@ updated.
}
```
Each session object:
Each saved workstream object:
| Field | Type | Description |
|-----------------|-------------|--------------------------------------------|
| `session_id` | string | Unique 12-char hex session identifier |
| `ws_id` | string | Unique workstream identifier |
| `alias` | string/null | User-assigned short name |
| `title` | string/null | LLM-generated title |
| `created` | string | ISO timestamp of session creation |
| `created` | string | ISO timestamp of workstream creation |
| `updated` | string | ISO timestamp of last message |
| `message_count` | int | Number of messages in the session |
| `message_count` | int | Number of messages in the workstream |
---
@@ -538,6 +709,43 @@ containing the resumed session's messages.
---
### `POST /v1/api/cancel`
Cancels the active generation in a workstream. Sets a cooperative cancellation
flag that is checked at multiple points in the generation loop (per streaming
chunk, before tool execution, inside bash commands). The session transitions to
`idle` state and preserves any partial content already streamed.
If the workstream is waiting for tool approval or plan review, the pending
prompt is automatically denied/rejected to unblock the worker thread.
Calling this endpoint when the workstream is already idle is a harmless no-op.
**Request body:**
```json
{"ws_id": "abc123"}
```
| Field | Type | Required | Description |
|--------|--------|----------|----------------------|
| `ws_id`| string | yes | Target workstream ID |
**Response:**
```json
{"status": "ok"}
```
**Error responses:**
| Status | Body | Condition |
|--------|------------------------------------|------------------------|
| 400 | `{"error": "No session"}` | Session not initialized|
| 404 | `{"error": "Unknown workstream"}` | `ws_id` not found |
---
### `POST /v1/api/workstreams/new`
Creates a new workstream. The server supports up to 10 concurrent workstreams.
@@ -550,22 +758,26 @@ Creates a new workstream. The server supports up to 10 concurrent workstreams.
All fields are optional. The body can be empty or an empty JSON object.
| Field | Type | Default | Description |
|----------------|--------|---------|------------------------------------------------|
| `name` | string | auto | Workstream display name |
| `model` | string | default | Model alias from the registry (`[models.*]`) |
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| Field | Type | Default | Description |
|------------------|--------|---------|----------------------------------------------------------------|
| `name` | string | auto | Workstream display name |
| `model` | string | default | Model alias from the registry (`[models.*]`) |
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `resume_ws` | string | "" | Workstream ID to resume atomically during creation (empty = fresh)|
| `template` | string | "" | Prompt template name (replaces default templates; 400 if not found)|
**Response (success):**
```json
{"ws_id": "ghi789", "name": "ws-3"}
{"ws_id": "ghi789", "name": "ws-3", "resumed": false, "message_count": 0}
```
| Field | Type | Description |
|---------|--------|------------------------------------|
| `ws_id` | string | Unique ID of the new workstream |
| `name` | string | Auto-generated workstream name |
| Field | Type | Description |
|-----------------|--------|-----------------------------------------------------|
| `ws_id` | string | Unique ID of the new workstream |
| `name` | string | Auto-generated workstream name |
| `resumed` | bool | Whether a previous session was successfully resumed |
| `message_count` | int | Number of messages in the resumed session (0 if fresh) |
**Error (limit reached):**
@@ -608,6 +820,79 @@ Status code: `400`
---
### `GET /v1/api/watches`
List active watches on this server node. Optionally filter by workstream.
Requires `write` scope.
**Query parameters:**
| Parameter | Type | Required | Description |
|-----------|--------|----------|------------------------------------|
| `ws_id` | string | no | Filter to watches for this workstream. If omitted, returns all watches on the node. |
**Response:**
```json
{
"watches": [
{
"watch_id": "abc123def456...",
"ws_id": "ws-1",
"node_id": "host_a1b2",
"name": "pr-review",
"command": "gh pr view --json state",
"interval_secs": 300.0,
"stop_on": "data[\"state\"] == \"MERGED\"",
"max_polls": 100,
"poll_count": 5,
"last_output": "{\"state\": \"OPEN\"}",
"last_poll": "2026-03-09T12:00:00",
"next_poll": "2026-03-09T12:05:00",
"active": 1,
"created": "2026-03-09T11:30:00"
}
]
}
```
---
### `POST /v1/api/watches/{watch_id}/cancel`
Cancel an active watch. Sets `active=0` and clears `next_poll`.
Requires `write` scope. Verifies node ownership in multi-node deployments.
**Path parameters:**
| Parameter | Type | Description |
|------------|--------|-----------------|
| `watch_id` | string | Watch ID to cancel |
**Response (success):**
```json
{"status": "ok", "watch_id": "abc123def456..."}
```
**Error (not found):**
```json
{"error": "Watch not found"}
```
Status code: `404`
**Error (wrong node):**
```json
{"error": "Watch belongs to another node"}
```
Status code: `403`
---
### `OPTIONS` (any path)
Handles CORS preflight requests.
@@ -692,7 +977,8 @@ liveness probes.
```json
{
"status": "ok",
"version": "0.3.0",
"version": "0.4.0",
"node_id": "worker-01_a3f2",
"uptime_seconds": 3614.72,
"model": "llama-3.1-70b-instruct",
"workstreams": {
@@ -714,6 +1000,7 @@ liveness probes.
|-------|------|-------------|
| `status` | string | `"ok"` or `"degraded"` (degraded when backend unreachable) |
| `version` | string | turnstone server version |
| `node_id` | string | Server-generated node identity (`{hostname}_{4hex}`) |
| `uptime_seconds` | number | Seconds since the server process started |
| `model` | string | Model name detected or configured at startup |
| `workstreams.total` | integer | Total active workstreams |
+288 -63
View File
@@ -3,7 +3,7 @@
Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent
memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or
Anthropic's native Messages API via pluggable provider adapters, and gives the
model 14 built-in tools plus external tools via MCP (Model Context Protocol) for
model 18 built-in tools plus external tools via MCP (Model Context Protocol) for
reading, writing, searching, planning, and executing code.
The core design principle is a **UI-agnostic engine with pluggable frontends**.
@@ -21,6 +21,8 @@ plugs in.
| `turnstone-bridge` | `turnstone.mq.bridge` | Bridge | Message queue ↔ HTTP API bridge |
| `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
| `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) via Redis MQ |
| `turnstone-admin` | `turnstone.core.admin_cli` | — | Offline user and API token management |
---
@@ -40,7 +42,9 @@ turnstone/
__init__.py create_provider() + create_client() factory functions
workstream.py Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
tools.py Tool schema loader (JSON -> OpenAI function-calling format)
mcp_client.py MCPClientManager — MCP server connections, tool discovery, async-sync bridge
mcp_client.py MCPClientManager — MCP server connections, tool discovery, dynamic refresh, async-sync bridge
tool_search.py Dynamic tool search — BM25 index, session-scoped tool visibility
watch.py WatchRunner daemon — periodic command polling, condition DSL, result dispatch
model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
memory.py Persistence facade (delegates to storage backend)
storage/ Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
@@ -73,8 +77,15 @@ turnstone/
client.py TurnstoneClient library + TurnResult for MQ-based access
console/
collector.py ClusterCollector — aggregates state from all nodes via Redis + HTTP
scheduler.py TaskScheduler — background cron/at scheduler, dispatches via MQ
server.py Cluster dashboard HTTP server + SSE + CLI entry point
static/ Cluster dashboard web UI (page-specific HTML, CSS, JS)
channels/
cli.py Unified channel gateway entry point (turnstone-channel)
_protocol.py ChannelAdapter protocol, ChannelEvent dataclass
_routing.py ChannelRouter — channel/thread ↔ workstream mapping via MQ
_config.py Base ChannelConfig dataclass
discord/ Discord adapter (bot, cog, views, streaming, config)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
ui/
colors.py ANSI color constants with NO_COLOR support
@@ -85,7 +96,7 @@ turnstone/
style.css Page-specific UI styles (dashboard layout, approval blocks)
app.js Page-specific client-side JavaScript (SSE, workstreams, markdown)
tools/
*.json 14 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 15 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -118,6 +129,7 @@ A user message flows through the system as follows:
| on_reasoning_token() / on_content_token()
| accumulate tool_calls from deltas
| track finish_reason
| _check_cancelled() per chunk (cooperative cancel)
v
finish_reason check:
+--- "length" --> warn, discard partial tool_calls
@@ -163,11 +175,13 @@ Phase 2: APPROVE (serial, blocking)
_emit_state("running")
Phase 3: EXECUTE (parallel)
_check_cancelled() <-- cancellation checkpoint before execution starts
if len(items) == 1:
run_one(items[0])
else:
ThreadPoolExecutor(max_workers=4).map(run_one, items)
Bash tool streams stdout line-by-line via ui.on_tool_output_chunk(call_id, line)
(cancel_event also checked per line — kills process group on cancel)
Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
call_id links tool_info items → streaming chunks → final result
For plan tool: post-execution gate via ui.on_plan_review()
@@ -198,6 +212,11 @@ The engine emits state changes via `_emit_state()` which calls
"idle" ---> no more tool calls, turn complete
|
(or "error" ---> exception or KeyboardInterrupt)
cancel() may be called from any state. It sets a cooperative flag
checked at each streaming chunk, before tool execution, and inside
bash commands. The session transitions to "idle" with partial
content preserved, emitting on_info("[Generation cancelled]").
```
---
@@ -463,7 +482,7 @@ independently, then returns the final content as the tool result.
- **task**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
to `.plan-<session_id>.md` — unique per `ChatSession` so concurrent workstreams
to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
don't collide. On repeat invocations the prior `plan` tool call and its result
are forwarded from `self.messages` so the agent refines the existing plan rather
than starting over. Planning instructions are injected as a developer message
@@ -488,17 +507,32 @@ bridges this with a background asyncio event loop in a daemon thread.
1. `create_mcp_client()` reads server configs from TOML or JSON
2. `MCPClientManager.start()` launches the background event loop thread
3. `_connect_all()` connects to each server (stdio subprocess or HTTP), runs
`initialize()` + `list_tools()`, converts schemas to OpenAI format
4. `ChatSession.__init__` receives the manager and builds `self._tools` (built-in + MCP)
`initialize()` + `list_tools()`, converts schemas to OpenAI format, detects
`tools.listChanged` capability for push notification support
4. `ChatSession.__init__` receives the manager, builds `self._tools` (built-in + MCP),
and registers a listener callback for tool-change notifications
5. `_prepare_tool()` routes MCP tools to `_prepare_mcp_tool()` / `_exec_mcp_tool()`
6. `_exec_mcp_tool()` calls `call_tool_sync()` which dispatches to the async loop
via `asyncio.run_coroutine_threadsafe()`
**Tool refresh:** Three mechanisms keep tools up-to-date without restart:
- **Push:** Servers declaring `tools.listChanged` send `ToolListChangedNotification`;
the registered `message_handler` triggers immediate single-server refresh.
- **Periodic:** Servers without push support are polled on a staggered interval
(default 4 h, configurable via `[mcp] refresh_interval` or `--mcp-refresh-interval`).
- **Manual:** `/mcp refresh [server]` calls `refresh_sync()` for on-demand refresh
(also attempts reconnection for disconnected servers).
When tools change, `_rebuild_tools()` creates new `_tools`/`_tool_map` objects
(copy-on-write for thread safety) and notifies listener callbacks. Each `ChatSession`
rebuilds its merged tool lists and reconstructs `ToolSearchManager` (preserving
expanded tools).
**Tool naming:** `mcp__{server}__{tool}` — double underscore delimiter, validated
at connection time (server names with `__` are rejected).
**Error isolation:** Per-server connection failures are caught and logged; other
servers still connect. Tool execution errors return error strings to the LLM
**Error isolation:** Per-server connection/refresh failures are caught and logged; other
servers are unaffected. Tool execution errors return error strings to the LLM
rather than crashing the session.
### Provider Adapter Layer
@@ -535,21 +569,23 @@ LLMProvider (protocol)
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens` |
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
already in OpenAI format). Model capability lookup table covers
GPT-5/5.1/5.2, O-series, and search models (`gpt-5-search-api`).
already in OpenAI format), including multi-part content blocks (text + images)
in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4,
O-series, and search models (`gpt-5-search-api`) — all with `supports_vision`.
For search models, injects `web_search_options` and removes the `web_search`
function tool (the model always searches). Citations from `url_citation`
annotations are formatted as footnotes. Unknown models (local servers) get
permissive defaults and use Tavily for web search.
permissive defaults with `supports_vision=False` and use Tavily for web search.
**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
Anthropic content blocks, maps `system`/`developer` roles to the `system`
parameter, groups consecutive `tool` result messages into user-role content
blocks, and translates tool schemas from OpenAI function-calling format to
blocks (converting `image_url` parts to Anthropic's `image` source format),
and translates tool schemas from OpenAI function-calling format to
Anthropic's `input_schema` format. Supports both manual and adaptive thinking
modes, with effort parameter support for models like Claude Opus 4.6 and
Sonnet 4.6. Replaces the `web_search` function tool with Anthropic's native
@@ -595,6 +631,18 @@ agent_model = "claude"
Each `[models.*]` entry produces a `ModelConfig` with a `provider` field
(default: `"openai"`). Supported values: `"openai"` and `"anthropic"`.
An optional `[models.*.capabilities]` sub-table overrides per-model
`ModelCapabilities` flags (useful for local models whose capabilities
cannot be detected programmatically):
```toml
[models.qwen-vl]
base_url = "http://localhost:8000/v1"
model = "qwen-3.5-vl"
[models.qwen-vl.capabilities]
supports_vision = true
```
**Lifecycle:**
1. `load_model_registry()` reads `[models.*]` sections from config.toml and
@@ -679,8 +727,11 @@ memories
created TEXT NOT NULL
updated TEXT NOT NULL
sessions
session_id TEXT PRIMARY KEY
workstreams
ws_id TEXT PRIMARY KEY
node_id TEXT NOT NULL
name TEXT NOT NULL
state TEXT NOT NULL DEFAULT 'idle'
alias TEXT UNIQUE -- user-assigned short name (nullable)
title TEXT -- LLM-generated title (nullable)
created TEXT NOT NULL
@@ -688,7 +739,7 @@ sessions
conversations
id INTEGER PRIMARY KEY AUTOINCREMENT
session_id TEXT NOT NULL
ws_id TEXT NOT NULL
timestamp TEXT NOT NULL
role TEXT NOT NULL -- user | assistant | tool_call | tool_result
content TEXT
@@ -697,8 +748,8 @@ conversations
tool_call_id TEXT -- links tool_call ↔ tool_result for resume
provider_data TEXT -- raw provider content (e.g. Anthropic encrypted)
session_config
session_id TEXT NOT NULL -- composite PK with key
workstream_config
ws_id TEXT NOT NULL -- composite PK with key
key TEXT NOT NULL
value TEXT
@@ -713,18 +764,21 @@ and are the single source of truth for both backends and Alembic migrations.
| Method | Purpose |
|--------|---------|
| `register_session(session_id, title)` | Create a sessions row (no-op if exists) |
| `save_message(session_id, role, content, ...)` | Log a message to conversations |
| `load_session_messages(session_id)` | Reconstruct OpenAI message format from DB rows |
| `list_sessions(limit)` | List sessions with >=1 message, ordered by updated DESC |
| `delete_session(session_id)` | Delete session and all its messages |
| `prune_sessions(retention_days)` | Remove empty sessions and old unnamed sessions |
| `resolve_session(alias_or_id)` | Resolve alias, exact id, or id prefix to full session_id |
| `save_session_config(session_id, config)` | Persist session configuration key/value pairs |
| `load_session_config(session_id)` | Retrieve session configuration |
| `set_session_alias(session_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_session_name(session_id)` | Return alias if set, else title, else None |
| `update_session_title(session_id, title)` | Set/update LLM-generated title |
| `register_workstream(ws_id, node_id, name, state)` | Create a workstreams row (no-op if exists) |
| `save_message(ws_id, role, content, ...)` | Log a message to conversations |
| `load_messages(ws_id)` | Reconstruct OpenAI message format from DB rows |
| `list_workstreams_with_history(limit)` | List workstreams with >=1 message, ordered by updated DESC |
| `delete_workstream(ws_id)` | Delete workstream and cascade conversations + config |
| `prune_workstreams(retention_days)` | Remove empty workstreams and old unnamed workstreams |
| `resolve_workstream(alias_or_id)` | Resolve alias, exact id, or id prefix to full ws_id |
| `save_workstream_config(ws_id, config)` | Persist workstream configuration key/value pairs |
| `load_workstream_config(ws_id)` | Retrieve workstream configuration |
| `set_workstream_alias(ws_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_workstream_display_name(ws_id)` | Return alias if set, else title, else None |
| `update_workstream_title(ws_id, title)` | Set/update LLM-generated title |
| `update_workstream_state(ws_id, state)` | Update workstream state and bump timestamp |
| `update_workstream_name(ws_id, name)` | Update workstream display name |
| `list_workstreams(node_id, limit)` | List workstreams, optionally by node |
| `kv_get(key)` / `kv_set(key, value)` / `kv_delete(key)` | Generic key-value store (backs memories table) |
| `kv_list()` / `kv_search(query)` | List or search key-value pairs |
| `search_history(query, limit)` | Full-text search (FTS5 on SQLite, tsvector on PostgreSQL) |
@@ -743,57 +797,59 @@ pool_size = 5 # PostgreSQL connection pool size
Environment variables: `TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`, `TURNSTONE_DB_PATH`.
### Session Persistence and Resume
### Persistence and Resume
Each `ChatSession` generates a 12-char hex `_session_id` on creation and
registers it in the `sessions` table. Messages are saved to `conversations`
as they happen via `save_message()`.
`ws_id` is the sole persistent identity for both routing and conversation
history. There is no separate `session_id` — the `workstreams` table holds
alias, title, and state alongside the routing fields (`node_id`, `name`).
Messages are saved to `conversations` (keyed by `ws_id`) as they happen
via `save_message()`. Workstream state changes are tracked via
`update_workstream_state()`.
**Auto-titling:** After the first complete exchange (user message + assistant
response), a background thread calls the LLM with a title-generation prompt
(`reasoning_effort: "low"`, `max_completion_tokens: 200`). The generated
title (3-8 words) is stored in `sessions.title`.
title (3-8 words) is stored in `workstreams.title`.
**Resume flow:** `ChatSession.resume_session(session_id)` calls
`load_session_messages()` which reconstructs the OpenAI message format from
database rows:
**Resume flow:** `ChatSession.resume(ws_id)` calls `load_messages()` which
reconstructs the OpenAI message format from database rows:
- `user` and `assistant` rows map directly
- Consecutive `tool_call` rows are grouped into one assistant message's
`tool_calls` array, paired with subsequent `tool_result` rows via
`tool_call_id` (or positional matching for legacy data)
- **Interrupted session repair:** If the last assistant message has
`tool_calls` but fewer tool results than expected (session was
- **Interrupted conversation repair:** If the last assistant message has
`tool_calls` but fewer tool results than expected (conversation was
interrupted mid-execution), the incomplete turn is stripped so the
LLM can re-generate cleanly
- The session adopts the old `_session_id`, so new messages continue in
the same session
- The `ChatSession` adopts the resumed `_ws_id`, so new messages continue
in the same workstream
**Config persistence:** LLM-affecting parameters (`temperature`,
`reasoning_effort`, `max_tokens`, `instructions`, `creative_mode`) are
persisted to the `session_config` table on creation and whenever changed
via slash commands. `resume_session()` restores these values so resumed
sessions behave identically to the original.
persisted to the `workstream_config` table on creation and whenever changed
via slash commands. `resume()` restores these values so resumed workstreams
behave identically to the original.
**`/clear` vs `/new`:** `/clear` wipes in-memory context but preserves
messages in the database for future resume. `/new` starts a fresh session
(new `_session_id`), leaving the old session resumable.
messages in the database for future resume. `/new` starts a fresh workstream
(new `_ws_id`), leaving the old workstream resumable.
**Resolution:** `resolve_session()` accepts aliases, exact session IDs, or
session ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.
**Resolution:** `resolve_workstream()` accepts aliases, exact workstream IDs,
or ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.
**Session listing:** `list_sessions()` only returns sessions that have at
least one saved message (`WHERE EXISTS` on `conversations`). Sessions
registered but never used (e.g., from process startup) are invisible until
a message is sent.
**Workstream listing:** `list_workstreams_with_history()` only returns
workstreams that have at least one saved message (`WHERE EXISTS` on
`conversations`). Workstreams registered but never used (e.g., from process
startup) are invisible until a message is sent.
**Session pruning:** `prune_sessions(retention_days, log_fn)` runs once at
startup (CLI and server). It removes:
- Sessions with no messages (orphaned registrations)
- Unnamed sessions (`alias IS NULL`) older than `retention_days` days (default 90)
**Workstream pruning:** `prune_workstreams(retention_days, log_fn)` runs once
at startup (CLI and server). It removes:
- Workstreams with no messages (orphaned registrations)
- Unnamed workstreams (`alias IS NULL`) older than `retention_days` days (default 90)
Named (aliased) sessions are never age-pruned. Configure with
`--session-retention-days N` (0 = disable age pruning).
Named (aliased) workstreams are never age-pruned. Configure with
`--retention-days N` (0 = disable age pruning).
---
@@ -910,6 +966,98 @@ limits using a token-bucket algorithm. Each IP gets a `TokenBucket` with
---
## User Identity and Authentication
Turnstone supports three authentication mechanisms, unified behind an
`AuthResult` dataclass that carries `user_id`, `scopes`, and `token_source`:
1. **Config-file tokens** — static secrets in `config.toml` `[[auth.tokens]]`
or the `TURNSTONE_AUTH_TOKEN` env var. Validated in-memory via
`hmac.compare_digest`. Map to scopes through their role (`read` or `full`).
2. **API tokens** — database-backed, prefixed `ts_`, stored as SHA-256 hashes
in the `api_tokens` table. Can be exchanged for JWTs via
`POST /v1/api/auth/login`.
3. **JWTs** — short-lived HMAC-SHA256 session tokens (default 24h) issued after
successful credential validation. Contain `sub` (user_id), `scopes`, and
`src` (origin) in claims.
### Scope Model
Three hierarchical scopes control endpoint access:
| Scope | Grants | Endpoints |
|-------|--------|-----------|
| `read` | SSE streams, workstream listing, history | GET endpoints |
| `write` | `read` + send, command, workstream create/close | POST to `/api/send`, `/api/command`, etc. |
| `approve` | `write` + tool approval, admin operations | POST to `/api/approve`, `/api/admin/*` |
### Middleware Flow
`AuthMiddleware` (ASGI) intercepts every request:
1. **Public path check**`/`, `/static/*`, `/shared/*`, `/health`,
`/metrics`, `/openapi.json`, `/docs`, `/api/auth/*`, and `/api/auth/setup`
are always allowed.
2. **Token extraction**`Authorization: Bearer <token>` header first, then
`turnstone_auth` cookie as fallback.
3. **Token type detection** — dots in the token indicate JWT; `ts_` prefix
indicates API token; otherwise config-file token.
4. **Validation** — JWT signature check, API token hash lookup in storage, or
config-token hmac comparison.
5. **Scope check**`required_scope(method, path)` determines the minimum
scope; the request is rejected with 403 if the token lacks it.
6. **Context propagation** — on success, `ctx_user_id` is set so structured
logging includes the authenticated identity on every log event.
### Architecture Split
- **Console** is the auth management hub — it hosts the admin endpoints for
creating users, issuing API tokens, and managing channel mappings. User
records and token hashes live in the shared storage backend. The console
dashboard includes an **admin panel** (Users and Tokens tabs) for managing
credentials through the browser.
- **Server** is a JWT validator only — it validates tokens on each request but
never creates users or tokens. Both processes share the same `jwt_secret`
(via `TURNSTONE_JWT_SECRET` env var or `[auth].jwt_secret` config).
- **First-time setup** — both server and console expose
`POST /v1/api/auth/setup`, a public endpoint that creates the initial admin
user when no users exist. This avoids the chicken-and-egg problem of needing
`approve` scope to create the first user via `/api/admin/users`.
### Auth Storage Tables
Three tables in `storage/_schema.py` support identity:
```sql
users
user_id TEXT PRIMARY KEY
username TEXT NOT NULL UNIQUE
display_name TEXT NOT NULL
password_hash TEXT NOT NULL -- bcrypt
created TEXT NOT NULL
api_tokens
token_id TEXT PRIMARY KEY
token_hash TEXT NOT NULL UNIQUE -- SHA-256 of raw token
token_prefix TEXT NOT NULL -- first 8 chars for display
user_id TEXT NOT NULL
name TEXT NOT NULL -- human-readable label
scopes TEXT NOT NULL -- comma-separated
created TEXT NOT NULL
expires TEXT -- optional expiry timestamp
channel_users
channel_type TEXT NOT NULL -- e.g. "slack", "discord"
channel_user_id TEXT NOT NULL -- platform-specific user ID
user_id TEXT NOT NULL -- FK to users
PRIMARY KEY (channel_type, channel_user_id)
```
See [docs/security.md](security.md) for full security details including token
lifecycle, password hashing, and deployment hardening.
---
## Threading Model
### CLI
@@ -973,7 +1121,7 @@ context manager handles startup/shutdown (health monitor, MCP client,
registry).
Each workstream's `WebUI` has:
- `_event_queue` (per-workstream SSE events, `queue.Queue`)
- `_listeners` (per-client SSE queues, fan-out on `_enqueue()`)
- `_approval_event` / `_plan_event` (`threading.Event` for blocking)
- `_global_queue` (class variable, shared, for state broadcasts)
@@ -1033,17 +1181,27 @@ bridge auto-approves via `POST /v1/api/approve`. Otherwise, it publishes an
`BLPOP` of a Redis response queue (`turnstone:resp:{request_id}`) until the client pushes
a response or the approval timeout (default 3600s / 1 hour) expires.
**Cancellation:** The `CancelMessage` (type `"cancel"`) is a routed inbound message.
The bridge dispatches it to `POST /v1/api/cancel` on the server owning the workstream,
which sets the cooperative cancel flag and unblocks any pending approval/plan waits.
**Completion detection:** The bridge tracks which `correlation_id` maps to which
`ws_id` for active sends. When the global SSE reports `ws_state → idle` for a tracked
workstream, the bridge emits a synthetic `TurnCompleteEvent` with the correlation ID.
**Multi-node routing:** Each bridge has a `node_id` (defaults to hostname) and BLPOPs
**Multi-node routing:** Each bridge retrieves its `node_id` from the server's
`/health` endpoint on startup (with exponential backoff retry). The server
generates the `node_id` (`{hostname}_{4hex}`) and is the sole authority for
node identity. The bridge BLPOPs
from both `turnstone:inbound:{node_id}` (directed, priority) and `turnstone:inbound` (shared).
Messages with `target_node` set are pushed to the target's per-node queue. Messages
for existing workstreams are auto-routed via `turnstone:ws:{ws_id}` ownership keys in Redis.
If a bridge picks up a shared-queue message for a workstream owned by another node, it
re-routes to that node's queue (1 extra hop). Bridges publish heartbeats to
`turnstone:node:{node_id}` with configurable TTL for node discovery.
On startup, `_recover_workstreams` re-registers ownership of existing
workstreams and publishes `WorkstreamCreatedEvent` to the cluster channel
so the console collector picks them up immediately.
### Cluster Console
@@ -1069,7 +1227,10 @@ The console HTTP layer is a Starlette/ASGI app served by uvicorn. The SSE
endpoint uses `EventSourceResponse` with the same listener queue pattern as
the main server. `ClusterCollector`'s background threads (event subscriber,
node discovery, poll loop) use sync Redis clients and `ThreadPoolExecutor`
for parallel HTTP polling.
for parallel HTTP polling. The poll loop diffs workstream IDs between poll
cycles and fans out synthetic `ws_created`/`ws_closed` SSE events for any
changes, ensuring browser clients stay in sync even when real-time cluster
events are missed (e.g. bridge startup recovery).
The console has two write-path capabilities:
@@ -1128,7 +1289,7 @@ typed event dataclasses.
**Two client pairs** (sync + async):
- `TurnstoneServer` / `AsyncTurnstoneServer` — server API (workstreams, chat, streaming, sessions)
- `TurnstoneServer` / `AsyncTurnstoneServer` — server API (workstreams, chat, streaming)
- `TurnstoneConsole` / `AsyncTurnstoneConsole` — console API (cluster overview, nodes, workstreams)
**Design**: async-first with thin sync wrappers. `_BaseClient` provides httpx
@@ -1152,3 +1313,67 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
result = client.send_and_wait("Hello!", ws.ws_id)
print(result.content)
```
---
## Channel Integrations
> See also: [Channel Integrations guide](channels.md)
The `turnstone-channel` gateway bridges external messaging platforms
(Discord, Slack, Teams) to the turnstone cluster via Redis MQ. Each
platform adapter implements the `ChannelAdapter` protocol and translates
between platform-native events and turnstone MQ messages.
The `ChannelRouter` manages bidirectional routing: it maps platform
channel/thread IDs to turnstone workstream IDs, handles workstream
creation and stale-route recovery, and resolves platform users to
turnstone identities via the `channel_users` table. When an evicted
workstream is reactivated, the router uses atomic resume via the
`resume_ws` field on `CreateWorkstreamMessage` — the server resumes
the old workstream's conversation during creation in a single HTTP
request, eliminating ordering fragility. The bridge emits a
`WorkstreamResumedEvent` to confirm success.
Discord ships as the first adapter. See [channels.md](channels.md) for
setup instructions, configuration reference, and the adapter development
guide.
### Notification Subsystem
The `notify` tool enables the LLM to send notifications to users or
channels without going through MQ. The server calls the channel gateway
directly over HTTP for lower latency: `_exec_notify()` queries the
`services` database table for healthy channel gateways (heartbeat within
120 seconds), authenticates with a service JWT (`aud: turnstone-channel`),
and POSTs to `POST /v1/api/notify` on the first healthy gateway. The
gateway validates the JWT, resolves the target (username lookup via
`channel_users` or direct `channel_type`+`channel_id`), and delegates to
the appropriate `ChannelAdapter.send()`. Delivery retries up to 3 times
with backoff, re-querying the service registry on each attempt. See
[Notification Flow diagram](diagrams/png/17-notify-flow.png).
---
## Governance
> See also: [Governance documentation](governance.md) | [Governance Architecture diagram](diagrams/19-governance-architecture.puml)
Turnstone governance extends the Phase 1 auth system with role-based access
control (RBAC), tool execution policies, prompt templates, usage tracking,
and audit logging. The permission model has two layers: legacy scopes
(`read`, `write`, `approve`) checked by `AuthMiddleware`, and 15 granular
permissions checked per-endpoint by `require_permission()`. Three built-in
roles (admin, operator, viewer) are seeded by migration 008; custom roles
can be created with any permission subset. JWTs carry both `scopes` and
`permissions` claims for backward compatibility.
Tool policies use glob pattern matching (`fnmatch`) with priority-ordered
first-match-wins evaluation to control tool execution (allow/deny/ask).
Prompt templates provide reusable system messages with `{{variable}}`
substitution. Usage events are recorded per-LLM-request for token
accounting. An append-only audit log captures all admin mutations.
The console admin panel adds 5 governance tabs (Roles, Policies, Templates,
Usage, Audit) for a total of 10 tabs, all permission-gated. Both Python
and TypeScript SDKs expose governance methods on the console client.
+346
View File
@@ -0,0 +1,346 @@
# Channel Integrations
The `turnstone-channel` gateway connects external messaging platforms to
turnstone workstreams via Redis MQ. Each platform adapter translates
platform-native events (messages, button clicks, slash commands) into
turnstone MQ messages, and renders workstream output back into the
platform's UI.
Discord ships as the first adapter. The adapter protocol is designed for
future Slack and Teams integrations.
---
## Architecture
```
Discord Gateway
|
v
turnstone-channel (Discord adapter)
|
v
Redis MQ
|
v
turnstone-bridge ──> turnstone-server
```
Key components:
- **ChannelAdapter protocol** (`turnstone/channels/_protocol.py`) — generic
interface for any messaging platform. Defines `start()`, `stop()`,
`send()`, `edit_message()`, `send_approval_request()`,
`send_plan_review()`, and `create_thread()`.
- **ChannelRouter** (`turnstone/channels/_routing.py`) — maps
channel/thread IDs to turnstone workstream IDs. Handles workstream
creation via MQ, stale route detection, and user identity resolution.
- **AsyncRedisBroker** (`turnstone/mq/async_broker.py`) — async Redis
client compatible with discord.py's event loop. Used by the router for
pub/sub and queue operations.
- **channel_users table** — maps `(channel_type, channel_user_id)` to a
turnstone `user_id`. Messages from unlinked users are silently dropped.
- **channel_routes table** — persistent channel-to-workstream mappings.
Survives bot restarts. Stale routes (evicted workstreams) are detected
and refreshed on the next message.
---
## Discord Setup
### 1. Create a Discord Application
1. Go to https://discord.com/developers/applications
2. Click **New Application** and give it a name
3. Navigate to the **Bot** tab and click **Reset Token** to generate a
bot token. Copy it immediately — it is shown only once.
4. On the same **Bot** tab, scroll down to **Privileged Gateway Intents**
and enable **MESSAGE CONTENT INTENT**
5. Navigate to **OAuth2 > URL Generator**
6. Under **Scopes**, check `bot` and `applications.commands`
7. Under **Bot Permissions**, check:
- View Channels
- Send Messages
- Send Messages in Threads
- Create Public Threads
- Read Message History
- Add Reactions
- Embed Links
8. Copy the generated URL, open it in a browser, and add the bot to your
Discord server
### 2. Configure Turnstone
**Environment variables** (recommended for Docker):
```bash
TURNSTONE_DISCORD_TOKEN=your-bot-token-here
TURNSTONE_DISCORD_GUILD=123456789 # optional, restrict to one guild
```
**CLI flags** (bare-metal):
```bash
turnstone-channel \
--discord-token "your-bot-token" \
--discord-guild 123456789 \
--redis-host localhost \
--redis-port 6379
```
**Docker Compose** (production profile):
```bash
# In .env file:
TURNSTONE_DISCORD_TOKEN=your-bot-token
TURNSTONE_DISCORD_GUILD=123456789
```
Then start the stack:
```bash
docker compose --profile production up
```
The `channel` service starts automatically when
`TURNSTONE_DISCORD_TOKEN` is set.
### 3. Link User Accounts
Discord users must link their account to a turnstone user before they can
interact with the bot. Unlinked users' messages are silently ignored.
1. The user must have a turnstone API token — created via the admin panel
or `turnstone-admin create-token`
2. In Discord, the user runs `/link`. A modal appears prompting for the
API token (the token is never visible in Discord audit logs because it
is submitted via modal, not as a slash command argument).
3. The token is validated against the database. If valid, a
`channel_users` mapping is created.
4. The user can now @mention the bot or use slash commands.
An admin can also force-link or unlink users via the console admin panel
(Admin > Channels tab).
---
## Usage
### Conversations
- **@mention** the bot in any allowed channel to start a new conversation.
The bot creates a Discord thread from the message and a turnstone
workstream behind it.
- All subsequent messages in the thread are routed to the same workstream.
- The bot streams responses via message edits, updated approximately every
1.5 seconds.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the bridge emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
### Slash Commands
| Command | Description |
|---------|-------------|
| `/link` | Link Discord account to turnstone (opens modal for API token) |
| `/unlink` | Unlink Discord account |
| `/ask <message>` | Create a new thread and workstream with an initial message |
| `/status` | Show workstream info for the current thread (ephemeral) |
| `/close` | Close the workstream, delete the route, and archive the thread |
### Tool Approvals
When manual approval is enabled (the default), tool calls are displayed as
an orange embed with:
- Tool name and argument preview
- **Approve** (green), **Reject** (red), **Always Approve** (gray) buttons
- Only linked users can interact with approval buttons
- The approval decision is forwarded through MQ to the bridge, which
relays it to the server
Buttons use static `custom_id` values so they survive bot restarts.
Correlation data (`ws_id`, `correlation_id`) is stored in the embed footer.
**Auto-approval:** When `auto_approve` is true (via `--auto-approve`), or when
all tools in the request match the `auto_approve_tools` list in the adapter
config, the bot auto-responds with approval and posts a
"*Tool auto-approved.*" notice to the thread instead of showing buttons. The
`auto_approve_tools` list is set via the `ChannelConfig.auto_approve_tools`
field (useful for allowing specific tools like `bash` or `read_file` while
still requiring manual approval for others).
### Plan Reviews
Plan review requests are displayed as a blue embed with:
- **Approve Plan** (green) button — approves the plan with empty feedback
- **Request Changes** (gray) button — opens a modal for feedback text
(up to 2000 characters)
- Feedback is forwarded through MQ as a `PlanFeedbackMessage`
---
## Configuration Reference
| CLI Flag | Env Var | Default | Description |
|----------|---------|---------|-------------|
| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Bot token (required to enable Discord) |
| `--discord-guild` | — | `0` (all guilds) | Restrict to a single Discord guild |
| `--discord-channels` | — | empty (all) | Comma-separated channel IDs to allow |
| `--redis-host` | `REDIS_HOST` | `localhost` | Redis host |
| `--redis-port` | — | `6379` | Redis port |
| `--redis-password` | `REDIS_PASSWORD` | — | Redis password |
| `--redis-db` | — | `0` | Redis DB number |
| `--model` | — | server default | Default model for new workstreams |
| `--auto-approve` | — | `false` | Auto-approve ALL tool calls (skips approval buttons entirely) |
| `--http-host` | — | `127.0.0.1` | HTTP server bind address for notify endpoint |
| `--http-port` | `TURNSTONE_CHANNEL_PORT` | `8091` | HTTP server port |
| `--auth-token` | `TURNSTONE_CHANNEL_AUTH_TOKEN` | — | Static auth token for `/v1/api/notify` (alternative to JWT) |
| `--log-level` | `TURNSTONE_LOG_LEVEL` | `INFO` | Log level |
| `--log-format` | `TURNSTONE_LOG_FORMAT` | `auto` | Log format (`auto`/`json`/`text`) |
---
## User Identity
- The `channel_users` table maps `(channel_type, channel_user_id)` to a
turnstone `user_id`
- Self-service linking via the `/link` slash command (modal input, not
visible in Discord audit logs)
- Admin can force-link or unlink via the console admin panel (Admin >
Channels tab). Unlinking uses a styled confirmation modal.
- Unlinked users' messages are silently dropped
- A user can be linked across multiple platforms (e.g. Discord + Slack)
See [Security: Database Schema](security.md#database-schema) for the
`channel_users` table definition.
---
## Workstream Lifecycle
1. **Creation**@mention or `/ask` creates a Discord thread and a
turnstone workstream. The `ChannelRouter` persists the mapping in the
`channel_routes` table.
2. **Active** — messages are routed bidirectionally. The bot streams
responses via message edits (updated every ~1.5 seconds).
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route (no MQ owner) and creates a new workstream with the old `ws_id`
as `resume_ws` on the `CreateWorkstreamMessage`. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The bridge emits a `WorkstreamResumedEvent` to the channel, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
5. **Close**`/close` command closes the workstream via MQ, deletes the
route, unsubscribes from events, and archives the Discord thread.
---
## Notifications
> See also: [Notification Flow diagram](diagrams/png/17-notify-flow.png)
The `notify` tool allows the LLM to proactively send notifications to
users or channels on external platforms. This is useful for alerting
people about task completion, errors, or important updates without
waiting for them to check in.
### Targeting
Two modes:
- **Username** — provide a turnstone `username`. The gateway resolves
it via the `channel_users` table and sends to all linked channels
(e.g. Discord + future Slack).
- **Direct** — provide `channel_type` + `channel_id` to target a
specific platform channel or user DM.
### Delivery Flow
Notifications bypass MQ for lower latency. The server calls the channel
gateway directly over HTTP:
1. The LLM calls the `notify` tool with a message and target
2. `_exec_notify()` queries the `services` table for healthy channel
gateways (heartbeat within the last 120 seconds)
3. The server mints a service JWT (`aud: turnstone-channel`) via
`ServiceTokenManager` and POSTs to the first healthy gateway
4. The gateway validates the JWT, resolves the target, and calls
`adapter.send()` on the appropriate platform adapter
5. On failure, the server tries the next gateway. If all fail, it
retries up to 2 more times (delays: 1s, 3s), re-querying the
service registry on each attempt
### Service Registry
The channel gateway registers itself in the `services` database table
on startup and sends a heartbeat every 30 seconds. On shutdown it
deregisters. Services are considered stale after 120 seconds (4 missed
heartbeats) and are excluded from `list_services()` queries.
The `services` table schema:
| Column | Description |
|--------|-------------|
| `service_type` | Service category (e.g. `"channel"`) |
| `service_id` | Unique instance ID (`channel-<hostname>-<random>`) |
| `url` | HTTP base URL for the service |
| `last_heartbeat` | ISO 8601 timestamp of last heartbeat |
| `created` | ISO 8601 timestamp of initial registration |
### Security
- **Authentication** — the gateway's `POST /v1/api/notify` endpoint
requires authentication. Configure either `TURNSTONE_JWT_SECRET`
(the server mints JWTs with `aud: turnstone-channel` automatically)
or a static token via `--auth-token`. If neither is set, the
gateway fails closed and rejects all requests with 401. Server JWTs
(`aud: turnstone-server`) are rejected.
- **Rate limit** — maximum 5 notifications per turn. The counter only
increments on successful delivery, so failures don't consume the
budget.
- **SSRF protection** — only `http://` and `https://` service URLs
are allowed. Other schemes are silently skipped.
- **Mention sanitization** — `discord.utils.escape_mentions()` is
applied before sending, preventing `@everyone` / `@here` abuse.
- **Error redaction** — generic error messages are returned to the
LLM. Internal details (service IDs, URLs, exception messages) are
logged server-side only.
---
## Adding New Adapters
The `ChannelAdapter` protocol defines the interface any platform adapter
must implement:
```python
class ChannelAdapter(Protocol):
channel_type: str
async def start(self) -> None: ...
async def stop(self) -> None: ...
async def send(self, channel_id: str, content: str) -> str: ...
async def edit_message(self, channel_id: str, message_id: str, content: str) -> None: ...
async def send_approval_request(self, channel_id: str, ws_id: str, correlation_id: str, items: list[dict]) -> None: ...
async def send_plan_review(self, channel_id: str, ws_id: str, correlation_id: str, content: str) -> None: ...
async def create_thread(self, parent_channel_id: str, name: str, message_id: str = "") -> str: ...
```
To add a new platform:
1. Create `turnstone/channels/<platform>/` package
2. Implement the `ChannelAdapter` protocol
3. Add a `--<platform>-token` flag and detection logic in
`turnstone/channels/cli.py`
4. Add the optional dependency in `pyproject.toml` (e.g.
`turnstone[slack]`)
See `turnstone/channels/discord/` as a reference implementation.
+329 -6
View File
@@ -61,6 +61,8 @@ The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot
3. **Poll loop** — fetches `GET /v1/api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
A `get_snapshot()` method builds the full cluster state under a single lock acquisition — overview aggregates and per-node workstream lists in one atomic read. This is served both as a REST endpoint and as the initial SSE event on client connect.
### Thread Safety
All reads and writes to the node/workstream map are protected by a single `threading.Lock`. Query methods acquire the lock, copy data, and release before returning.
@@ -146,9 +148,41 @@ Single node detail with all its workstreams.
}
```
### `GET /v1/api/cluster/snapshot`
Full cluster state in a single response — all nodes with their workstreams plus overview aggregates. Built under a single lock for internal consistency. Used by the browser on initial load and SSE reconnect.
```json
{
"nodes": [
{
"node_id": "db-west-04",
"server_url": "http://10.0.3.4:8080",
"max_ws": 10,
"reachable": true,
"version": "0.3.0",
"health": {"status": "ok", "version": "0.3.0"},
"aggregate": {"total_tokens": 48200, "total_tool_calls": 156},
"workstreams": [
{"id": "a1b2c3d4", "name": "perf-db-west", "state": "running", ...}
]
}
],
"overview": {
"nodes": 847,
"workstreams": 4219,
"states": {"running": 1847, "thinking": 312, "attention": 89, "idle": 1940, "error": 31},
"aggregate": {"total_tokens": 12400000, "total_tool_calls": 34200},
"version_drift": false,
"versions": ["0.3.0"]
},
"timestamp": 1709294400.0
}
```
### `POST /v1/api/cluster/workstreams/new`
Create a new workstream on a target node. Dispatches a `CreateWorkstreamMessage` through the Redis MQ pipeline — the bridge on the target node picks it up and creates the workstream on the server. Requires `"full"` auth role.
Create a new workstream on a target node. Dispatches a `CreateWorkstreamMessage` through the Redis MQ pipeline — the bridge on the target node picks it up and creates the workstream on the server. Requires `write` scope.
Request:
@@ -182,7 +216,7 @@ Creation is asynchronous — the response confirms the MQ message was dispatched
### `GET /v1/api/cluster/events`
Server-Sent Events stream for real-time cluster updates.
Server-Sent Events stream for real-time cluster updates. The first event is always a `snapshot` containing the full cluster state (same shape as `GET /v1/api/cluster/snapshot` with an added `type: "snapshot"` field), followed by incremental events:
```
data: {"type":"cluster_state","ws_id":"a1b2","node_id":"db-west-04","state":"running"}
@@ -207,6 +241,96 @@ Keepalive comments (`: keepalive\n\n`) are sent every 5 seconds. Clients should
}
```
### Admin API
User and token management endpoints. All admin endpoints require `approve` scope, except for the setup endpoint which is public.
#### `POST /v1/api/auth/setup`
Creates the first admin user when no users exist. Public endpoint (no auth required). Returns a JWT and sets a session cookie. Returns `409` if users already exist. See [Security: First-time setup](security.md#first-time-setup) for full details.
#### `POST /v1/api/admin/users`
Create a new user.
```json
{
"username": "alice",
"password": "s3cret",
"scopes": ["read", "write"]
}
```
#### `GET /v1/api/admin/users`
List all users.
```json
{
"users": [
{"user_id": "u_abc123", "username": "alice", "scopes": ["read", "write"], "created": "2026-03-01T12:00:00Z"}
]
}
```
#### `DELETE /v1/api/admin/users/{user_id}`
Delete a user and revoke all their tokens.
#### `POST /v1/api/admin/users/{user_id}/tokens`
Create an API token for the given user. Returns a `ts_`-prefixed token string that can be used for Bearer auth or passed to `client.login(token="ts_xxx")`.
```json
{
"name": "CI pipeline",
"scopes": ["read", "write"]
}
```
#### `GET /v1/api/admin/users/{user_id}/tokens`
List active tokens for a user (token strings are not returned, only metadata).
#### `DELETE /v1/api/admin/tokens/{token_id}`
Revoke a specific API token.
### Channel links
| Method | Path | Description |
|--------|------|-------------|
| GET | `/v1/api/admin/users/{user_id}/channels` | List channel links for a user |
| POST | `/v1/api/admin/users/{user_id}/channels` | Link a channel account (channel_type, channel_user_id) |
| DELETE | `/v1/api/admin/channels/{channel_type}/{channel_user_id}` | Unlink a channel account |
These endpoints manage the `channel_users` table mappings that connect external platform identities (e.g. Discord user IDs) to turnstone users. See [Channel Integrations](channels.md) for details on the linking flow.
#### `GET /v1/api/auth/status`
Public endpoint for login UI state detection. Returns auth configuration, not
current-user identity.
```json
{
"auth_enabled": true,
"has_users": true,
"setup_required": false
}
```
### Auth Scopes
The auth system uses three scopes instead of the earlier read/full role model:
| Scope | Grants |
|-------|--------|
| `read` | Read-only access: dashboards, workstream lists, SSE streams, health |
| `write` | Send messages, create/close workstreams, approve tool calls |
| `approve` | Admin operations: manage users and API tokens |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
---
## Reverse Proxy
@@ -236,17 +360,17 @@ The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/share
### SSE Proxy
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied by creating a per-connection `httpx.AsyncClient(timeout=None)`, streaming the upstream response via `aiter_text()`, parsing SSE framing (`\n\n` delimiters), and re-emitting events through `EventSourceResponse`. Each proxied SSE stream requires its own httpx client since the shared client's 30-second timeout would kill long-lived connections.
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
### Authentication
The proxy forwards requests to server nodes using the console's `--auth-token`. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require the `"full"` auth role, preventing read-only tokens from escalating to write operations.
The proxy forwards the user's JWT to upstream server nodes — it extracts the token from the incoming request's cookie (or `Authorization` header) and adds it as a `Bearer` header on the proxied request. Since all services share the same `TURNSTONE_JWT_SECRET`, the user's JWT is valid on every node without re-authentication. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require `write` scope, preventing read-only tokens from escalating via proxy. The static `--auth-token` / `proxy_auth_token` is used as a fallback when no user JWT is present.
---
## Browser Dashboard
The web UI has four views, toggled client-side:
The web UI has five views, toggled client-side:
### 1. Cluster Overview (landing)
@@ -276,7 +400,206 @@ Triggered by the "+ new" header button. A modal dialog with:
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
All four views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
### 5. Admin Panel
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, and channel link management
with three tabs:
**Users tab:**
- Grid table listing all users (username, display name, role, creation date)
- "Create User" button opens a modal with fields for username, display name,
and password (validated: username 1-64 ASCII, password min 8 characters)
- Delete button on each row opens a styled confirmation modal before
removing the user and cascading to revoke all their tokens
**Tokens tab:**
- User selector dropdown to pick which user's tokens to manage
- Grid table listing tokens for the selected user (name, prefix, scopes,
creation date)
- Scope badges rendered as colored pills for visual clarity
- "Create Token" button opens a modal with fields for token name and scope
checkboxes
- On creation, a "Token Created" modal displays the raw `ts_`-prefixed
token with a copy button. The token is shown once and cannot be retrieved
again.
- Revoke button on each row opens a styled confirmation modal before
deleting the token
**Channels tab:**
- User selector dropdown to pick which user's channel links to manage
- Grid table listing linked channel accounts for the selected user
(channel type, channel user ID, creation date)
- "Link Channel" button opens a modal with fields for channel type
(e.g. `discord`) and the platform user ID
- Unlink button on each row opens a styled confirmation modal before
removing the channel mapping
- Admins can force-link users who have not self-linked via `/link` in
Discord
**Accessibility:**
- Full keyboard navigation: focus traps in modals, Escape to close, arrow
keys for tab switching
- Responsive layout with column hiding at 700px breakpoint
**First-time setup:**
The console also exposes `POST /v1/api/auth/setup` for first-time
bootstrap. When no users exist, the setup wizard calls this public endpoint
to create the initial admin user and receive a JWT in one step. See
[Security: First-time setup](security.md#first-time-setup) for details.
---
## Scheduled Tasks
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via the MQ broker. It supports cron-based recurring schedules and one-shot `at` schedules.
### Architecture
The scheduler runs as a daemon thread inside the console process. Every `check_interval` seconds (default 15) it:
1. Acquires a distributed lock via Redis `SET NX EX` (prevents duplicate dispatch in multi-console deployments)
2. Queries the storage backend for tasks whose `next_run <= now` and `enabled = true`
3. Dispatches each due task as one or more `CreateWorkstreamMessage` via MQ
4. Updates `last_run` and computes the next `next_run` (or disables one-shot `at` tasks)
5. Releases the lock via Lua script (safe conditional delete)
Run history is automatically pruned (runs older than 90 days) approximately once per hour.
### Schedule Types
| Type | Field | Behavior |
|------|-------|----------|
| `cron` | `cron_expr` | Recurring schedule using standard 5-field cron syntax. Requires `croniter`. |
| `at` | `at_time` | One-shot: fires once at the given ISO 8601 timestamp (must include timezone), then auto-disables. |
### Target Modes
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Pushes to the shared inbound queue (any bridge picks it up) |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
### Configuration
| Parameter | Default | Description |
|-----------|---------|-------------|
| `check_interval` | `15.0` | Seconds between scheduler ticks |
| `lock_ttl` | `60` | Distributed lock TTL in seconds |
| `max_fan_out` | `20` | Maximum nodes for `all` target mode |
Dependency: `croniter` (installed with turnstone).
### Schedule API
All schedule endpoints require `approve` scope. Maximum 200 schedules.
#### `GET /v1/api/admin/schedules`
List all scheduled tasks.
```json
{
"schedules": [
{
"task_id": "a1b2c3d4",
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"auto_approve_tools": [],
"enabled": true,
"created_by": "u_admin",
"last_run": "2026-03-05T02:00:00Z",
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-01T12:00:00Z",
"updated": "2026-03-05T02:00:01Z"
}
]
}
```
#### `POST /v1/api/admin/schedules`
Create a scheduled task.
Request:
```json
{
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"target_mode": "auto",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"enabled": true
}
```
Required fields: `name`, `schedule_type`, `initial_message`. For `cron` schedules provide `cron_expr`; for `at` schedules provide `at_time` (ISO 8601 with timezone, must be in the future).
Response: `ScheduleInfo` (same shape as list items above). Returns `400` for invalid cron syntax, naive timestamps, or past `at_time`. Returns `409` if the 200-schedule cap is reached.
#### `GET /v1/api/admin/schedules/{task_id}`
Get a single scheduled task. Returns `ScheduleInfo` or `404`.
#### `PUT /v1/api/admin/schedules/{task_id}`
Partial update — only include fields to change. If `schedule_type`, `cron_expr`, or `at_time` change, `next_run` is recomputed automatically.
```json
{
"enabled": false
}
```
Response: updated `ScheduleInfo`. Returns `400` for validation errors, `404` if not found.
#### `DELETE /v1/api/admin/schedules/{task_id}`
Delete a scheduled task and all its run history. Returns `{"status": "ok"}` or `404`.
#### `GET /v1/api/admin/schedules/{task_id}/runs?limit=50`
List execution history for a task (most recent first). `limit` defaults to 50, max 200.
```json
{
"runs": [
{
"run_id": "r_abc123",
"task_id": "a1b2c3d4",
"node_id": "db-west-04",
"ws_id": "ws_xyz",
"correlation_id": "corr_789",
"started": "2026-03-05T02:00:00Z",
"status": "dispatched",
"error": ""
}
]
}
```
Status is `dispatched` on success or `failed` with an `error` message (e.g. no reachable nodes). Failed runs do not advance `next_run`.
---
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5a2bd1c55ac8cf3b777a8decb6f3bb3d063c10c8f3a9e63457079830e48f456
size 162310
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:09cee5819bb7820641a466ed29a53f1b479bc2ded83079fc798c7410bf741a62
size 329703
+4 -2
View File
@@ -40,7 +40,8 @@ package "turnstone/core/" <<Rectangle>> {
component [auth.py\nAuthentication] as auth <<core>>
component [healthcheck.py\nBackendHealthMonitor] as healthcheck <<core>>
component [ratelimit.py\nRateLimiter] as ratelimit <<core>>
component [mcp_client.py\nMCPClientManager] as mcp <<core>>
component [mcp_client.py\nMCPClientManager\n(push + periodic refresh)] as mcp <<core>>
component [tool_search.py\nToolSearchManager, BM25] as toolsearch <<core>>
component [model_registry.py\nModelRegistry] as registry <<core>>
}
@@ -95,7 +96,7 @@ package "turnstone/sdk/" <<Rectangle>> {
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\n14 tool schemas] as schemas <<artifact>>
component [*.json\n18 tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
@@ -136,6 +137,7 @@ session --> edit
session --> web
session --> healthcheck
session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
healthcheck --> metrics
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b3f76042c8046560502fa3132351821d56e8526b13d38d7be8b839fcf3d5f648
size 373463
+49 -3
View File
@@ -41,7 +41,7 @@ class "WorkstreamTerminalUI" as WsTermUI {
}
class "WebUI" as WebUI {
- _event_queue: Queue
- _listeners: list[Queue]
- _approval_event: Event
- _plan_event: Event
- _ws_prompt_tokens: int
@@ -108,6 +108,8 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ thinking_mode: str
+ supports_effort: bool
+ supports_web_search: bool
+ supports_tool_search: bool
+ supports_vision: bool
}
' ChatSession
@@ -118,8 +120,9 @@ class "ChatSession" as ChatSession {
- ui: SessionUI
- messages: list[dict]
- _msg_tokens: list[int]
- _session_id: str
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
+ model_alias: str | None {property}
- _tools: list[dict]
@@ -130,7 +133,7 @@ class "ChatSession" as ChatSession {
--
+ send(user_input: str)
+ handle_command(command: str)
+ resume_session(session_id: str)
+ resume(ws_id: str)
- _save_config()
- _stream_response(stream) → dict
- _create_stream_with_retry(msgs) → Stream (+ fallback)
@@ -139,6 +142,12 @@ class "ChatSession" as ChatSession {
- _prepare_tool(tc) → item dict
- _prepare_mcp_tool(call_id, name, args) → item dict
- _exec_mcp_tool(item) → (call_id, output)
- _get_active_tools() → list[dict]
- _prepare_tool_search() → None
- _exec_tool_search(item) → (call_id, output)
- _on_mcp_tools_changed()
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
@@ -201,22 +210,58 @@ enum "WorkstreamState" as WsState {
' MCPClientManager
class "MCPClientManager" as MCPMgr {
- _sessions: dict[str, ClientSession]
- _per_server_tools: dict[str, list[dict]]
- _per_server_resources: dict[str, list[dict]]
- _per_server_prompts: dict[str, list[dict]]
- _tools: list[dict]
- _tool_map: dict[str, tuple]
- _resource_map: dict[str, tuple]
- _prompt_map: dict[str, tuple]
- _supports_list_changed: dict[str, bool]
- _listeners: list[Callable]
--
+ start()
+ get_tools() → list[dict]
+ get_resources() → list[dict]
+ get_prompts() → list[dict]
+ is_mcp_tool(name) → bool
+ call_tool_sync(name, args) → str
+ read_resource_sync(uri) → str
+ get_prompt_sync(name, args?) → list[dict]
+ refresh_sync(server?) → dict
+ add_listener(callback)
+ remove_listener(callback)
+ server_names: list[str] {property}
+ shutdown()
--
Background asyncio event loop
bridges async MCP SDK to
sync ChatSession dispatch.
Push + periodic + manual refresh.
Resources + prompts discovered
alongside tools at startup.
--
core/mcp_client.py
}
' ToolSearchManager
class "ToolSearchManager" as ToolSearchMgr {
- _all_tools: list[dict]
- _always_on: list[dict]
- _deferred: list[dict]
- _expanded: dict[str, None]
- _index: BM25Index
--
+ should_activate() → bool
+ get_visible_tools() → list[dict]
+ get_deferred_tools() → list[dict]
+ get_expanded_names() → list[str]
+ search(query, k) → list[dict]
+ expand_visible(names) → list[dict]
+ get_search_tool_definition() → dict
+ format_search_results(tools) → str
}
' ModelRegistry
class "ModelRegistry" as ModelReg {
- _models: dict[str, ModelConfig]
@@ -317,6 +362,7 @@ LLMProvider <|.. AnthropicProv
ChatSession --> SessionUI : uses
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fca0957b54101ce5c2b2e06d639b04dc5fff733f2641af0d732c49ea86883882
size 279397
+19 -5
View File
@@ -18,7 +18,7 @@ User -> CS : send(user_input)
activate CS
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(session_id, "user", input)
CS -> DB : save_message(ws_id, "user", input)
== LLM Call Loop ==
@@ -57,6 +57,14 @@ group loop [while tool_calls present]
end
end
note right of CS
**Cancellation checkpoint:**
_check_cancelled() runs per chunk.
If cancel_event is set, raises
GenerationCancelled — preserves
partial content, emits idle state.
end note
LLM --> CS : stream complete (usage stats)
deactivate LLM
@@ -65,8 +73,8 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> DB : save_message(session_id, "assistant", content)
CS -> DB : save_message(session_id, "tool_call", ...) ×N
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
@@ -112,7 +120,7 @@ group loop [while tool_calls present]
note right of TP
Parallel execution:
bash → Popen + line-by-line streaming
read_file → open().read()
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task/plan → _run_agent() sub-loop
@@ -136,7 +144,7 @@ group loop [while tool_calls present]
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(session_id, "tool_result", ...)
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
@@ -144,6 +152,12 @@ group loop [while tool_calls present]
end
note right of CS : Loop back for next LLM call
else GenerationCancelled
CS -> CS : Preserve partial content\nor roll back incomplete tools
CS -> UI : on_info("[Generation cancelled]")
CS -> UI : on_state_change("idle")
CS --> User : return (no re-raise)
end
end
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06fb9faa29c11395c6fc54ebddc79994b000dee78d56e0c13cb689fd6a82e37a
size 237255
+32 -22
View File
@@ -24,27 +24,31 @@ partition "Phase 1: Prepare" #E8F5E9 {
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (14 tools):**
┌─────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├─────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✓ Yes
│ man │ ✗ Auto-approve │
│ web_fetch │ ✓ Yes
│ web_search │ ✓ Yes
│ task │ ✓ Yes
plan │ ✓ Yes │
remember │ ✗ Auto-approve
│ recall │ ✗ Auto-approve │
forget │ ✗ Auto-approve │
├─────────────┼──────────────────┤
mcp__* │ ✓ Yes (external)
└─────────────┴──────────────────┘
**Dispatch table (18 tools):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✗ Auto-approve
│ man │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve
│ web_search │ ✗ Auto-approve
│ tool_search │ ✗ Auto-approve
task │ ✓ Yes │
plan │ ✓ Yes
│ remember │ ✗ Auto-approve │
recall │ ✗ Auto-approve │
│ forget │ ✗ Auto-approve │
notify │ ✗ Auto-approve
│ read_resource │ ✓ Yes │
│ use_prompt │ ✓ Yes │
├───────────────┼──────────────────┤
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
@@ -86,6 +90,8 @@ partition "Phase 2: Approve" #FFF3E0 {
}
partition "Phase 3: Execute" #E3F2FD {
:_check_cancelled();
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
if (single tool call?) then (yes)
:Execute sequentially:\nrun_one(items[0]);
else (multiple)
@@ -98,7 +104,7 @@ partition "Phase 3: Execute" #E3F2FD {
if item.denied → return denial message
else → item["execute"](item)
├─ _exec_bash: subprocess.run(["bash", script.sh])
├─ _exec_read_file: open().readlines()
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
@@ -106,11 +112,15 @@ partition "Phase 3: Execute" #E3F2FD {
├─ _exec_man: man/info subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: Tavily API POST (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_plan: _run_agent(AGENT_TOOLS, read-only)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_remember: SQLite INSERT OR REPLACE
├─ _exec_recall: SQLite FTS5/LIKE search
├─ _exec_forget: SQLite DELETE
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
end note
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:13b2da77312e5abb44c81aabb7f0addccab31d9bc7e8ef2f1c3563ef985ed503
size 186941
+7
View File
@@ -79,6 +79,12 @@ package "Inbound Messages (Client → Bridge)" #FFF3E0 {
type = "list_nodes"
}
class CancelMessage {
type = "cancel"
--
+ ws_id: str
}
IM <|-- SendMessage
IM <|-- ApproveMessage
IM <|-- PlanFeedbackMessage
@@ -88,6 +94,7 @@ package "Inbound Messages (Client → Bridge)" #FFF3E0 {
IM <|-- ListWorkstreamsMessage
IM <|-- HealthMessage
IM <|-- ListNodesMessage
IM <|-- CancelMessage
}
package "Outbound Events (Bridge → Client)" #E3F2FD {
+6
View File
@@ -40,6 +40,12 @@ running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
thinking --> idle : cancel() called\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval/plan wait\n_emit_state("idle")
note right of thinking
**Emitted via:**
session._emit_state(state)
+25 -1
View File
@@ -78,7 +78,19 @@ activate NodeA
NodeA --> CC : {status:"ok", version:"0.3.0",\nmodel:"...", workstreams:{...}}
deactivate NodeA
CC -> CC : Diff old vs new workstream IDs
CC -> CC : Replace NodeSnapshot["nodeA"]\n.workstreams, .health, .aggregate
CC -> CC : _fanout(ws_created) for\nnewly appeared workstreams
CC -> CC : _fanout(ws_closed) for\nremoved workstreams
note right of CC
Poll-diff fanout ensures
browser SSE clients learn
about workstreams that
appeared without a real-time
cluster event (e.g. bridge
startup recovery).
end note
CC -x NodeB : (SKIPPED: sim:// URL)
@@ -89,10 +101,15 @@ deactivate CC
Browser -> Server : GET /v1/api/cluster/events
activate Server
Server -> CC : get_snapshot()
CC --> Server : ClusterSnapshot\n(full current state)
Server -> CC : register_listener(queue)
note right : Per-client queue.Queue(maxsize=500)\nSSE via EventSourceResponse + run_in_executor()
loop continuous
Server -> Browser : data: {"type":"snapshot",...}\n(full state as first SSE event)
loop continuous (incremental updates)
CC -> Server : event via listener queue\n(from any of the 3 threads)
Server -> Browser : data: {"type":"cluster_state",...}\n\n
end
@@ -105,6 +122,13 @@ Browser -> Server : connection closed
Server -> CC : unregister_listener(queue)
deactivate Server
== Browser REST: Snapshot ==
Browser -> Server : GET /v1/api/cluster/snapshot
Server -> CC : get_snapshot()
CC --> Server : ClusterSnapshot\n(full current state)
Server --> Browser : JSON response
== Browser REST Requests ==
Browser -> Server : GET /v1/api/cluster/overview
+4 -1
View File
@@ -32,10 +32,11 @@ package "turnstone/sdk/ (Python)" {
+ approve()
+ plan_feedback()
+ command()
+ cancel(ws_id)
+ stream_events(ws_id)
+ stream_global_events()
+ send_and_wait()
+ list_sessions()
+ list_saved_workstreams()
+ login() / logout()
+ health()
}
@@ -45,6 +46,7 @@ package "turnstone/sdk/ (Python)" {
+ nodes()
+ workstreams()
+ node_detail()
+ snapshot()
+ create_workstream()
+ stream_cluster_events()
+ login() / logout()
@@ -129,6 +131,7 @@ package "sdk/typescript/ (TypeScript)" {
class "TurnstoneConsole" as TSConsole <<ts>> {
+ overview()
+ nodes()
+ snapshot()
+ clusterEvents()
...
}
+29 -17
View File
@@ -13,18 +13,19 @@ skinparam class {
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+register_session(session_id, title)
+save_message(session_id, role, content, ...)
+load_session_messages(session_id) → list[dict]
+list_sessions(limit) → list
+delete_session(session_id) → bool
+prune_sessions(retention_days) → (int, int)
+resolve_session(alias_or_id) → str | None
+save_session_config(session_id, config)
+load_session_config(session_id) → dict
+set_session_alias(session_id, alias) → bool
+get_session_name(session_id) → str | None
+update_session_title(session_id, title)
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
@@ -32,6 +33,11 @@ interface "StorageBackend" as SB <<protocol>> {
+kv_search(query) → list[(str, str)]
+search_history(query, limit) → list
+search_history_recent(limit) → list
+create_user(user_id, username, display_name, pw_hash)
+get_user(user_id) / get_user_by_username(username)
+list_users() / delete_user(user_id)
+create_api_token(...) / get_api_token_by_hash(hash)
+list_api_tokens(user_id) / delete_api_token(id)
+close()
}
@@ -58,8 +64,11 @@ class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+sessions: Table
+session_config: Table
+workstreams: Table (node_id, alias, title, state)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
--
SQLAlchemy Core
Single source of truth
@@ -76,6 +85,7 @@ class "_migrate.py" as Migrate <<migration>> {
class "migrations/" as Versions <<migration>> {
001_initial_schema.py
002_user_identity.py
}
' -- Registry --
@@ -91,12 +101,14 @@ class "_registry.py" as Registry {
' -- Facade --
class "memory.py" as Facade <<facade>> {
+register_session()
+save_message()
+load_session_messages()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all 18 functions)
+... (all delegated functions)
--
Thin delegation to
get_storage()
+179
View File
@@ -0,0 +1,179 @@
@startuml
!theme plain
title Turnstone — Authentication Architecture
skinparam class {
BackgroundColor<<core>> #E8EAF6
BackgroundColor<<jwt>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<endpoint>> #FFE0B2
BackgroundColor<<scope>> #F3E5F5
}
' -- Core Auth --
class "AuthConfig" as AC <<core>> {
+enabled: bool
+tokens: dict[str, str]
+check(token) → role | None
--
Static config-file tokens
hmac.compare_digest
}
class "AuthResult" as AR <<core>> {
+user_id: str
+scopes: frozenset[str]
+token_source: str
+has_scope(scope) → bool
}
class "check_request()" as CR <<core>> {
auth_config, method, path,
auth_header, cookie_header,
jwt_secret, storage
→ (allowed, status, msg, AuthResult)
--
1. Auth disabled → allow
2. Public path → allow
3. Extract Bearer / cookie
4. Detect token type
5. Validate → AuthResult
6. Check scope vs path
}
' -- Token Types --
class "JWT (HS256)" as JWT <<jwt>> {
sub: user_id
scopes: "read,write,approve"
src: "password" | "database"
iat, exp (24h default)
--
Detected by: contains "."
Validated locally
No DB call
}
class "API Token" as AT <<jwt>> {
Format: ts_ + 64 hex
Stored: SHA-256 hash
--
Detected by: starts with "ts_"
Lookup by hash in DB
Expiry check
}
class "Config Token" as CT <<core>> {
Raw value in memory
Role: "read" | "full"
--
Detected by: fallback
hmac.compare_digest
No DB needed
}
' -- Scopes --
class "Scope Hierarchy" as SH <<scope>> {
read: {read}
write: {read, write}
approve: {read, write, approve}
--
GET → read
POST write paths → write
POST /api/approve → approve
/api/admin/* → approve
}
' -- Storage --
class "users" as UT <<storage>> {
user_id (PK)
username (unique)
display_name
password_hash (bcrypt)
created
}
class "api_tokens" as TT <<storage>> {
token_id (PK)
token_hash (SHA-256, unique)
token_prefix
user_id → users
name, scopes
created, expires
}
' -- Endpoints --
class "POST /api/auth/login" as Login <<endpoint>> {
{username, password}
OR {token: "ts_xxx"}
→ {jwt, role, scopes, user_id}
--
Sets HttpOnly cookie
}
class "GET /api/auth/status" as Status <<endpoint>> {
→ {auth_enabled, has_users,
setup_required}
--
Public (no auth)
Drives UI setup wizard
}
class "POST /api/auth/setup" as Setup <<endpoint>> {
{username, display_name, password}
→ {jwt, user_id, scopes}
--
Public (no auth)
Only when zero users exist
Returns 409 if already set up
}
class "Admin API (Console)" as Admin <<endpoint>> {
POST/GET/DELETE users
POST/GET tokens
DELETE tokens/{id}
--
Requires approve scope
}
' -- Relationships --
CR --> AC : config tokens
CR --> JWT : validate
CR --> AT : hash lookup
CR --> CT : hmac check
CR --> AR : returns
CR --> SH : checks
Login --> JWT : issues
Login --> UT : verify password
Login --> TT : verify API token
Setup --> UT : create first user
Setup --> JWT : issues
AT --> TT : lookup by hash
Admin --> UT : CRUD
Admin --> TT : CRUD
AR --> SH : scopes from
JWT ..> AR : produces
AT ..> AR : produces
CT ..> AR : produces
note right of CR
**Middleware Flow**
AuthMiddleware on every request:
1. Extract token from header/cookie
2. Detect type (JWT / ts_ / config)
3. Validate → AuthResult
4. Set ctx_user_id for logging
5. Store auth_result in scope state
end note
note bottom of SH
**Console** owns admin endpoints
**Server** validates JWT + config only
Both share JWT signing secret
end note
@enduml
+250
View File
@@ -0,0 +1,250 @@
@startuml
!theme plain
title Turnstone — Channel Integration Architecture
skinparam class {
BackgroundColor<<platform>> #E1BEE7
BackgroundColor<<service>> #E8EAF6
BackgroundColor<<mq>> #FFCDD2
BackgroundColor<<bridge>> #C8E6C9
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
}
' -- External Platforms --
class "Discord" as Discord <<platform>> {
Gateway WebSocket (v10)
Message events
Interaction callbacks (buttons)
Thread-per-workstream
--
discord.py 2.x
asyncio event loop
}
class "Slack (future)" as Slack <<platform>> {
Socket Mode / Events API
Block Kit messages
--
Planned integration
}
class "Teams (future)" as Teams <<platform>> {
Bot Framework
Adaptive Cards
--
Planned integration
}
' -- Channel Service --
class "turnstone-channel" as ChannelService <<service>> {
entry point: turnstone-channel
--
One process per platform
asyncio event loop
Structured logging (structlog)
--log-level, --log-format
--
POST /v1/api/notify (HTTP)
GET /health
}
class "DiscordBot" as Bot <<service>> {
+on_message(msg)
+on_interaction(interaction)
+send(channel_id, content)
+run(token)
--
discord.py Client
Receives message events
Sends replies + embeds
Creates threads for workstreams
Renders approval buttons
escape_mentions() on send
}
class "ChannelRouter" as Router <<service>> {
+resolve_route(platform, channel_id)
→ ws_id | None
+register_route(channel_id, ws_id)
+resolve_identity(platform, platform_user_id)
→ user_id | None
--
Maps channels → workstreams
Maps platform users → turnstone users
Caches routes in memory
}
class "AsyncRedisBroker" as Broker <<service>> {
+push_inbound(msg)
+subscribe(ws_id) → AsyncIterator
+subscribe_global() → AsyncIterator
+push_response(correlation_id, msg)
--
redis.asyncio client
Pub/sub + queue operations
}
' -- Redis MQ --
class "Redis MQ" as Redis <<mq>> {
turnstone:inbound (LIST)
turnstone:events:{ws_id} (PUBSUB)
turnstone:events:global (PUBSUB)
turnstone:resp:{corr_id} (LIST)
--
Shared message bus
Same queues as bridge protocol
}
' -- Bridge + Server --
class "turnstone-bridge" as Bridge <<bridge>> {
BLPOP turnstone:inbound
Drive server via HTTP
Relay SSE → Redis pub/sub
--
Owns workstream lifecycle
Auto-approve / manual approve
}
class "turnstone-server" as Server <<server>> {
POST /v1/api/send
POST /v1/api/approve
POST /v1/api/workstreams/new
GET /v1/api/events?ws_id=
--
LLM execution + tool use
SSE event stream
--
notify tool: _exec_notify()
ServiceTokenManager (JWT)
}
' -- Storage --
class "channel_users" as CU <<storage>> {
channel_user_id (PK)
platform: "discord" | "slack"
platform_user_id
user_id → users
linked_at
--
/link command creates row
Resolved on each inbound message
}
class "channel_routes" as CR <<storage>> {
channel_type (PK)
channel_id (PK)
ws_id
node_id
created
--
Maps platform channels
to turnstone workstreams
}
class "services" as SVC <<storage>> {
service_type (PK)
service_id (PK)
url
last_heartbeat
created
--
Heartbeat every 30s
Stale after 120s
ON CONFLICT DO UPDATE
}
' -- Relationships --
Discord --> Bot : gateway\nevents
Bot --> Router : on_message\non_interaction
Router --> Broker : SendMessage\nApproveMessage
Router --> CU : resolve identity
Router --> CR : resolve / register route
Broker --> Redis : RPUSH inbound\nRPUSH resp:{id}
Redis --> Bridge : BLPOP inbound
Bridge --> Server : HTTP API
Server --> Bridge : SSE events
Bridge --> Redis : PUBLISH events:{ws_id}\nPUBLISH events:global
Redis --> Broker : SUBSCRIBE events:{ws_id}
Broker --> Bot : event stream
Bot --> Discord : reply / embed\nbutton callback
Slack .[hidden]. Discord
Teams .[hidden]. Slack
ChannelService --> Bot : creates + runs
ChannelService --> Router : creates
ChannelService --> Broker : creates
ChannelService --> SVC : register / heartbeat /\nderegister
' -- Notification path (direct HTTP, bypasses MQ) --
Server --> ChannelService : POST /v1/api/notify\n(JWT: aud=turnstone-channel)
Server --> SVC : list_services("channel",\nmax_age_seconds=120)
' -- Notes --
note right of Bot
**Inbound Flow**
1. Discord message arrives via gateway
2. Bot.on_message() fires
3. ChannelRouter resolves channel → ws_id
(or creates new workstream)
4. ChannelRouter resolves platform user → user_id
via channel_users table
5. Broker.push_inbound(SendMessage)
6. Bridge pops from Redis, drives server
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no MQ owner)
2. Existing ws_id reused directly from route
3. CreateWorkstreamMessage sent with
resume_ws=<ws_id>
4. Server resumes atomically during creation
5. Bridge emits WorkstreamResumedEvent → thread
end note
note right of Broker
**Outbound Flow**
1. Server emits SSE events
2. Bridge relays to Redis events:{ws_id}
3. Broker.subscribe(ws_id) yields events
4. Bot formats and sends to Discord thread
end note
note bottom of CR
**Approval Flow**
1. ApprovalRequestEvent arrives via events:{ws_id}
2. Bot renders Discord buttons (Approve / Deny)
3. User clicks button → on_interaction()
4. Router builds ApproveMessage
5. Broker.push_response(correlation_id, msg)
6. Bridge pops from resp:{id}, calls POST /api/approve
end note
note bottom of CU
**Identity Linking**
1. User runs /link in Discord
2. Bot opens modal requesting API token
3. User submits ts_... API token
4. Bot validates token against storage
5. On success, inserts channel_users row
6. Subsequent messages carry user_id
7. AuthResult scopes applied by server
end note
note bottom of SVC
**Notification Flow** (direct HTTP, bypasses MQ)
1. LLM calls notify tool → _prepare_notify()
2. _exec_notify() checks rate limit (5/turn)
3. Queries services table for healthy gateways
4. Mints JWT (aud: turnstone-channel) via
ServiceTokenManager
5. POSTs to first healthy gateway
6. Gateway validates JWT, resolves target
7. adapter.send() → Discord API
8. On failure: retry up to 3× (1s, 3s backoff)
9. SSRF: only http(s) URLs allowed
end note
@enduml
+116
View File
@@ -0,0 +1,116 @@
@startuml
!theme plain
title Turnstone — Notification Delivery Flow
skinparam participant {
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<service>> #E8EAF6
BackgroundColor<<platform>> #E1BEE7
}
participant "ChatSession\n(turnstone-server)" as Session <<server>>
participant "StorageBackend" as Storage <<storage>>
participant "ServiceTokenManager" as STM <<server>>
participant "Channel Gateway\n(_http.py)" as Gateway <<service>>
participant "ChannelAdapter\n(Discord bot)" as Adapter <<service>>
participant "Discord API" as Discord <<platform>>
== Prepare Phase ==
Session -> Session : _prepare_notify(call_id, args)
note right
Validates:
- message (required, ≤2000 chars)
- target: username OR channel_type+channel_id
- no ambiguous targeting (both set)
- partial targeting errors
end note
== Execute Phase ==
Session -> Session : _exec_notify(item)
Session -> Session : check rate limit\n(≥5 per turn?)
alt rate limit exceeded
Session --> Session : "Error: rate limit exceeded"
end
loop up to 3 attempts (retry delays: 1s, 3s)
Session -> Storage : list_services("channel",\nmax_age_seconds=120)
Storage --> Session : services[] (sorted by\nlast_heartbeat DESC)
alt no healthy services
Session -> Session : log.warning("notify.no_services")
Session -> Session : sleep(delay)
else services available
Session -> STM : bearer_header
note right
Lazy-init ServiceTokenManager
aud: turnstone-channel
scope: write
Auto-rotates 1h JWTs
end note
STM --> Session : Authorization: Bearer <jwt>
loop for each gateway (first-healthy)
Session -> Session : SSRF check:\nurl.startswith("http://"|"https://")
Session -> Gateway : POST /v1/api/notify\n+ Authorization header
Gateway -> Gateway : _check_auth()\nvalidate JWT (aud=turnstone-channel)\nor static token
alt auth failed
Gateway --> Session : 401 Unauthorized
else auth ok
alt username target
Gateway -> Storage : get_user_by_username()
Storage --> Gateway : user
Gateway -> Storage : list_channel_users_by_user()
Storage --> Gateway : linked channels
else direct target
Gateway -> Gateway : use channel_type + channel_id
end
Gateway -> Adapter : send(channel_id, content)
note right
escape_mentions() applied
Chunked for 2000-char limit
end note
Adapter -> Discord : POST message
Discord --> Adapter : message_id
Adapter --> Gateway : message_id
Gateway --> Session : 200 {results: [{status: "sent"}]}
Session -> Session : _notify_count += 1
Session --> Session : "Notification sent successfully"
note right : Return — no further\ngateways tried
end
end
alt all gateways failed
Session -> Session : log.warning(\n"notify.all_gateways_failed")
Session -> Session : sleep(delay)
end
end
end
alt all retries exhausted
Session -> Session : log.warning("notify.delivery_failed")
Session --> Session : "Error: notification delivery failed"
end
== Service Registry (Background) ==
note over Gateway, Storage
**Heartbeat Lifecycle**
1. Gateway startup: register_service("channel", id, url)
2. Every 30s: heartbeat_service("channel", id)
3. Shutdown: deregister_service("channel", id)
4. Stale after 120s (4 missed heartbeats)
end note
@enduml
+166
View File
@@ -0,0 +1,166 @@
@startuml
!theme plain
title Turnstone — Watch Tool Architecture
skinparam participant {
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<ui>> #E8EAF6
}
participant "ChatSession\n(session.py)" as Session <<session>>
participant "WatchRunner\n(watch.py)" as Runner <<server>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
== Create Phase ==
Session -> Session : _prepare_watch(action="create")
note right
Validates:
- command via is_command_blocked()
- poll_every → parse_duration()
- stop_on → validate_condition()
- max watches limit (5)
- duplicate name check
needs_approval = True
end note
Session -> Storage : create_watch(watch_id, ws_id,\nnode_id, command, interval,\nstop_on, max_polls, next_poll)
Session --> UI : tool_result:\n"Watch 'pr-review' created"
== Poll Phase (WatchRunner daemon, every 15s) ==
Runner -> Storage : list_due_watches(now)
Storage --> Runner : due_watches[]
note right
Filters:
active=1 AND
next_poll <= now AND
node_id matches
end note
loop for each due watch
Runner -> Runner : is_command_blocked()?
alt blocked
Runner -> Storage : update_watch(active=False)
else safe
Runner -> Runner : subprocess.run(command)
note right
timeout = tool_timeout
start_new_session = True
output truncated at 64KB
end note
Runner -> Runner : evaluate_condition(\nstop_on, output,\nexit_code, prev_output)
note right
**Variables:**
output, data, exit_code,
prev_output, changed
**Safe builtins only:**
len, str, int, sorted, ...
No import/open/exec/eval
**stop_on=None:**
fires on change (skip 1st poll)
end note
alt condition fired OR max_polls reached
Runner -> Storage : update_watch(\npoll_count++,\nlast_output, active=False)
Runner -> Runner : format_watch_message()
Runner -> Runner : _dispatch_result(ws_id, msg)
else not fired
Runner -> Storage : update_watch(\npoll_count++,\nlast_output, next_poll)
end
end
end
== Dispatch Phase ==
note over Runner, Session
**Three dispatch paths:**
end note
alt Path A: workstream active + idle
Runner -> Session : dispatch_fn(message)\n→ _watch_pending.put()
Session -> Session : _dispatch_pending_watch()\n→ self.send(message)
Session -> UI : SSE: thinking, content,\ntool calls...
note right
Watch result appears as
synthetic user message.
Model sees it and responds.
Depth guard: max 5 chains.
end note
else Path B: workstream active + busy
Runner -> Session : dispatch_fn(message)\n→ _watch_pending.put()
note right
Queued. Dispatched when
current send() reaches IDLE.
end note
else Path C: workstream evicted
Runner -> Runner : restore_fn(ws_id)
note right
1. mgr.create() — may evict
another idle workstream
2. session.resume(ws_id)
3. set_watch_runner()
4. register new dispatch_fn
end note
Runner -> Session : restored dispatch_fn(message)
end
== Cancel / List ==
Session -> Storage : list_watches_for_ws(ws_id)
note right : action="list" (auto-approve)
Session -> Storage : update_watch(active=False)
note right : action="cancel" (auto-approve)
== Server Lifecycle ==
note over Runner, Storage
**Startup:**
1. WatchRunner created in main() with storage + node_id
2. restore_fn closure captures WorkstreamManager
3. Initial workstream: session.set_watch_runner(runner)
4. _lifespan(): runner.start() — daemon thread begins
**New workstream:**
session.set_watch_runner(runner) in create_workstream()
→ registers dispatch_fn for ws_id
**Eviction / close:**
session.close() → runner.remove_dispatch_fn(ws_id)
Watches remain active in DB — WatchRunner uses restore_fn
**Restart recovery:**
Overdue watches fire ONE immediate poll
next_poll updated to now + interval
Normal cadence resumes
**Shutdown:**
_lifespan(): runner.stop() — joins thread
end note
== REST API ==
note over UI, Storage
**GET /v1/api/watches[?ws_id=X]**
List active watches (for node or workstream)
**POST /v1/api/watches/{watch_id}/cancel**
Cancel a watch (sets active=False)
Both require write scope
end note
@enduml
@@ -0,0 +1,81 @@
@startuml
!theme plain
skinparam backgroundColor #FFFFFF
skinparam defaultFontName "IBM Plex Mono"
skinparam componentStyle rectangle
title Turnstone Governance Architecture
package "Auth Flow" {
[Login/Token Auth] as auth
[_load_user_permissions()] as perms
[_permissions_to_scopes()] as scopes
[create_jwt()] as jwt
}
package "Middleware" {
[AuthMiddleware\n(scope check)] as mw
[require_permission()\n(granular check)] as rp
}
package "Governance Storage" {
database "roles" as roles_db
database "user_roles" as ur_db
database "orgs" as orgs_db
database "tool_policies" as tp_db
database "prompt_templates" as pt_db
database "usage_events" as ue_db
database "audit_events" as ae_db
}
package "Runtime Enforcement" {
[evaluate_tool_policies_batch()] as eval
[WebUI.approve_tools()] as approve
[record_usage_event()] as usage
[record_audit()] as audit
}
package "Template Runtime" {
[_load_templates()] as tload
[_render_template()\n{{model}}, {{ws_id}}, {{node_id}}] as trender
[_init_system_messages()] as tsys
[set_template() / /template] as tset
}
package "Console UI" {
[Admin Panel\n10 tabs] as ui
[governance.js] as govjs
[sessionStorage\npermissions] as ss
}
auth --> perms : user_id
perms --> roles_db : JOIN user_roles + roles
perms --> scopes : permission set
scopes --> jwt : scopes + permissions
jwt --> mw : JWT in cookie/header
mw --> rp : scope OK → check permission
rp --> ui : 403 or allow
eval --> tp_db : list_tool_policies()
approve --> eval : tool names
approve --> ae_db : (via audit)
usage --> ue_db : on_status()
audit --> ae_db : admin handlers
govjs --> roles_db : /v1/api/admin/roles
govjs --> tp_db : /v1/api/admin/policies
govjs --> pt_db : /v1/api/admin/templates
govjs --> ue_db : /v1/api/admin/usage
govjs --> ae_db : /v1/api/admin/audit
tload --> pt_db : list_default_templates()\nor get_by_name()
tload --> trender : template content
trender --> tsys : rendered content
tset --> tload : name or None
auth -[hidden]-> mw
mw -[hidden]-> approve
@enduml
+158
View File
@@ -0,0 +1,158 @@
@startuml
!theme plain
title Turnstone — MCP Architecture (Resources, Prompts, Tools)
skinparam participant {
BackgroundColor<<mcp>> #E1BEE7
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<ui>> #E8EAF6
}
participant "MCP Server\n(external)" as MCPSrv <<mcp>>
participant "MCPClientManager\n(mcp_client.py)" as MCPMgr <<mcp>>
participant "ChatSession\n(session.py)" as Session <<session>>
participant "StorageBackend\n(governance)" as Storage <<storage>>
participant "Server / Console\n(health + UI)" as UI <<server>>
== Startup: Connection & Discovery ==
MCPMgr -> MCPSrv : initialize (stdio or HTTP)
MCPSrv --> MCPMgr : capabilities\n(tools, resources, prompts)
MCPMgr -> MCPSrv : tools/list
MCPSrv --> MCPMgr : Tool[]
opt resources capability
MCPMgr -> MCPSrv : resources/list
MCPSrv --> MCPMgr : Resource[]
MCPMgr -> MCPSrv : resources/templates/list
MCPSrv --> MCPMgr : ResourceTemplate[]
end
opt prompts capability
MCPMgr -> MCPSrv : prompts/list
MCPSrv --> MCPMgr : Prompt[]
end
note over MCPMgr
Per-server storage:
_per_server_tools, _per_server_resources, _per_server_prompts
Copy-on-write rebuild into _tools, _resources, _prompts
Prefix: mcp__{server}__{name}
end note
MCPMgr -> Session : notify tool listeners
MCPMgr -> Session : notify resource listeners
== Governance Sync (on connect & refresh) ==
MCPMgr -> Storage : sync_prompts_to_storage()
note right
For each MCP prompt:
- Manual template exists? → skip
- MCP template exists? → update
(reset is_default=False)
- New? → create (origin="mcp",
readonly=True, is_default=False)
Removed prompts → delete
Protected by _sync_lock
end note
== set_storage() from entry point ==
UI -> MCPMgr : set_storage(backend)
note right
If servers already connected,
triggers immediate sync
end note
== Runtime: Tool Execution ==
Session -> Session : _prepare_mcp_tool(func_name, args)
note right
approval_label = func_name
(e.g. mcp__github__search)
needs_approval = True
end note
Session -> MCPMgr : call_tool_sync(name, args)
MCPMgr -> MCPSrv : tools/call
MCPSrv --> MCPMgr : ToolResult
MCPMgr --> Session : output (text)
== Runtime: Resource Read ==
Session -> Session : _prepare_read_resource(uri)
note right
approval_label = mcp_resource__{normalized_uri}
URI normalized (.. resolved)
needs_approval = True
end note
Session -> MCPMgr : read_resource_sync(uri)
MCPMgr -> MCPSrv : resources/read
MCPSrv --> MCPMgr : ReadResourceResult
MCPMgr --> Session : content (text/blob)
== Runtime: Prompt Invocation ==
Session -> Session : _prepare_use_prompt(name, arguments)
note right
approval_label = mcp__srv__prompt
Validated via is_mcp_prompt()
needs_approval = True
end note
Session -> MCPMgr : get_prompt_sync(name, args)
MCPMgr -> MCPSrv : prompts/get
MCPSrv --> MCPMgr : GetPromptResult
MCPMgr --> Session : messages [{role, content}]
== Three-Tier Refresh ==
group Push Notifications
MCPSrv -> MCPMgr : ToolListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_tools()
MCPSrv -> MCPMgr : ResourceListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_resources()
MCPSrv -> MCPMgr : PromptListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_prompts()
MCPMgr -> Storage : sync_prompts_to_storage()
end
group Periodic Polling (default 4h)
MCPMgr -> MCPMgr : _periodic_refresh()
note right
Only polls capabilities
without push support.
Staggered per-server.
end note
end
group Manual Refresh
Session -> MCPMgr : refresh_sync()
note right: /mcp refresh [server]
end
== Policy Evaluation ==
note over Session
Tool policies use fnmatch on approval_label:
- mcp__github__* → allow (all GitHub tools/prompts)
- mcp_resource__file:///docs/* → allow
- mcp_resource__* → deny (block all resource reads)
- mcp__untrusted__* → ask
end note
== UI Visibility ==
UI -> MCPMgr : server_count, get_resources(), get_prompts()
note over UI
/health → mcp.servers, mcp.resources, mcp.prompts
Server UI: magenta status badge
Console: cluster status bar + node detail
System message: <mcp-resources> + <mcp-prompts> catalogs
end note
@enduml
+286
View File
@@ -0,0 +1,286 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 540" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif">
<defs>
<!-- Arrowhead markers -->
<marker id="arrow" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#484f58"/>
</marker>
<marker id="arrow-blue" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#58a6ff"/>
</marker>
<marker id="arrow-green" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#3fb950"/>
</marker>
<marker id="arrow-orange" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#f0883e"/>
</marker>
<marker id="arrow-coral" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#f47067"/>
</marker>
<marker id="arrow-muted" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#8b949e"/>
</marker>
<!-- Card shadow filter -->
<filter id="shadow" x="-4%" y="-4%" width="108%" height="112%">
<feDropShadow dx="0" dy="1" stdDeviation="2" flood-color="#000" flood-opacity="0.4"/>
</filter>
</defs>
<!-- Background -->
<rect width="1200" height="540" rx="8" fill="#0d1117"/>
<!-- Title -->
<text x="600" y="36" text-anchor="middle" fill="#e6edf3" font-size="15" font-weight="700" letter-spacing="3">TURNSTONE</text>
<text x="600" y="54" text-anchor="middle" fill="#8b949e" font-size="11" letter-spacing="1">SYSTEM ARCHITECTURE</text>
<!-- ==================== COLUMN HEADERS ==================== -->
<text x="90" y="86" text-anchor="middle" fill="#58a6ff" font-size="9" font-weight="600" letter-spacing="2">CLIENTS</text>
<text x="276" y="86" text-anchor="middle" fill="#3fb950" font-size="9" font-weight="600" letter-spacing="2">GATEWAYS</text>
<text x="480" y="86" text-anchor="middle" fill="#f0883e" font-size="9" font-weight="600" letter-spacing="2">MESSAGE QUEUE</text>
<text x="700" y="86" text-anchor="middle" fill="#f47067" font-size="9" font-weight="600" letter-spacing="2">CLUSTER NODES</text>
<text x="940" y="86" text-anchor="middle" fill="#f778ba" font-size="9" font-weight="600" letter-spacing="2">LLM PROVIDERS</text>
<!-- ==================== CLIENT BOXES ==================== -->
<!-- CLI -->
<g filter="url(#shadow)">
<rect x="30" y="108" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="108" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="108" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="111" width="120" height="2" fill="#161b22"/>
<text x="90" y="130" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">CLI</text>
<text x="90" y="145" text-anchor="middle" fill="#8b949e" font-size="9">terminal REPL</text>
</g>
<!-- Browser UI -->
<g filter="url(#shadow)">
<rect x="30" y="174" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="174" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="174" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="177" width="120" height="2" fill="#161b22"/>
<text x="90" y="196" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Browser UI</text>
<text x="90" y="211" text-anchor="middle" fill="#8b949e" font-size="9">HTTP + SSE</text>
</g>
<!-- SDK / API -->
<g filter="url(#shadow)">
<rect x="30" y="244" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="244" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="244" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="247" width="120" height="2" fill="#161b22"/>
<text x="90" y="266" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">SDK / API</text>
<text x="90" y="281" text-anchor="middle" fill="#8b949e" font-size="9">programmatic</text>
</g>
<!-- Discord / Slack -->
<g filter="url(#shadow)">
<rect x="30" y="314" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="314" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="314" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="317" width="120" height="2" fill="#161b22"/>
<text x="90" y="336" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Discord / Slack</text>
<text x="90" y="351" text-anchor="middle" fill="#8b949e" font-size="9">chat platforms</text>
</g>
<!-- ==================== GATEWAY BOXES ==================== -->
<!-- Console -->
<g filter="url(#shadow)">
<rect x="216" y="118" width="120" height="58" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="216" y="118" width="120" height="5" rx="5" fill="#3fb950"/>
<rect x="216" y="118" width="120" height="5" fill="#3fb950"/>
<rect x="216" y="121" width="120" height="2" fill="#161b22"/>
<text x="276" y="142" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<text x="276" y="157" text-anchor="middle" fill="#8b949e" font-size="9">dashboard + proxy</text>
<text x="276" y="169" text-anchor="middle" fill="#8b949e" font-size="9">cluster management</text>
</g>
<!-- Channel Gateway -->
<g filter="url(#shadow)">
<rect x="216" y="292" width="120" height="58" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="216" y="292" width="120" height="5" rx="5" fill="#3fb950"/>
<rect x="216" y="292" width="120" height="5" fill="#3fb950"/>
<rect x="216" y="295" width="120" height="2" fill="#161b22"/>
<text x="276" y="316" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Channel Gateway</text>
<text x="276" y="331" text-anchor="middle" fill="#8b949e" font-size="9">platform adapter</text>
<text x="276" y="343" text-anchor="middle" fill="#8b949e" font-size="9">Discord, Slack, ...</text>
</g>
<!-- ==================== REDIS MQ ==================== -->
<g filter="url(#shadow)">
<rect x="420" y="168" width="120" height="132" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="420" y="168" width="120" height="5" rx="5" fill="#f0883e"/>
<rect x="420" y="168" width="120" height="5" fill="#f0883e"/>
<rect x="420" y="171" width="120" height="2" fill="#161b22"/>
<text x="480" y="198" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Redis MQ</text>
<line x1="438" y1="210" x2="522" y2="210" stroke="#30363d" stroke-width="1"/>
<text x="480" y="228" text-anchor="middle" fill="#8b949e" font-size="9">inbound queues</text>
<text x="480" y="243" text-anchor="middle" fill="#8b949e" font-size="9">event pub/sub</text>
<text x="480" y="258" text-anchor="middle" fill="#8b949e" font-size="9">node heartbeats</text>
<text x="480" y="273" text-anchor="middle" fill="#8b949e" font-size="9">workstream routing</text>
<text x="480" y="288" text-anchor="middle" fill="#8b949e" font-size="9">cluster state</text>
</g>
<!-- ==================== CLUSTER NODES ==================== -->
<!-- Cluster outline -->
<rect x="598" y="100" width="204" height="310" rx="8" fill="none" stroke="#30363d" stroke-width="1" stroke-dasharray="4,3"/>
<text x="700" y="422" text-anchor="middle" fill="#30363d" font-size="9" letter-spacing="1">CLUSTER</text>
<!-- Node A -->
<g filter="url(#shadow)">
<rect x="614" y="118" width="170" height="100" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="118" width="170" height="5" rx="5" fill="#f47067"/>
<rect x="614" y="118" width="170" height="5" fill="#f47067"/>
<rect x="614" y="121" width="170" height="2" fill="#161b22"/>
<text x="699" y="142" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Node A</text>
<line x1="632" y1="152" x2="766" y2="152" stroke="#30363d" stroke-width="1"/>
<!-- Bridge -->
<rect x="626" y="162" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="661" y="180" text-anchor="middle" fill="#8b949e" font-size="9">bridge</text>
<!-- Server -->
<rect x="704" y="162" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="739" y="180" text-anchor="middle" fill="#8b949e" font-size="9">server</text>
<!-- Arrow bridge to server -->
<line x1="696" y1="176" x2="702" y2="176" stroke="#484f58" stroke-width="1" marker-end="url(#arrow)"/>
<!-- Tools label -->
<text x="699" y="206" text-anchor="middle" fill="#484f58" font-size="8">14 tools + MCP</text>
</g>
<!-- Node B -->
<g filter="url(#shadow)">
<rect x="614" y="238" width="170" height="100" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="238" width="170" height="5" rx="5" fill="#f47067"/>
<rect x="614" y="238" width="170" height="5" fill="#f47067"/>
<rect x="614" y="241" width="170" height="2" fill="#161b22"/>
<text x="699" y="262" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Node B</text>
<line x1="632" y1="272" x2="766" y2="272" stroke="#30363d" stroke-width="1"/>
<!-- Bridge -->
<rect x="626" y="282" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="661" y="300" text-anchor="middle" fill="#8b949e" font-size="9">bridge</text>
<!-- Server -->
<rect x="704" y="282" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="739" y="300" text-anchor="middle" fill="#8b949e" font-size="9">server</text>
<!-- Arrow bridge to server -->
<line x1="696" y1="296" x2="702" y2="296" stroke="#484f58" stroke-width="1" marker-end="url(#arrow)"/>
<!-- Tools label -->
<text x="699" y="326" text-anchor="middle" fill="#484f58" font-size="8">14 tools + MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
<!-- OpenAI -->
<g filter="url(#shadow)">
<rect x="870" y="130" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="130" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="130" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="133" width="140" height="2" fill="#161b22"/>
<text x="940" y="153" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">OpenAI</text>
<text x="940" y="167" text-anchor="middle" fill="#8b949e" font-size="9">GPT-5, o-series</text>
</g>
<!-- Anthropic -->
<g filter="url(#shadow)">
<rect x="870" y="196" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="196" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="196" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="199" width="140" height="2" fill="#161b22"/>
<text x="940" y="219" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Anthropic</text>
<text x="940" y="233" text-anchor="middle" fill="#8b949e" font-size="9">Claude 4.5 / 4.6</text>
</g>
<!-- Local / vLLM -->
<g filter="url(#shadow)">
<rect x="870" y="262" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="262" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="262" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="265" width="140" height="2" fill="#161b22"/>
<text x="940" y="285" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Local / vLLM</text>
<text x="940" y="299" text-anchor="middle" fill="#8b949e" font-size="9">llama.cpp, NIM</text>
</g>
<!-- ==================== STORAGE ==================== -->
<g filter="url(#shadow)">
<rect x="614" y="450" width="170" height="52" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="450" width="170" height="5" rx="5" fill="#bc8cff"/>
<rect x="614" y="450" width="170" height="5" fill="#bc8cff"/>
<rect x="614" y="453" width="170" height="2" fill="#161b22"/>
<text x="699" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="699" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
</g>
<text x="699" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
<!-- ==================== CONNECTION LINES ==================== -->
<!-- CLIENT -> GATEWAY connections -->
<!-- Browser -> Console -->
<line x1="150" y1="197" x2="214" y2="155" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- Discord -> Channel -->
<line x1="150" y1="337" x2="214" y2="325" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- CLI -> direct to Node A server (top path, curved) -->
<path d="M 150 131 C 200 131, 200 100, 400 100 L 400 100 C 500 100, 570 140, 612 168" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.5" fill="none" stroke-dasharray="6,3" marker-end="url(#arrow-blue)"/>
<text x="370" y="96" fill="#484f58" font-size="8" text-anchor="middle">direct</text>
<!-- SDK -> Redis (direct push) -->
<line x1="150" y1="267" x2="418" y2="240" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- GATEWAY -> REDIS connections -->
<!-- Console -> Redis -->
<line x1="336" y1="160" x2="418" y2="200" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-green)"/>
<!-- Channel -> Redis -->
<line x1="336" y1="318" x2="418" y2="272" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-green)"/>
<!-- REDIS -> NODE connections -->
<!-- Redis -> Node A bridge -->
<line x1="540" y1="210" x2="624" y2="176" stroke="#f0883e" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-orange)"/>
<!-- Redis -> Node B bridge -->
<line x1="540" y1="260" x2="624" y2="296" stroke="#f0883e" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-orange)"/>
<!-- Console -> Node (proxy, dashed) -->
<path d="M 336 147 C 380 130, 500 108, 612 145" stroke="#3fb950" stroke-width="1" stroke-opacity="0.4" fill="none" stroke-dasharray="4,3" marker-end="url(#arrow-green)"/>
<text x="468" y="120" fill="#484f58" font-size="8" text-anchor="middle">proxy</text>
<!-- NODE -> LLM connections -->
<!-- Node A -> LLM providers -->
<line x1="784" y1="168" x2="868" y2="155" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.5" marker-end="url(#arrow-coral)"/>
<line x1="784" y1="176" x2="868" y2="219" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.3"/>
<line x1="784" y1="180" x2="868" y2="282" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.2"/>
<!-- Node B -> LLM providers -->
<line x1="784" y1="288" x2="868" y2="163" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.2"/>
<line x1="784" y1="296" x2="868" y2="222" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.3"/>
<line x1="784" y1="300" x2="868" y2="288" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.5" marker-end="url(#arrow-coral)"/>
<!-- NODE -> STORAGE connections -->
<line x1="680" y1="338" x2="680" y2="448" stroke="#bc8cff" stroke-width="1.2" stroke-opacity="0.4" stroke-dasharray="4,3" marker-end="url(#arrow-muted)"/>
<line x1="718" y1="218" x2="718" y2="236" stroke="#484f58" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="2,2"/>
<!-- Extensibility hint -->
<text x="699" y="392" text-anchor="middle" fill="#30363d" font-size="10">...</text>
<!-- Event flow: Bridges -> Redis (dashed, bidirectional feel) -->
<line x1="624" y1="186" x2="542" y2="220" stroke="#f0883e" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="3,3"/>
<line x1="624" y1="286" x2="542" y2="250" stroke="#f0883e" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="3,3"/>
<text x="574" y="242" fill="#484f58" font-size="7" text-anchor="middle">events</text>
<!-- ==================== FLOW LABELS ==================== -->
<!-- Interactive flow label -->
<rect x="30" y="395" width="10" height="10" rx="2" fill="none" stroke="#58a6ff" stroke-width="1.5" stroke-dasharray="3,2"/>
<text x="46" y="404" fill="#8b949e" font-size="9">interactive (direct)</text>
<!-- Queue flow label -->
<rect x="160" y="395" width="10" height="10" rx="2" fill="none" stroke="#f0883e" stroke-width="1.5"/>
<text x="176" y="404" fill="#8b949e" font-size="9">queue-driven</text>
<!-- Proxy/event label -->
<rect x="275" y="395" width="10" height="10" rx="2" fill="none" stroke="#3fb950" stroke-width="1.5" stroke-dasharray="3,2"/>
<text x="291" y="404" fill="#8b949e" font-size="9">proxy / events</text>
<!-- ==================== BOTTOM DETAILS ==================== -->
<line x1="30" y1="430" x2="1170" y2="430" stroke="#21262d" stroke-width="1"/>
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#f47067" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">target_node set &#x2192; route to specific node queue</text>
<circle cx="44" cy="494" r="3" fill="#f0883e" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">ws_id set &#x2192; route to owning node</text>
<circle cx="44" cy="514" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="517" fill="#484f58" font-size="9">neither &#x2192; shared queue, any node picks up</text></svg>

After

Width:  |  Height:  |  Size: 18 KiB

+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a1b0361c466327d0011a488847ea3c0365983713537d4a7c27cd7f5538ba33c
size 164829
oid sha256:d8ce6d2a43a991655c3f64a20b6e810fdb2f78eb767acc3d3d1b8d2c9f443181
size 165011
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7d75da92a657525bcbb7a425dc6c8d3cafe074c3ac3bff3cf4b1d44aea607b50
size 330156
oid sha256:c9daca81971ba7a8ed6736d23d5373c69435158fa6240b9880d14fc4759ab580
size 329673
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:637458e0d78df82752746e519cd7300a830c8ce211f21625694ad0c162ca316d
size 481637
oid sha256:01fbb3338df6426cefc2811541a865f268673b4febf32f524c264d120bc068fa
size 589546
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dc3b64c9e48153641af62ed43fbc1d89a31d1a8a61e7e71cfc550c805000310d
size 288290
oid sha256:da9d32000e3d92d92ce621661ced60f276f9b5be652f5ed6123b400505415f4a
size 319702
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b842683d238664a3e35d04358fecfc56cefd013f7dca5f13357b0376f881e1b3
size 245043
oid sha256:6dd3c923d1e1c49b5f91d8d342fb4b0d49a46d432460379ad146a9e3b075a05a
size 277234
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b22d5980fe5cc4b8466ba0797113dc8fa83fab8df24b5dacceaf97e62e2e25b0
size 187649
oid sha256:d17f3feacf7bc9f64dfea19464143bc9b6ef0da5d55e6d57c0bc5a73d5724eba
size 184466
+1 -1
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:90e4f74be795b530e711faa87bc6eb2b3bf6abb68d8fac8ebff7aaf30c6fbe53
oid sha256:09535722ba975e47cf0557a40b6c481f125ff2022c396f79715c3bba9f715871
size 222032
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d33b9b3affcdb07086b5aebca8a3b9c2b009cdfc6f360950a0e72e65fbcb8f17
size 201602
oid sha256:ed457b10b534b5fc2a5e190b281d7ded4dd1615da2229d67a373cf5dddccd059
size 201601
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:adac93a0bb062d7199b819a600a0983ff011a75d16928fb80322cbb41f9284ea
size 158866
oid sha256:7896c6e041b6dbb89d034468fa980c8fe645df5eb969d45ef966ccc6399edac2
size 200083
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:69f201cff948cb0a19810b7c4ad26d346f869ee2dd3141eba4f353332efa2e21
size 373649
oid sha256:35cf3a6942f62dabcbbe012ac2f9e6f155332c894692981b076de5a25c1f3330
size 374055
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:97e7210cd8f1ad195f4d5e25e778d82df3c08c5c6e0f09722e84a7a453714867
size 411664
oid sha256:a74b4b8b5dbfb1a51a01100b731477968942b01218bad9451a3d5a9cb3003294
size 411665
+1 -1
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c3214ef416c1dfe4fa17834c2b6f4071a8093cfdb2b862848ca79938f726a13
oid sha256:84524f4bc900708ac8adf081591d336f862830188eb8505e71a0f071b339d923
size 252599
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9823a41e09611c5c0530d9fc12ad4139cfcc3ae238dc665b2888ec94d7d6781
size 195708
oid sha256:e7c3e40c10425d721f833390ae3531c09af501157fd3142531ba4eba86ff719d
size 197112
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0c615984373b4893b6cc5755604f137541e9d391862122746a4fcbae63543563
size 201041
oid sha256:5faa5335152685cf1c8bf77ed93847d751cde59e1afed651e5991113f2f0f31b
size 242670
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1380065cbb5f95b5ea7dc6b2a00986c455b82888af60784980dffbd936460dcf
size 431129
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f0f6097840fccdbfe16cd5e4c9f5d063b2c36942460a944df68b8ec947e63ea3
size 221452
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
size 258547
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a889d4bb84c4afa3c822c3acb7021395a9463462f7aeae5382e583b783412814
size 144960
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8e6dc5142c7908314ce01229b3c4f13bf9450adcbb62a178838bd4cf81d9f4da
size 250417
+48 -6
View File
@@ -27,6 +27,9 @@ Console dashboard: http://localhost:8090
| `server` | 8080 | default | Web UI + chat workstreams + LLM |
| `bridge` | — | default | Redis-to-HTTP bridge (multi-node routing) |
| `console` | 8090 | default | Cluster dashboard |
| `channel` | — | production | Channel gateway (Discord, Slack, etc.) |
| `server-1``server-10` | — | cluster | 10-node server fleet (PostgreSQL required) |
| `bridge-1``bridge-10` | — | cluster | Matching bridge fleet |
| `sim` | — | sim | Multi-node cluster simulator |
## Profiles
@@ -37,6 +40,18 @@ Console dashboard: http://localhost:8090
docker compose up
```
**Production** — adds PostgreSQL and the channel gateway. Requires `POSTGRES_PASSWORD` and (for Discord) `TURNSTONE_DISCORD_TOKEN`:
```bash
docker compose --profile production up
```
**Cluster** — 10-node server/bridge fleet sharing PostgreSQL and Redis. Access all nodes via the console at `:8090`. Requires `POSTGRES_PASSWORD`:
```bash
docker compose --profile cluster up
```
**Sim** — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:
```bash
@@ -84,8 +99,35 @@ All configuration is via environment variables in `.env` (copy from `.env.exampl
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_AUTH_ENABLED` | — | Set to `1` to require Bearer token auth |
| `TURNSTONE_AUTH_TOKEN` | — | Shared auth token for server/bridge/console |
| `TURNSTONE_AUTH_ENABLED` | — | Set to `1` to require authentication |
| `TURNSTONE_AUTH_TOKEN` | — | Config-file token for server/bridge/console (backward compat, works alongside JWT) |
| `TURNSTONE_JWT_SECRET` | — | Secret key for signing JWTs (required when using user identity / JWT auth) |
### Database
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql://user:pass@db:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.
> **First-time setup:** After deploying with auth enabled, create an initial admin user by running `turnstone-admin create-user` inside the container:
>
> ```bash
> docker compose exec server turnstone-admin create-user --username admin --name "Admin"
> ```
>
> You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass `--token --scopes read,write,approve` to also generate an initial API token.
### Channel Gateway
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord adapter) |
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages through Redis MQ to the bridge and server. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
### Simulator
@@ -101,13 +143,13 @@ All configuration is via environment variables in `.env` (copy from `.env.exampl
## Scaling
Scale to multiple server/bridge pairs:
For multi-node testing, use the `cluster` profile which provides 10 dedicated server+bridge pairs with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
docker compose up --scale server=3 --scale bridge=3
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
Each bridge auto-generates a unique node ID from its container hostname. When scaling `server`, remove the host port mapping (or use a reverse proxy) to avoid port conflicts.
The default `server` and `bridge` also run alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
## Volumes
@@ -128,7 +170,7 @@ docker compose build
docker compose build --no-cache
```
All five entry points are installed in a single image: `turnstone-server`, `turnstone-bridge`, `turnstone-console`, `turnstone-sim`, `turnstone-eval`.
All entry points are installed in a single image: `turnstone-server`, `turnstone-bridge`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`, `turnstone-sim`, `turnstone-eval`.
## Cleanup
+184
View File
@@ -0,0 +1,184 @@
# Governance
Turnstone governance provides role-based access control (RBAC), tool execution
policies, prompt templates, usage tracking, and audit logging for the admin
console.
## Architecture
See [diagram: 19-governance-architecture.puml](diagrams/19-governance-architecture.puml).
### RBAC (Roles & Permissions)
The permission model has two layers:
1. **Scopes** (legacy) — `read`, `write`, `approve`. Checked by `AuthMiddleware`
on every request based on URL path classification.
2. **Permissions** (granular) — 15 permission strings checked per-endpoint by
`require_permission()`.
**Built-in roles** (seeded by migration 008):
| Role | Permissions |
|------|-------------|
| admin | read, write, approve, admin.users, admin.roles, admin.orgs, admin.policies, admin.templates, admin.audit, admin.usage, admin.schedules, admin.watches, tools.approve, workstreams.create, workstreams.close |
| operator | read, write, workstreams.create, workstreams.close |
| viewer | read |
Custom roles can be created with any subset of the 15 valid permissions.
**Auth flow:**
1. User logs in (password or API token) → `_load_user_permissions()` aggregates
permissions from all assigned roles
2. `_permissions_to_scopes()` derives legacy scopes (any `admin.*``approve`)
3. JWT created with both `scopes` and `permissions` claims
4. Middleware checks scope → handler checks permission via `require_permission()`
### Tool Policies
Admin-defined rules that control tool execution:
- **Pattern matching**: Glob syntax via `fnmatch` (e.g., `bash*`, `file_write`, `*`)
- **Actions**: `allow` (auto-approve), `deny` (block), `ask` (normal approval flow)
- **Priority**: Higher priority evaluated first, first match wins
- **Enforcement**: `evaluate_tool_policies_batch()` called in `WebUI.approve_tools()`
before the `auto_approve` check
- **MCP granular policies**: MCP resources and prompts are evaluated using their
`approval_label` for fine-grained control:
- Resource reads: `mcp_resource__{uri}` (e.g., `mcp_resource__file:///docs/*` to allow,
`mcp_resource__*` to deny all)
- Prompt invocations: `mcp__{server}__{prompt}` (e.g., `mcp__trusted__*` to allow,
`mcp__*` to require approval for all)
- Built-in tools continue to use `func_name` for backward compatibility
### Prompt Templates
Admin-curated system message templates injected at workstream startup:
- **Runtime behavior**: Templates are loaded once at session creation and injected
into the system message *before* user `instructions`. Templates set the baseline;
instructions customize per-workstream behavior.
- **Default templates**: All `is_default=true` templates auto-apply to new
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--template <name>` CLI flag, `template` field on
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
config, and channel adapter config. An explicit template *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
`{{node_id}}` (server node ID). Unrecognized placeholders are kept as-is.
- **Runtime switching**: `/template <name>` to switch, `/template clear` to revert
to defaults, `/template` to show current. Persisted across resume.
- **Categories**: general, engineering, support, custom, mcp
- **Content limit**: 32 KB per template (enforced on create/update)
- **Storage**: `prompt_templates` table with JSON `variables` array. Migration 010
adds `template` column to `scheduled_tasks`.
- **MCP sync**: MCP server prompts auto-sync into prompt_templates with
`origin="mcp"`, `mcp_server` set, and `readonly=True`. Manual templates take
precedence on name collision. MCP-synced content updates reset `is_default` to
prevent compromised servers from injecting defaults. Admin UI shows origin badge
and disables edit/delete for MCP-sourced templates.
### Usage Tracking
Per-LLM-request token and tool call metrics:
- **Recording**: `on_status()` in `WebUI` records a `usage_event` after each
LLM response with prompt/completion tokens, tool call count, model, ws_id
- **Querying**: `GET /v1/api/admin/usage` with `group_by` (day/hour/model/user)
and time range filtering
- **Pruning**: `prune_usage_events(retention_days=90)` and
`prune_audit_events(retention_days=365)` run automatically via the
console scheduler's periodic cleanup cycle
### Audit Logging
Append-only trail of admin actions:
- **Recording**: `record_audit()` helper called from all admin mutation handlers
- **Events captured**: user.create, user.delete, token.create, token.revoke,
channel.link, channel.unlink, role.create, role.update, role.delete,
role.assign, role.unassign, policy.create, policy.update, policy.delete,
template.create, template.update, template.delete, org.update
- **Querying**: `GET /v1/api/admin/audit` with action/user/time filters + pagination
## Database Schema
Migration 008 adds 7 tables:
| Table | Purpose |
|-------|---------|
| `orgs` | Organizations (single default org for now) |
| `roles` | Named permission bundles (3 builtin + custom) |
| `user_roles` | User-to-role assignments (composite PK) |
| `tool_policies` | Per-tool approve/deny/ask rules |
| `prompt_templates` | Reusable system message templates |
| `usage_events` | Per-request token/tool metrics |
| `audit_events` | Admin action log |
Also adds `org_id` column to `users` table.
## API Endpoints
All under `/v1/api/admin/` (requires `approve` scope + granular permission).
| Group | Endpoints | Permission |
|-------|-----------|------------|
| Users / Tokens / Channels | 9 (CRUD) | `admin.users` |
| Roles | 7 (CRUD + assignment) | `admin.roles` / `admin.users` |
| Orgs | 3 (list, get, update) | `admin.orgs` |
| Tool Policies | 4 (CRUD) | `admin.policies` |
| Prompt Templates | 4 (CRUD) | `admin.templates` |
| Schedules | 6 (CRUD + runs) | `admin.schedules` |
| Watches | 3 (list, create, cancel) | `admin.watches` |
| Usage | 1 (aggregated query) | `admin.usage` |
| Audit | 1 (paginated, filtered) | `admin.audit` |
Full OpenAPI spec at `/openapi.json` and Swagger UI at `/docs`.
## Admin Console UI
5 new tabs added to the admin panel (10 total):
- **Roles** — CRUD roles, permission checkbox grid, user role assignment modal
- **Policies** — CRUD tool policies with colored action badges (green/red/amber)
- **Templates** — CRUD prompt templates with wide modal, textarea editor
- **Usage** — Summary readouts + CSS bar chart, time range + group-by selectors
- **Audit** — Filterable log with relative timestamps, load-more pagination
Tabs are permission-gated: hidden if the user lacks the required permission.
## SDK
Both Python and TypeScript console SDKs expose governance methods:
**Python** (`TurnstoneConsole` / `AsyncTurnstoneConsole`):
- `list_roles()`, `create_role()`, `update_role()`, `delete_role()`
- `list_user_roles()`, `assign_role()`, `unassign_role()`
- `list_orgs()`, `get_org()`, `update_org()`
- `list_policies()`, `create_policy()`, `update_policy()`, `delete_policy()`
- `list_templates()`, `create_template()`, `update_template()`, `delete_template()`
- `get_usage(since, group_by=...)`, `get_audit(action=..., limit=...)`
**TypeScript** (`TurnstoneConsole`):
- Same methods with camelCase naming and typed interfaces
## Security Considerations
- **Privilege escalation prevented**: `admin_assign_role` blocks self-assignment
and requires caller to hold a superset of the target role's permissions
- **Permission validation**: Role create/update validates permissions against
a 15-item allowlist (`_VALID_PERMISSIONS`)
- **Self-deletion blocked**: `admin_delete_user` rejects attempts to delete
your own account (matching the self-assignment guard on role endpoints)
- **Field allowlists**: Storage `update_*` methods filter fields against
allowlists (`_ROLE_MUTABLE`, `_POLICY_MUTABLE`, etc.) — handler bugs
cannot overwrite `role_id`, `builtin`, `created`, or other protected columns
- **Bootstrap safety**: `handle_auth_setup` fails and rolls back if admin role
assignment fails, preventing locked-out first user
- **API token RBAC**: `_authenticate_api_token` loads permissions from user's
roles, ensuring API tokens are subject to RBAC enforcement
- **Policy evaluation is fail-open**: If storage is unavailable, tool policies
degrade to the existing approval flow (not auto-approve)
- **Audit IP resolution**: `_audit_context()` prefers `X-Forwarded-For` for
client IP when behind a reverse proxy, falling back to `request.client.host`
+77 -16
View File
@@ -15,8 +15,10 @@ The Python SDK is included in the `turnstone` package — no extra install requi
```python
from turnstone.sdk import TurnstoneServer
# Synchronous client
with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
# Synchronous client — login with username/password
with TurnstoneServer("http://localhost:8080") as client:
client.login(username="alice", password="s3cret")
# Create a workstream
ws = client.create_workstream(name="Analysis")
@@ -33,6 +35,15 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
client.close_workstream(ws.ws_id)
```
Alternatively, authenticate with an API token:
```python
with TurnstoneServer("http://localhost:8080") as client:
client.login(token="ts_abc123...")
ws = client.create_workstream(name="CI run")
result = client.send_and_wait("Run the test suite.", ws.ws_id)
```
### Async Client
```python
@@ -40,7 +51,8 @@ import asyncio
from turnstone.sdk import AsyncTurnstoneServer
async def main():
async with AsyncTurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
async with AsyncTurnstoneServer("http://localhost:8080") as client:
await client.login(username="alice", password="s3cret")
ws = await client.create_workstream(name="demo")
async for event in client.stream_events(ws.ws_id):
if event.type == "content":
@@ -63,12 +75,15 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| | `plan_feedback(*, ws_id, feedback)` | `StatusResponse` |
| | `command(*, ws_id, command)` | `StatusResponse` |
| | `cancel(ws_id)` | `StatusResponse` |
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
| | `stream_global_events()` | `Iterator[ServerEvent]` |
| **High-level** | `send_and_wait(message, ws_id, *, timeout, on_event)` | `TurnResult` |
| **Sessions** | `list_sessions()` | `ListSessionsResponse` |
| **Auth** | `login(token)` | `AuthLoginResponse` |
| **Saved** | `list_saved_workstreams()` | `ListSavedWorkstreamsResponse` |
| **Auth** | `login(username=..., password=...)` | `AuthLoginResponse` |
| | `login(token="ts_xxx")` | `AuthLoginResponse` |
| | `logout()` | `StatusResponse` |
| | `auth_status()` | `AuthStatusResponse` |
| **Health** | `health()` | `HealthResponse` |
### Console Client API
@@ -81,9 +96,17 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
| | `nodes(*, sort, limit, offset)` | `ClusterNodesResponse` |
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
| | `node_detail(node_id)` | `NodeDetailResponse` |
| | `snapshot()` | `ClusterSnapshotResponse` |
| | `create_workstream(*, node_id, name, model, initial_message)` | `ConsoleCreateWsResponse` |
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
| | `get_schedule(task_id)` | `ScheduleInfo` |
| | `update_schedule(task_id, *, name=..., enabled=..., ...)` | `ScheduleInfo` |
| | `delete_schedule(task_id)` | `StatusResponse` |
| | `list_schedule_runs(task_id, *, limit=50)` | `ListScheduleRunsResponse` |
| **Streaming** | `stream_cluster_events()` | `Iterator[ClusterEvent]` |
| **Auth** | `login(token)` / `logout()` | `AuthLoginResponse` / `StatusResponse` |
| **Auth** | `login(username=..., password=...)` / `login(token="ts_xxx")` | `AuthLoginResponse` |
| | `logout()` | `StatusResponse` |
| **Health** | `health()` | `ConsoleHealthResponse` |
### Event Types
@@ -107,6 +130,7 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `error` | `ErrorEvent` | `message` |
| `info` | `InfoEvent` | `message` |
| `stream_end` | `StreamEndEvent` | — |
| `cancelled` | `CancelledEvent` | — |
**Global events** (from `stream_global_events()`):
@@ -125,6 +149,9 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `node_lost` | `NodeLostEvent` | `node_id` |
| `cluster_state` | `ClusterStateEvent` | `ws_id`, `node_id`, `state`, `tokens` |
| `ws_created` | `ClusterWsCreatedEvent` | `ws_id`, `node_id`, `name` |
| `ws_closed` | `ClusterWsClosedEvent` | `ws_id` |
| `ws_rename` | `ClusterWsRenameEvent` | `ws_id`, `name` |
| `snapshot` | `ClusterSnapshotEvent` | `nodes`, `overview`, `timestamp` |
### TurnResult
@@ -165,10 +192,11 @@ Located at `sdk/typescript/`. Zero runtime dependencies for browsers; uses nativ
```typescript
import { TurnstoneServer } from "@turnstone/sdk";
const client = new TurnstoneServer({
baseUrl: "http://localhost:8080",
token: "tok_xxx",
});
const client = new TurnstoneServer({ baseUrl: "http://localhost:8080" });
// Login with username/password or API token
await client.login({ username: "alice", password: "s3cret" });
// or: await client.login({ token: "ts_abc123..." });
// Create workstream and send message
const ws = await client.createWorkstream({ name: "demo" });
@@ -188,16 +216,14 @@ for await (const event of client.streamEvents(ws.ws_id)) {
```typescript
import { TurnstoneConsole } from "@turnstone/sdk";
const console = new TurnstoneConsole({
baseUrl: "http://localhost:8081",
token: "tok_xxx",
});
const client = new TurnstoneConsole({ baseUrl: "http://localhost:8090" });
await client.login({ username: "alice", password: "s3cret" });
const overview = await console.overview();
const overview = await client.overview();
console.log(`Nodes: ${overview.nodes}, Workstreams: ${overview.workstreams}`);
// Stream cluster events
for await (const event of console.clusterEvents()) {
for await (const event of client.clusterEvents()) {
console.log(event.type, event);
}
```
@@ -256,3 +282,38 @@ sdk/typescript/ TypeScript SDK (npm package)
The Python SDK reuses Pydantic models from `turnstone/api/` directly — no schema duplication. The TypeScript SDK has hand-written interfaces matching those models.
Both SDKs follow the same design: typed methods for REST endpoints, async iterators for SSE streams, and a high-level `send_and_wait` method for simple request-response patterns.
---
## Authentication
When auth is enabled on the server, the SDK handles JWT-based authentication automatically.
### Login Flow
There are two ways to authenticate:
1. **Username + password** — calls `POST /v1/api/auth/login` with credentials. The server validates against the user database and returns a JWT.
2. **API token** — calls `POST /v1/api/auth/login` with a `ts_`-prefixed token string. The server looks up the token, resolves the associated user, and returns a JWT.
In both cases the server returns the JWT in the response body and as a `Set-Cookie` header. The SDK extracts the JWT and includes it as a `Bearer` token in the `Authorization` header on all subsequent requests.
```python
# Username + password
client.login(username="alice", password="s3cret")
# API token (created via admin API or turnstone-admin CLI)
client.login(token="ts_abc123...")
```
### Token Lifecycle
- JWTs have a configurable expiry (default: 24 hours).
- `client.auth_status()` returns the current user identity and scopes without refreshing the token.
- `client.logout()` clears the stored JWT from the client.
- If a request returns 401, the SDK raises `TurnstoneAPIError` — the caller is responsible for re-authenticating.
### Backward Compatibility
The config-file token (`TURNSTONE_AUTH_TOKEN`) still works as a simple Bearer token for environments that do not use the user/JWT system. When the server receives a non-JWT Bearer token, it falls back to the legacy token check.
+485
View File
@@ -0,0 +1,485 @@
# Security and Authentication
Turnstone uses a layered authentication system with three token types,
hierarchical scopes, and a split architecture where the console manages
credentials while individual server nodes validate JWTs locally.
---
## Token Types
### Config-file tokens
Static tokens defined in `config.toml` or the `TURNSTONE_AUTH_TOKEN`
environment variable. Validated in-memory using `hmac.compare_digest`
(timing-safe). Each token maps to a role that determines its scopes.
```toml
[[auth.tokens]]
value = "tok_legacy"
role = "full" # full → {read, write, approve}
```
Role mappings: `"read"``{read}`, `"full"``{read, write, approve}`.
Config tokens are sent directly as `Authorization: Bearer tok_legacy`
on every request. No JWT exchange is needed.
### API tokens
Database-backed tokens prefixed with `ts_`. Created via the admin CLI
(`turnstone-admin create-token`) or the console admin API. Stored as
SHA-256 hashes — the raw token is shown exactly once at creation and
never persisted in plaintext.
```
$ turnstone-admin create-token --user abc123 --scopes read,write --name "CI bot"
Token created: ts_a1b2c3d4e5f6...
(save this — it will not be shown again)
```
API tokens can be used directly as `Bearer ts_xxx` headers or exchanged
for a JWT via the login endpoint.
### JWTs
Short-lived session tokens (24 hours by default). Issued after
authenticating with username/password or by exchanging an API token.
HS256-signed with a shared secret. Validated locally on every service
node — no database call per request.
Claims:
| Claim | Description |
|-------|-------------|
| `sub` | User ID |
| `scopes` | Comma-separated scope list (`read,write,approve`) |
| `src` | Token source (`password`, `api_token`, `config`) |
| `iss` | Issuer — always `turnstone` |
| `aud` | Audience — `turnstone-server` or `turnstone-console` |
| `iat` | Issued-at timestamp |
| `exp` | Expiry timestamp |
The `aud` claim prevents cross-service token reuse — a JWT issued for the
console cannot be used to authenticate against a server node, and vice versa.
Tokens without an `aud` claim are accepted during the rollout window when
`audience` validation is not specified.
---
## Scope Model
Scopes are hierarchical — higher scopes imply all lower ones.
| Scope | Grants | Implies |
|-------|--------|---------|
| `read` | View workstreams, saved workstreams, history | — |
| `write` | Send messages, create/close workstreams | `read` |
| `approve` | Approve tool calls, admin endpoints | `read`, `write` |
### Path-to-scope mapping
| Method | Path pattern | Required scope |
|--------|-------------|----------------|
| GET | Any protected path | `read` |
| POST | `/api/send`, `/api/plan`, `/api/command` | `write` |
| POST | `/api/workstreams/new`, `/api/workstreams/close` | `write` |
| POST | `/api/cluster/workstreams/new` | `write` |
| POST | `/api/approve` | `approve` |
| Any | `/api/admin/*` | `approve` |
Public paths bypass authentication entirely: `/`, `/health`, `/metrics`,
`/static/*`, `/shared/*`, `/docs`, `/openapi.json`, `/api/auth/login`,
`/api/auth/logout`, `/api/auth/status`, `/api/auth/setup`.
### RBAC (Granular Permissions)
> See also: [Governance documentation](governance.md)
Scopes provide coarse endpoint-level access control. For finer-grained
enforcement, the governance layer adds 15 named permissions checked
per-endpoint by `require_permission()`. Permissions are bundled into
roles; users are assigned roles via the `user_roles` join table.
At login, `_load_user_permissions()` aggregates all permissions from
the user's assigned roles. `_permissions_to_scopes()` derives legacy
scopes for backward compatibility (e.g., any `admin.*` permission
implies the `approve` scope). The JWT carries both `scopes` and
`permissions` claims.
Three built-in roles are seeded by migration 008:
| Role | Permissions |
|------|-------------|
| admin | All 15 permissions |
| operator | read, write, workstreams.create, workstreams.close |
| viewer | read |
Custom roles can be created with any subset of the valid permissions.
Role creation and update validate permissions against a static allowlist.
Self-assignment is blocked, and assigning a role requires the caller to
hold a superset of the target role's permissions.
---
## Login Flows
### Username and password
```
POST /v1/api/auth/login
Content-Type: application/json
{"username": "admin", "password": "s3cret"}
```
Returns a JWT in the response body and sets an `HttpOnly` session cookie.
### API token exchange
```
POST /v1/api/auth/login
Content-Type: application/json
{"token": "ts_a1b2c3d4e5f6..."}
```
The API token is hashed, looked up in the database, and exchanged for a
JWT with the token's scopes. This is the recommended flow for SDKs and
automated clients that need cookie-based sessions.
### Config-file tokens (direct)
Config tokens are validated per-request via `hmac.compare_digest`. No
login exchange is needed — include the token as a `Bearer` header:
```
Authorization: Bearer tok_legacy
```
### First-time setup
When no users exist in the database:
1. `GET /v1/api/auth/status` returns `{"setup_required": true}`
2. The UI presents a setup wizard
3. `POST /v1/api/auth/setup` creates the first admin user and returns a
JWT in one atomic step (no auth required — this is a public endpoint)
4. The endpoint returns `409 Conflict` if setup has already been completed
(i.e. users already exist in the database)
5. Subsequent admin requests require `approve` scope
The `/api/auth/setup` endpoint is available on both the server and
console. It validates input before creating the user:
- **username**: 1-64 ASCII characters
- **display_name**: required (non-empty)
- **password**: minimum 8 characters
```
POST /v1/api/auth/setup
Content-Type: application/json
{"username": "admin", "display_name": "Admin", "password": "strongpass"}
```
Response:
```json
{
"status": "ok",
"user_id": "u_abc123",
"username": "admin",
"role": "full",
"scopes": "approve,read,write",
"jwt": "eyJhbGciOiJIUzI1NiIs..."
}
```
The response also sets an `HttpOnly` session cookie containing the JWT,
so the browser is immediately authenticated after setup completes.
---
## Token Detection Order
The auth middleware inspects the `Authorization: Bearer <token>` header
and classifies the token:
1. **Contains `.`** → JWT → validate HS256 signature and expiry
2. **Starts with `ts_`** → API token → SHA-256 hash, database lookup
3. **Otherwise** → config-file token → `hmac.compare_digest` against
each configured token
If a session cookie is present and no `Authorization` header is sent,
the cookie value is treated as a JWT (step 1).
---
## Password Storage
Passwords are hashed with **bcrypt** using a random salt per password.
Plaintext passwords are only accepted over HTTPS in production
deployments.
---
## Cookie Security
| Attribute | Value | Purpose |
|-----------|-------|---------|
| `HttpOnly` | `true` | Prevents JavaScript access |
| `SameSite` | `Lax` | CSRF protection |
| `Path` | `/` | Available to all routes |
| `Max-Age` | 24 hours | Matches JWT expiry |
| `Secure` | `true` (default) | Always set unless explicitly disabled for dev |
---
## JWT Configuration
| Setting | Config key | Env var | Default |
|---------|-----------|---------|---------|
| Signing secret | `[auth] jwt_secret` | `TURNSTONE_JWT_SECRET` | Auto-generated ephemeral (warning logged) |
| Expiry | `[auth] jwt_expiry_hours` | — | 24 hours |
| Algorithm | — | — | HS256 (not configurable) |
| Minimum secret length | — | — | 32 characters (warning if shorter) |
All service nodes that need to validate JWTs must share the same signing
secret. If no secret is configured, an ephemeral key is generated at
startup and a warning is logged — JWTs will not survive restarts or work
across nodes.
The bridge and console **require** `TURNSTONE_JWT_SECRET` when no
`--auth-token` is provided. They exit with an error if the secret is
missing, since ephemeral secrets would silently break inter-service
communication.
---
## Admin API Endpoints
All admin endpoints require `approve` scope.
### Users
| Method | Path | Description |
|--------|------|-------------|
| POST | `/v1/api/admin/users` | Create user (username, display_name, password) |
| GET | `/v1/api/admin/users` | List all users |
| DELETE | `/v1/api/admin/users/{user_id}` | Delete user and cascade tokens |
### API tokens
| Method | Path | Description |
|--------|------|-------------|
| POST | `/v1/api/admin/users/{user_id}/tokens` | Create API token (returns raw value once) |
| GET | `/v1/api/admin/users/{user_id}/tokens` | List tokens (prefix only, no hashes) |
| DELETE | `/v1/api/admin/tokens/{token_id}` | Revoke token |
---
## CLI Administration
The `turnstone-admin` command provides offline user and token management:
```
turnstone-admin create-user --username admin --name "Admin" [--password] [--token]
turnstone-admin create-token --user <user_id> --scopes read,write --name "CI bot"
turnstone-admin list-users
turnstone-admin list-tokens
turnstone-admin revoke-token <token_id>
```
When `--password` is omitted, the CLI prompts interactively. When
`--token` is passed to `create-user`, an API token is created alongside
the user and printed to stdout.
---
## Database Schema
```sql
CREATE TABLE users (
user_id TEXT PRIMARY KEY,
username TEXT NOT NULL UNIQUE,
display_name TEXT NOT NULL,
password_hash TEXT NOT NULL,
created TEXT NOT NULL
);
CREATE TABLE api_tokens (
token_id TEXT PRIMARY KEY,
token_hash TEXT NOT NULL, -- SHA-256 of raw token
token_prefix TEXT NOT NULL, -- first 8 chars for display
user_id TEXT NOT NULL REFERENCES users(user_id),
name TEXT NOT NULL,
scopes TEXT NOT NULL, -- comma-separated
created TEXT NOT NULL,
expires TEXT -- nullable, ISO 8601
);
CREATE UNIQUE INDEX ix_api_tokens_hash ON api_tokens(token_hash);
CREATE TABLE channel_users (
channel_type TEXT NOT NULL,
channel_user_id TEXT NOT NULL,
user_id TEXT NOT NULL REFERENCES users(user_id),
created TEXT NOT NULL,
PRIMARY KEY (channel_type, channel_user_id)
);
```
The `sessions` and `workstreams` tables have a nullable `user_id`
column for attribution when auth is enabled.
---
## Revocation
- **API tokens**: Deleting a token via the admin API or CLI prevents new
JWTs from being issued with that token. Existing JWTs derived from the
token remain valid until they expire (at most 24 hours).
- **Config-file tokens**: Remove the token from `config.toml` and
restart the service. No JWTs are involved, so revocation is immediate.
- **JWTs**: Cannot be individually revoked. Rely on short expiry (24h)
and revoke the underlying credential to prevent renewal.
---
## Architecture
```
Console (cluster-wide) Server (per-node)
┌──────────────────────┐ ┌──────────────────────┐
│ User/Token CRUD (DB) │ │ JWT validation only │
│ Login: creds → JWT │ │ (shared signing key) │
│ Admin API endpoints │ │ Config tokens: hmac │
│ Storage: users, │ │ No auth DB needed │
│ api_tokens tables │ │ │
└──────────────────────┘ └──────────────────────┘
```
The console owns the credential database and handles all user/token
CRUD. Individual server nodes only need the JWT signing secret to
validate session tokens. Config-file tokens are validated locally
without any database.
### Proxy auth forwarding
When the console proxies requests to server nodes (via `/node/{id}/...`
routes), it uses a dedicated **service proxy token** with
`aud: turnstone-server` and `write` scope. The user's console JWT
(which has `aud: turnstone-console`) is **not** forwarded — it would be
rejected by the server's audience validation.
The proxy token is managed by a `ServiceTokenManager` that auto-rotates
1-hour JWTs, refreshing at 80% of lifetime. If `--auth-token` is
provided, that static token is used instead.
### Service-to-service authentication
The bridge and console collector use `ServiceTokenManager` for
auto-rotating JWTs when communicating with server nodes:
| Service | Identity | Scope | Audience | Purpose |
|---------|----------|-------|----------|---------|
| Bridge | `bridge` | `approve` | `turnstone-server` | Tool approval proxy, message relay |
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy | `console-proxy` | `write` | `turnstone-server` | Proxied API calls |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
Service tokens use 1-hour expiry with automatic refresh via
`ServiceTokenManager`. The bridge injects auth headers per-request via
httpx event hooks to ensure rotated tokens are picked up on SSE
reconnects.
Note that the channel gateway uses a distinct JWT audience
(`turnstone-channel`) from the server (`turnstone-server`) and console
(`turnstone-console`). A server-scoped JWT cannot authenticate to the
channel gateway endpoint, and vice versa.
---
## Configuration Reference
### config.toml
```toml
[auth]
enabled = true
jwt_secret = "your-secret-key-here"
jwt_expiry_hours = 24
[[auth.tokens]]
value = "tok_legacy"
role = "full"
```
### Environment variables
| Variable | Description |
|----------|-------------|
| `TURNSTONE_AUTH_ENABLED=1` | Enable authentication |
| `TURNSTONE_AUTH_TOKEN=tok_xxx` | Register a config-file token with `full` access |
| `TURNSTONE_JWT_SECRET=xxx` | JWT signing secret (must match across nodes) |
| `TURNSTONE_CORS_ORIGINS=` | CORS allowed origins (comma-separated; empty = same-origin only) |
---
## Login Rate Limiting
The `/api/auth/login` endpoint is protected by a dedicated
`LoginRateLimiter` (separate from the general API rate limiter).
Limits are enforced per-IP and per-username with a sliding window:
- **5 attempts** per **5-minute window** per key
- Failed logins record against both `ip:{client_ip}` and `user:{username}`
- Returns `429 Too Many Requests` with `Retry-After` header when exceeded
- Successful logins do not consume the budget
---
## CORS Policy
By default, no CORS headers are sent (same-origin only). To allow
cross-origin requests, set `TURNSTONE_CORS_ORIGINS`:
```bash
# Allow specific origins
TURNSTONE_CORS_ORIGINS=https://app.example.com,https://admin.example.com
# Allow all origins (development only)
TURNSTONE_CORS_ORIGINS=*
```
When the variable is empty or unset, the CORS middleware is not added
and browsers enforce same-origin policy.
---
## Security Properties
- **Timing-safe comparison** for config-file tokens via
`hmac.compare_digest` — no timing side-channel.
- **Hash-based lookup** for API tokens — the database stores only
SHA-256 hashes, eliminating timing attacks on token comparison.
- **Local JWT validation** — no network call or database query needed
per request on server nodes.
- **One-time display** of raw API tokens at creation. The plaintext is
never stored; `token_hash` never appears in API responses or logs.
- **Structured logging audit trail**`ctx_user_id` is set on every
authenticated request and injected into all log events.
- **Scope enforcement** at the middleware layer before any handler
executes. Path-to-scope mapping is defined statically.
- **JWT audience isolation** — server and console JWTs have distinct
`aud` claims, preventing cross-service token reuse.
- **Login brute-force protection** — per-IP and per-username rate
limiting on the login endpoint.
- **Secure cookies by default**`Secure` flag set unconditionally;
24-hour max-age matches JWT expiry.
- **CORS restriction** — no CORS headers by default (same-origin only).
- **Service JWT auto-rotation** — 1-hour expiry with transparent
refresh, eliminating long-lived static tokens for inter-service auth.
- **Secret strength validation** — warning logged when JWT secret is
shorter than 32 characters.
+381 -9
View File
@@ -1,6 +1,6 @@
# Tools Reference
turnstone exposes 14 built-in tools plus any number of external MCP tools to the
turnstone exposes 18 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -46,11 +46,12 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 14 tool definitions (sent to the model). |
| `TOOLS` | All 18 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 18 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -68,6 +69,9 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 18
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
@@ -113,6 +117,7 @@ Each item's `execute` callable is invoked:
- `remember` -- writes to persistent memory database (lightweight, always auto-approved)
- `recall` -- reads from persistent memory database
- `forget` -- deletes from persistent memory database (lightweight, always auto-approved)
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
**Requires user confirmation** (write operations, network access, side effects):
- `bash` -- arbitrary command execution
@@ -162,6 +167,9 @@ Every tool defines a `primary_key`. The mapping is:
| `remember` | `key` |
| `recall` | `query` |
| `forget` | `key` |
| `notify` | `message` |
| `read_resource` | `uri` |
| `use_prompt` | `name` |
---
@@ -183,15 +191,17 @@ Execute a bash command and return stdout + stderr.
### read_file
Read the contents of a file, returning numbered lines.
Read the contents of a file, returning numbered lines for text files or
base64-encoded image data for supported image formats.
| Parameter | Type | Required | Description |
|-----------|---------|----------|-------------|
| `path` | string | yes | Absolute or relative file path. |
| `offset` | integer | no | Line number to start from (1-based, default: 1). |
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. |
| `offset` | integer | no | Line number to start from (1-based, default: 1). Text files only. |
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. Text files only. |
- **What it does**: Reads the file and returns content with line numbers. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **What it does**: For text files, reads and returns content with line numbers. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns image data as multi-part content when the model supports vision, or a text description when it does not. SVG files are read as text. Images larger than 4 MB are rejected. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **Vision support**: Controlled by `ModelCapabilities.supports_vision`. All commercial OpenAI and Anthropic models have vision enabled. Local models (vLLM, llama.cpp, NIM) default to off — enable via `[models.*.capabilities] supports_vision = true` in config.toml.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
@@ -335,7 +345,7 @@ Plan before implementing -- an autonomous agent explores the codebase and writes
|-----------|--------|----------|-------------|
| `prompt` | string | yes | What to plan -- the goal, constraints, and scope. |
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<session_id>.md` (unique per session, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<ws_id>.md` (unique per workstream, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **Auto-approve**: No -- requires user confirmation, plus post-execution review gate.
- **Agent availability**: Not available to sub-agents (top-level only).
@@ -387,6 +397,108 @@ Remove a persistent memory by key.
---
## Notifications
### notify
Send a notification to a user or channel on an external platform.
| Parameter | Type | Required | Description |
|----------------|--------|----------|-------------|
| `message` | string | yes | Notification content (plain text, max 2000 chars). |
| `username` | string | no | Turnstone username — sends to all linked channels. |
| `channel_type` | string | no | Platform for direct targeting (`discord`). |
| `channel_id` | string | no | Platform-specific channel or user ID for direct targeting. |
| `title` | string | no | Optional short title (rendered as bold prefix). |
Provide either `username` for user-based targeting or `channel_type` +
`channel_id` for direct targeting. Do not combine both.
- **What it does**: Sends a notification via the channel gateway's HTTP endpoint (`POST /v1/api/notify`). The server queries the `services` table for healthy channel gateways, authenticates with a service JWT (`aud: turnstone-channel`), and delivers to the first healthy gateway. On failure, retries up to 2 additional times with backoff (1s, 3s). Rate-limited to 5 notifications per turn (counter only increments on success).
- **Auto-approve**: Yes — notifications are time-sensitive and auto-approved so the model can alert users urgently.
- **Agent availability**: `agent` and `task_agent`.
> See [Channel Integrations: Notifications](channels.md#notifications)
> for the full delivery flow, service registry details, and security
> measures.
---
### watch
Set up periodic polling of a shell command within the current workstream.
Results are injected back into the conversation as synthetic user messages,
triggering the model to respond and act. Use for monitoring CI/CD pipelines,
PR reviews, deployments, file changes, etc.
| Parameter | Type | Required | Description |
|-------------|---------|----------|-------------|
| `action` | string | yes | `create`, `list`, or `cancel`. |
| `command` | string | create | Shell command to poll periodically. |
| `poll_every`| string | no | Poll interval as duration (`30s`, `5m`, `1h`). Default: `5m`. |
| `stop_on` | string | no | Python expression for stop condition (see below). Omit for change detection. |
| `name` | string | create | Human-readable watch name (e.g. `pr-review`). Used as identifier for cancel. |
| `max_polls` | integer | no | Max poll cycles before auto-cancel. Default: 100. |
**Actions:**
- `create` — Start a new watch. Requires approval (same as bash — runs shell
commands). Persists to the `watches` table; the server-level `WatchRunner`
daemon polls every 15 seconds for due watches.
- `list` — Show all active watches in this workstream. Auto-approved.
- `cancel` — Stop a watch by name or ID prefix. Auto-approved.
**Stop condition DSL** — The `stop_on` parameter accepts a Python expression
evaluated after each poll. Available variables:
| Variable | Type | Description |
|---------------|------------|-------------|
| `output` | `str` | stdout (+stderr) of the command. |
| `data` | `Any` | `json.loads(output)`, or `None` if not valid JSON. |
| `exit_code` | `int` | Process exit code. |
| `prev_output` | `str|None` | Previous poll's stdout (`None` on first poll). |
| `changed` | `bool` | `True` if output differs from previous poll. |
Safe builtins: `len`, `str`, `int`, `float`, `bool`, `abs`, `min`, `max`,
`any`, `all`, `isinstance`, `sorted`. No `import`, `open`, `exec`, or
`eval`. Security model: equivalent to `bash` — the model already has shell
access.
**Examples:**
```
data["state"] == "MERGED"
"error" in output
exit_code != 0
changed and "ready" in output.lower()
data.get("mergedAt") is not None
```
**Lifecycle:**
1. Model calls `watch(action="create", ...)` — persisted to SQLite.
2. `WatchRunner` daemon polls for due watches every 15s.
3. Each poll runs the command, evaluates the condition.
4. When the condition fires (or max polls reached), the result is injected
as a synthetic user message and the watch auto-cancels.
5. If the workstream was evicted, it is restored before injection.
6. Watches survive server restart (overdue watches fire once on recovery).
**Constraints:**
- Max 5 active watches per workstream.
- Poll interval: 10s24h.
- Output truncated at 64 KB.
- Max 5 consecutive watch dispatches per worker thread (depth guard).
- Duplicate names rejected within the same workstream.
- **Auto-approve**: `create` requires approval; `list` and `cancel` are auto-approved.
- **Agent availability**: Main session only — not available to plan/task sub-agents.
> See [Watch Architecture](diagrams/png/18-watch-architecture.png) for the
> full poll → evaluate → dispatch flow.
---
## Summary Table
| Tool | Category | Auto-approve | agent | task_agent | primary_key |
@@ -405,11 +517,88 @@ Remove a persistent memory by key.
| `remember` | Memory | Yes | No | No | `key` |
| `recall` | Memory | Yes | No | No | `query` |
| `forget` | Memory | Yes | No | No | `key` |
| `notify` | Notify | Yes | Yes | Yes | `message` |
| `watch` | Monitor | No (create) | No | No | `command` |
| `read_resource`| MCP | No | Yes | Yes | `uri` |
| `use_prompt` | MCP | No | Yes | Yes | `name` |
| `tool_search`| Search | Yes | No | No | `query` |
---
## Dynamic Tool Search
When many MCP tools are connected, the total tool count can grow large enough to
consume significant context window tokens and reduce model accuracy. Dynamic tool
search addresses this by deferring tools the model is unlikely to need on the
current turn and letting it search for them on demand.
### Three-tier approach
Tool search uses the best available mechanism for each provider:
1. **Anthropic (native)** -- Models that support it receive `defer_loading: true`
on deferred tool definitions plus the `tool_search_tool_bm25_20251119` server-side
search tool. Anthropic's API handles search and expansion transparently.
2. **OpenAI GPT-5.4+ (native)** -- Models with hosted tool search receive
`defer_loading: true` on deferred definitions. The API handles search internally.
3. **vLLM / llama.cpp / NIM (client-side BM25)** -- A synthetic `tool_search`
function tool is injected into the tool list. When the model calls it,
`_exec_tool_search()` runs a pure-Python BM25 index over tool names and
descriptions, then expands the matched tools into the visible set.
### Configuration
Tool search is configured in `config.toml` under the `[tools]` section:
```toml
[tools]
search = "auto" # "auto", "on", or "off"
search_threshold = 20 # minimum total tool count to activate
search_max_results = 5 # max tools returned per search call
```
CLI flags override the config file:
- `--tool-search {auto,on,off}` -- force tool search on or off, or let turnstone
decide based on threshold (default: `auto`).
- `--tool-search-threshold N` -- minimum tool count to activate (default: 20).
- `--tool-search-max-results N` -- max results per search (default: 5).
### How it works
1. **Threshold check**: At session startup, `ToolSearchManager.should_activate()`
counts total tools (built-in + MCP). If the count is below the threshold, tool
search stays off and all tools are sent to the model directly.
2. **Partitioning**: When active, tools are split into two sets:
- **Always-on** -- the 18 built-in tools (members of `BUILTIN_TOOL_NAMES`).
These are always visible to the model.
- **Deferred** -- all MCP tools. These are not sent in the tool list unless
the model searches for them.
3. **Search and expand**: When the model calls `tool_search` (client-side) or the
provider's native search returns results, the matched tools are added to the
visible set via `expand_visible()`. Once expanded, a tool stays visible for
the remainder of the session.
4. **Multi-turn persistence**: Expanded tools are never removed. This avoids
confusing the model when it references a tool it discovered in an earlier turn.
### Agent exemption
Plan and task sub-agents do not use tool search. They operate on scoped tool
sets (`AGENT_TOOLS` for plan agents, `TASK_AGENT_TOOLS` for task agents) with
MCP tools merged in. Tool search is only active for the top-level session,
where the model can interactively search for tools it needs.
---
## MCP Tools (External)
> See also: [MCP Architecture diagram](diagrams/png/20-mcp-architecture.png)
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/)
(MCP) for connecting external tool servers — GitHub, databases, filesystems, or any
MCP-compatible service.
@@ -421,13 +610,17 @@ MCP-compatible service.
2. **Discovery**: At startup, `MCPClientManager` connects to each configured server
(via stdio subprocess or HTTP), performs the MCP `initialize` handshake, and calls
`tools/list` to discover available tools.
`tools/list` to discover available tools. During the handshake, the manager checks
each server's capabilities for `tools.listChanged` support (push notifications).
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 14 built-in tools via
4. **Merging**: MCP tools are appended after the 18 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
[Dynamic Tool Search](#dynamic-tool-search) above).
5. **Dispatch**: When the LLM calls an MCP tool, `_prepare_mcp_tool()` builds a
generic approval preview and `_exec_mcp_tool()` calls `MCPClientManager.call_tool_sync()`,
@@ -500,3 +693,182 @@ MCP tools (3):
mcp__github__create_issue [MCP: github] Create a GitHub issue
mcp__postgres__query [MCP: postgres] Run a SQL query
```
### Dynamic tool refresh
MCP tool lists stay up-to-date without restart through three mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
2. **Periodic timer** -- Servers that do *not* support push notifications are polled
on a configurable interval (default 4 hours). The timer is staggered using a
launch-time seed (`monotonic_ns ^ pid`) so cluster nodes don't all hit MCP
servers simultaneously. Configure via `[mcp] refresh_interval` in `config.toml`
or `--mcp-refresh-interval SECONDS` on the CLI. Set to `0` to disable.
3. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
manual refresh attempts reconnection.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`, `_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```toml
[mcp]
refresh_interval = 14400 # seconds (default 4h), 0 to disable
```
```
/mcp refresh
MCP refresh complete:
github: +1 added
+ mcp__github__create_pr
postgres: no changes
/mcp refresh github
MCP refresh complete:
github: no changes
```
---
## MCP Resources
MCP servers can expose **resources** -- named data items (files, database rows,
API responses) addressable by URI. turnstone discovers resources at startup and
makes them available to the model via the `read_resource` built-in tool.
### Discovery
During the MCP `initialize` handshake, `MCPClientManager` checks each server's
capabilities for the `resources` capability. For servers that declare it:
1. `list_resources` fetches static resources (fixed URIs).
2. `list_resource_templates` fetches URI templates (parameterized patterns like
`db://tables/{table}/rows/{id}`).
Both are stored as `{uri, name, description, mimeType, server}` dicts and
merged into a unified catalog.
### Resource catalog in system message
The first 50 resources are injected into the system message as an XML-delimited
block so the model knows what URIs are available:
```xml
<mcp-resources>
file:///project/README.md Project readme
db://users/schema User table schema
</mcp-resources>
Use read_resource(uri='...') to access the resources listed above.
```
### read_resource tool
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `uri` | string | yes | The resource URI to read. |
- **What it does**: Reads the resource from its MCP server via `MCPClientManager.read_resource_sync()`. Returns text content for text resources or base64-encoded data for binary resources. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (reads external data).
- **Agent availability**: `agent` and `task_agent`.
### Capability guards
The `read_resource` tool schema is always loaded (it is a built-in JSON schema),
but resource discovery only runs for servers that declare the `resources`
capability. Servers without the capability contribute zero resources to the
catalog.
### Refresh
Resource lists stay current through the same three-tier mechanism as tool lists:
1. **Push** -- Servers declaring `resources.listChanged: true` send
`notifications/resources/list_changed`, triggering an immediate refresh.
2. **Periodic** -- Servers without push are polled on the configured refresh
interval (default 4 hours, same timer as tools).
3. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
---
## MCP Prompts
MCP servers can also expose **prompts** -- reusable message templates with
optional arguments. turnstone discovers prompts at startup for servers that
declare the `prompts` capability.
### Discovery
Prompt discovery mirrors resource discovery: `list_prompts` is called during
the `initialize` handshake. Each prompt is stored with its prefixed name
(`mcp__{server}__{prompt}`), description, and argument schema.
### use_prompt tool
| Parameter | Type | Required | Description |
|-------------|--------|----------|-------------|
| `name` | string | yes | The prompt name (e.g. `mcp__server__prompt_name`). |
| `arguments` | object | no | Key-value argument pairs for the prompt. Values must be strings. |
- **What it does**: Invokes an MCP prompt template by name via `MCPClientManager.get_prompt_sync()`, expanding it into messages. Returns the expanded prompt content formatted as `[role]: content` blocks joined with blank lines. The prompt catalog is listed in the system message so the model knows which prompts are available. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (invokes external prompt servers).
- **Agent availability**: `agent` and `task_agent`.
### Invocation
`MCPClientManager.get_prompt_sync()` calls the server's `get_prompt` method
with the provided arguments and returns the expanded messages. The `use_prompt`
built-in tool exposes this to the model as a function call.
### Governance Sync
Discovered MCP prompts are automatically synced into the `prompt_templates`
governance table as first-class governed templates:
- **Origin tracking**: MCP-sourced templates have `origin="mcp"` and
`mcp_server` set to the server name. Manual templates have
`origin="manual"`.
- **Read-only**: MCP-sourced templates are `readonly=True`. The admin API
returns 403 on update/delete attempts. The admin UI disables edit/delete
buttons and shows an origin badge.
- **Precedence**: If a manual template and MCP prompt share the same name,
the manual template wins and the MCP prompt is skipped (with a log
warning).
- **Lifecycle**: Templates are created on connect, updated on prompt list
refresh, and removed when the MCP server no longer exposes the prompt.
The sync runs automatically on connect, on `PromptListChangedNotification`,
and on manual `/mcp refresh`.
- **Schema**: Migration 009 adds `origin`, `mcp_server`, and `readonly`
columns to the `prompt_templates` table.
The `use_prompt` tool allows the model to invoke any discovered MCP prompt at
runtime. A catalog of up to 30 prompts is injected into the system message
inside `<mcp-prompts>` XML tags so the model can discover available prompts.
---
## MCP UI Visibility
MCP server, resource, and prompt counts are surfaced across the UI:
- **Server `/health` endpoint**: Returns `mcp.servers`, `mcp.resources`,
`mcp.prompts` when MCP is configured
- **Server UI**: Magenta status badge in the header showing server count,
with resource/prompt counts in tooltip
- **Console cluster status bar**: MCP metrics (servers/resources/prompts)
with magenta LED dot indicator, shown after a divider from workstream
metrics
- **Console node detail**: Per-node MCP summary showing server, resource,
and prompt counts
- **Console collector**: Aggregates MCP counts across all nodes in the
cluster overview
MCP indicators use the `--magenta` design token for consistent theming
across light and dark modes.
+32 -3
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "0.3.5"
version = "0.5.6"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -32,6 +32,9 @@ dependencies = [
"pydantic>=2.0",
"sqlalchemy>=2.0",
"alembic>=1.14",
"structlog>=24.1",
"PyJWT>=2.8",
"bcrypt>=4.0",
]
[project.urls]
@@ -40,13 +43,14 @@ Repository = "https://github.com/turnstonelabs/turnstone"
Issues = "https://github.com/turnstonelabs/turnstone/issues"
[project.optional-dependencies]
test = ["pytest>=9.0", "pytest-cov>=6.0"]
test = ["pytest>=9.0", "pytest-cov>=6.0", "croniter>=3.0"]
dev = ["ruff>=0.9", "mypy>=1.14", "types-redis>=4.6"]
mq = ["redis>=7.2"]
console = ["redis>=7.2"]
console = ["redis>=7.2", "croniter>=3.0"]
sim = ["redis>=7.2"]
anthropic = ["anthropic>=0.39"]
postgres = ["psycopg[binary]>=3.2"]
discord = ["discord.py>=2.4", "redis>=7.2"]
[project.scripts]
@@ -56,6 +60,9 @@ turnstone-server = "turnstone.server:main"
turnstone-bridge = "turnstone.mq.bridge:main"
turnstone-console = "turnstone.console.server:main"
turnstone-sim = "turnstone.sim.cli:main"
turnstone-admin = "turnstone.admin:main"
turnstone-channel = "turnstone.channels.cli:main"
turnstone-bootstrap = "turnstone.bootstrap:main"
[tool.hatch.build.targets.wheel]
include = [
@@ -128,10 +135,32 @@ ignore_missing_imports = true
module = ["sqlalchemy", "sqlalchemy.*", "alembic", "alembic.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["structlog", "structlog.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["jwt", "jwt.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["anthropic", "anthropic.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["discord", "discord.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["croniter", "croniter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["turnstone.channels.discord.*"]
disallow_subclassing_any = false
disallow_untyped_decorators = false
warn_unused_ignores = false
[[tool.mypy.overrides]]
module = "tests.*"
disallow_untyped_defs = false
+29 -78
View File
@@ -10,9 +10,7 @@
"get": {
"summary": "Cluster state summary",
"operationId": "v1_api_cluster_overview_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"responses": {
"200": {
"description": "Success",
@@ -31,9 +29,7 @@
"get": {
"summary": "Paginated node list",
"operationId": "v1_api_cluster_nodes_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"parameters": [
{
"name": "sort",
@@ -42,11 +38,7 @@
"schema": {
"type": "string",
"default": "activity",
"enum": [
"activity",
"tokens",
"name"
]
"enum": ["activity", "tokens", "name"]
},
"description": "Sort field"
},
@@ -89,9 +81,7 @@
"get": {
"summary": "Filtered workstream list",
"operationId": "v1_api_cluster_workstreams_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"parameters": [
{
"name": "state",
@@ -99,13 +89,7 @@
"required": false,
"schema": {
"type": "string",
"enum": [
"running",
"thinking",
"attention",
"idle",
"error"
]
"enum": ["running", "thinking", "attention", "idle", "error"]
},
"description": "Filter by state"
},
@@ -134,11 +118,7 @@
"schema": {
"type": "string",
"default": "state",
"enum": [
"state",
"tokens",
"name"
]
"enum": ["state", "tokens", "name"]
},
"description": "Sort field"
},
@@ -181,9 +161,7 @@
"get": {
"summary": "Single node detail",
"operationId": "v1_api_cluster_node_{node_id}_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"parameters": [
{
"name": "node_id",
@@ -222,9 +200,7 @@
"post": {
"summary": "Create workstream via MQ dispatch",
"operationId": "v1_api_cluster_workstreams_new_post",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"requestBody": {
"required": true,
"content": {
@@ -283,9 +259,7 @@
"get": {
"summary": "Cluster SSE event stream",
"operationId": "v1_api_cluster_events_get",
"tags": [
"Streaming"
],
"tags": ["Streaming"],
"description": "Server-Sent Events stream for real-time cluster updates. Returns text/event-stream with node_joined, node_lost, cluster_state, ws_created, ws_closed, ws_rename events.",
"responses": {
"200": {
@@ -298,9 +272,7 @@
"post": {
"summary": "Authenticate with a token",
"operationId": "v1_api_auth_login_post",
"tags": [
"Auth"
],
"tags": ["Auth"],
"requestBody": {
"required": true,
"content": {
@@ -339,9 +311,7 @@
"post": {
"summary": "Clear auth cookie",
"operationId": "v1_api_auth_logout_post",
"tags": [
"Auth"
],
"tags": ["Auth"],
"responses": {
"200": {
"description": "Success",
@@ -360,9 +330,7 @@
"get": {
"summary": "Console health check",
"operationId": "health_get",
"tags": [
"Observability"
],
"tags": ["Observability"],
"responses": {
"200": {
"description": "Success",
@@ -389,9 +357,7 @@
"type": "string"
}
},
"required": [
"error"
],
"required": ["error"],
"title": "ErrorResponse",
"type": "object"
},
@@ -400,9 +366,7 @@
"properties": {
"status": {
"default": "ok",
"examples": [
"ok"
],
"examples": ["ok"],
"title": "Status",
"type": "string"
}
@@ -419,9 +383,7 @@
"type": "string"
}
},
"required": [
"token"
],
"required": ["token"],
"title": "AuthLoginRequest",
"type": "object"
},
@@ -435,17 +397,12 @@
},
"role": {
"description": "Assigned role",
"examples": [
"full",
"read"
],
"examples": ["full", "read"],
"title": "Role",
"type": "string"
}
},
"required": [
"role"
],
"required": ["role"],
"title": "AuthLoginResponse",
"type": "object"
},
@@ -557,9 +514,7 @@
"type": "integer"
}
},
"required": [
"nodes"
],
"required": ["nodes"],
"title": "ClusterNodesResponse",
"type": "object"
},
@@ -632,9 +587,7 @@
"type": "string"
}
},
"required": [
"node_id"
],
"required": ["node_id"],
"title": "ClusterNodeInfo",
"type": "object"
},
@@ -668,9 +621,7 @@
"type": "integer"
}
},
"required": [
"workstreams"
],
"required": ["workstreams"],
"title": "ClusterWorkstreamsResponse",
"type": "object"
},
@@ -726,9 +677,7 @@
"type": "integer"
}
},
"required": [
"id"
],
"required": ["id"],
"title": "ClusterWorkstreamInfo",
"type": "object"
},
@@ -771,9 +720,7 @@
"type": "boolean"
}
},
"required": [
"node_id"
],
"required": ["node_id"],
"title": "NodeDetailResponse",
"type": "object"
},
@@ -802,6 +749,12 @@
"description": "Optional first message sent after creation",
"title": "Initial Message",
"type": "string"
},
"template": {
"default": "",
"description": "Prompt template name (replaces default templates)",
"title": "Template",
"type": "string"
}
},
"title": "ConsoleCreateWsRequest",
@@ -832,9 +785,7 @@
"properties": {
"status": {
"default": "ok",
"examples": [
"ok"
],
"examples": ["ok"],
"title": "Status",
"type": "string"
},
+361 -175
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "0.3.0",
"version": "0.4.2",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -10,9 +10,7 @@
"get": {
"summary": "List active workstreams",
"operationId": "v1_api_workstreams_get",
"tags": [
"Workstreams"
],
"tags": ["Workstreams"],
"responses": {
"200": {
"description": "Success",
@@ -31,9 +29,7 @@
"get": {
"summary": "Dashboard with workstream details and aggregates",
"operationId": "v1_api_dashboard_get",
"tags": [
"Workstreams"
],
"tags": ["Workstreams"],
"responses": {
"200": {
"description": "Success",
@@ -52,9 +48,7 @@
"post": {
"summary": "Create a new workstream",
"operationId": "v1_api_workstreams_new_post",
"tags": [
"Workstreams"
],
"tags": ["Workstreams"],
"requestBody": {
"required": true,
"content": {
@@ -93,9 +87,7 @@
"post": {
"summary": "Close a workstream",
"operationId": "v1_api_workstreams_close_post",
"tags": [
"Workstreams"
],
"tags": ["Workstreams"],
"requestBody": {
"required": true,
"content": {
@@ -134,9 +126,7 @@
"post": {
"summary": "Send a user message",
"operationId": "v1_api_send_post",
"tags": [
"Chat"
],
"tags": ["Chat"],
"requestBody": {
"required": true,
"content": {
@@ -185,9 +175,7 @@
"post": {
"summary": "Approve or deny a tool call",
"operationId": "v1_api_approve_post",
"tags": [
"Chat"
],
"tags": ["Chat"],
"requestBody": {
"required": true,
"content": {
@@ -226,9 +214,7 @@
"post": {
"summary": "Respond to a plan review",
"operationId": "v1_api_plan_post",
"tags": [
"Chat"
],
"tags": ["Chat"],
"requestBody": {
"required": true,
"content": {
@@ -267,9 +253,7 @@
"post": {
"summary": "Execute a slash command",
"operationId": "v1_api_command_post",
"tags": [
"Chat"
],
"tags": ["Chat"],
"requestBody": {
"required": true,
"content": {
@@ -314,13 +298,60 @@
}
}
},
"/v1/api/cancel": {
"post": {
"summary": "Cancel the active generation in a workstream",
"operationId": "v1_api_cancel_post",
"tags": ["Chat"],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CancelRequest"
}
}
}
},
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/StatusResponse"
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/events": {
"get": {
"summary": "Per-workstream SSE event stream",
"operationId": "v1_api_events_get",
"tags": [
"Streaming"
],
"tags": ["Streaming"],
"description": "Opens a Server-Sent Events stream scoped to a single workstream. Returns text/event-stream. See API reference for event types.",
"parameters": [
{
@@ -354,9 +385,7 @@
"get": {
"summary": "Global SSE event stream",
"operationId": "v1_api_events_global_get",
"tags": [
"Streaming"
],
"tags": ["Streaming"],
"description": "Global Server-Sent Events stream for state-change broadcasts across all workstreams. Returns text/event-stream.",
"responses": {
"200": {
@@ -365,20 +394,18 @@
}
}
},
"/v1/api/sessions": {
"/v1/api/workstreams/saved": {
"get": {
"summary": "List saved sessions",
"operationId": "v1_api_sessions_get",
"tags": [
"Sessions"
],
"summary": "List saved workstreams",
"operationId": "v1_api_workstreams_saved_get",
"tags": ["Workstreams"],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ListSessionsResponse"
"$ref": "#/components/schemas/ListSavedWorkstreamsResponse"
}
}
}
@@ -390,9 +417,7 @@
"post": {
"summary": "Authenticate with a token",
"operationId": "v1_api_auth_login_post",
"tags": [
"Auth"
],
"tags": ["Auth"],
"requestBody": {
"required": true,
"content": {
@@ -427,13 +452,89 @@
}
}
},
"/v1/api/auth/setup": {
"post": {
"summary": "Create first admin user",
"operationId": "v1_api_auth_setup_post",
"tags": ["Auth"],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthSetupRequest"
}
}
}
},
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthSetupResponse"
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/auth/status": {
"get": {
"summary": "Return auth state",
"operationId": "v1_api_auth_status_get",
"tags": ["Auth"],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthStatusResponse"
}
}
}
}
}
}
},
"/v1/api/auth/logout": {
"post": {
"summary": "Clear auth cookie",
"operationId": "v1_api_auth_logout_post",
"tags": [
"Auth"
],
"tags": ["Auth"],
"responses": {
"200": {
"description": "Success",
@@ -452,9 +553,7 @@
"get": {
"summary": "Server health check",
"operationId": "health_get",
"tags": [
"Observability"
],
"tags": ["Observability"],
"responses": {
"200": {
"description": "Success",
@@ -481,9 +580,7 @@
"type": "string"
}
},
"required": [
"error"
],
"required": ["error"],
"title": "ErrorResponse",
"type": "object"
},
@@ -492,9 +589,7 @@
"properties": {
"status": {
"default": "ok",
"examples": [
"ok"
],
"examples": ["ok"],
"title": "Status",
"type": "string"
}
@@ -503,17 +598,27 @@
"type": "object"
},
"AuthLoginRequest": {
"description": "POST /v1/api/auth/login request body.",
"description": "POST /v1/api/auth/login request body.\n\nEither username+password or token must be provided.",
"properties": {
"username": {
"default": "",
"description": "Login username",
"title": "Username",
"type": "string"
},
"password": {
"default": "",
"description": "Login password",
"title": "Password",
"type": "string"
},
"token": {
"description": "Bearer token to authenticate",
"default": "",
"description": "Legacy: bearer token to authenticate",
"title": "Token",
"type": "string"
}
},
"required": [
"token"
],
"title": "AuthLoginRequest",
"type": "object"
},
@@ -525,22 +630,116 @@
"title": "Status",
"type": "string"
},
"user_id": {
"default": "",
"description": "Authenticated user ID",
"title": "User Id",
"type": "string"
},
"role": {
"description": "Assigned role",
"examples": [
"full",
"read"
],
"description": "Legacy role",
"examples": ["full", "read"],
"title": "Role",
"type": "string"
},
"scopes": {
"default": "",
"description": "Comma-separated scopes",
"examples": ["read,write,approve"],
"title": "Scopes",
"type": "string"
},
"jwt": {
"default": "",
"description": "JWT session token (if JWT auth is configured)",
"title": "Jwt",
"type": "string"
}
},
"required": [
"role"
],
"required": ["role"],
"title": "AuthLoginResponse",
"type": "object"
},
"AuthSetupRequest": {
"description": "POST /v1/api/auth/setup request body.",
"properties": {
"username": {
"description": "Login username (1-64 ASCII characters)",
"title": "Username",
"type": "string"
},
"display_name": {
"description": "Display name",
"title": "Display Name",
"type": "string"
},
"password": {
"description": "Password (minimum 8 characters)",
"title": "Password",
"type": "string"
}
},
"required": ["username", "display_name", "password"],
"title": "AuthSetupRequest",
"type": "object"
},
"AuthSetupResponse": {
"description": "POST /v1/api/auth/setup success response.",
"properties": {
"status": {
"default": "ok",
"title": "Status",
"type": "string"
},
"user_id": {
"title": "User Id",
"type": "string"
},
"username": {
"title": "Username",
"type": "string"
},
"role": {
"default": "full",
"title": "Role",
"type": "string"
},
"scopes": {
"default": "approve,read,write",
"title": "Scopes",
"type": "string"
},
"jwt": {
"default": "",
"description": "JWT session token",
"title": "Jwt",
"type": "string"
}
},
"required": ["user_id", "username"],
"title": "AuthSetupResponse",
"type": "object"
},
"AuthStatusResponse": {
"description": "GET /v1/api/auth/status response.",
"properties": {
"auth_enabled": {
"title": "Auth Enabled",
"type": "boolean"
},
"has_users": {
"title": "Has Users",
"type": "boolean"
},
"setup_required": {
"title": "Setup Required",
"type": "boolean"
}
},
"required": ["auth_enabled", "has_users", "setup_required"],
"title": "AuthStatusResponse",
"type": "object"
},
"SendRequest": {
"properties": {
"message": {
@@ -554,10 +753,7 @@
"type": "string"
}
},
"required": [
"message",
"ws_id"
],
"required": ["message", "ws_id"],
"title": "SendRequest",
"type": "object"
},
@@ -565,17 +761,12 @@
"properties": {
"status": {
"description": "'ok' or 'busy'",
"examples": [
"ok",
"busy"
],
"examples": ["ok", "busy"],
"title": "Status",
"type": "string"
}
},
"required": [
"status"
],
"required": ["status"],
"title": "SendResponse",
"type": "object"
},
@@ -611,10 +802,7 @@
"type": "string"
}
},
"required": [
"approved",
"ws_id"
],
"required": ["approved", "ws_id"],
"title": "ApproveRequest",
"type": "object"
},
@@ -631,10 +819,7 @@
"type": "string"
}
},
"required": [
"feedback",
"ws_id"
],
"required": ["feedback", "ws_id"],
"title": "PlanFeedbackRequest",
"type": "object"
},
@@ -651,13 +836,22 @@
"type": "string"
}
},
"required": [
"command",
"ws_id"
],
"required": ["command", "ws_id"],
"title": "CommandRequest",
"type": "object"
},
"CancelRequest": {
"properties": {
"ws_id": {
"description": "Target workstream ID",
"title": "Ws Id",
"type": "string"
}
},
"required": ["ws_id"],
"title": "CancelRequest",
"type": "object"
},
"CreateWorkstreamRequest": {
"properties": {
"name": {
@@ -677,6 +871,18 @@
"description": "Auto-approve all tool calls",
"title": "Auto Approve",
"type": "boolean"
},
"resume_ws": {
"default": "",
"description": "Workstream ID to resume atomically during creation (empty = fresh start)",
"title": "Resume Ws",
"type": "string"
},
"template": {
"default": "",
"description": "Prompt template name (replaces default templates)",
"title": "Template",
"type": "string"
}
},
"title": "CreateWorkstreamRequest",
@@ -693,12 +899,21 @@
"description": "Assigned workstream name",
"title": "Name",
"type": "string"
},
"resumed": {
"default": false,
"description": "Whether a previous workstream was resumed",
"title": "Resumed",
"type": "boolean"
},
"message_count": {
"default": 0,
"description": "Number of messages in the resumed workstream",
"title": "Message Count",
"type": "integer"
}
},
"required": [
"ws_id",
"name"
],
"required": ["ws_id", "name"],
"title": "CreateWorkstreamResponse",
"type": "object"
},
@@ -710,9 +925,7 @@
"type": "string"
}
},
"required": [
"ws_id"
],
"required": ["ws_id"],
"title": "CloseWorkstreamRequest",
"type": "object"
},
@@ -726,9 +939,7 @@
"type": "array"
}
},
"required": [
"workstreams"
],
"required": ["workstreams"],
"title": "ListWorkstreamsResponse",
"type": "object"
},
@@ -745,25 +956,9 @@
"state": {
"title": "State",
"type": "string"
},
"session_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Session Id"
}
},
"required": [
"id",
"name",
"state"
],
"required": ["id", "name", "state"],
"title": "WorkstreamInfo",
"type": "object"
},
@@ -780,10 +975,7 @@
"$ref": "#/components/schemas/DashboardAggregate"
}
},
"required": [
"workstreams",
"aggregate"
],
"required": ["workstreams", "aggregate"],
"title": "DashboardResponse",
"type": "object"
},
@@ -837,18 +1029,6 @@
"title": "State",
"type": "string"
},
"session_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Session Id"
},
"title": {
"default": "",
"title": "Title",
@@ -895,34 +1075,28 @@
"type": "string"
}
},
"required": [
"id",
"name",
"state"
],
"required": ["id", "name", "state"],
"title": "DashboardWorkstream",
"type": "object"
},
"ListSessionsResponse": {
"ListSavedWorkstreamsResponse": {
"properties": {
"sessions": {
"workstreams": {
"items": {
"$ref": "#/components/schemas/SessionInfo"
"$ref": "#/components/schemas/SavedWorkstreamInfo"
},
"title": "Sessions",
"title": "Workstreams",
"type": "array"
}
},
"required": [
"sessions"
],
"title": "ListSessionsResponse",
"required": ["workstreams"],
"title": "ListSavedWorkstreamsResponse",
"type": "object"
},
"SessionInfo": {
"SavedWorkstreamInfo": {
"properties": {
"session_id": {
"title": "Session Id",
"ws_id": {
"title": "Ws Id",
"type": "string"
},
"alias": {
@@ -962,22 +1136,14 @@
"type": "integer"
}
},
"required": [
"session_id",
"created",
"updated",
"message_count"
],
"title": "SessionInfo",
"required": ["ws_id", "created", "updated", "message_count"],
"title": "SavedWorkstreamInfo",
"type": "object"
},
"HealthResponse": {
"properties": {
"status": {
"examples": [
"ok",
"degraded"
],
"examples": ["ok", "degraded"],
"title": "Status",
"type": "string"
},
@@ -1017,38 +1183,58 @@
}
],
"default": null
},
"mcp": {
"anyOf": [
{
"$ref": "#/components/schemas/McpStatus"
},
{
"type": "null"
}
],
"default": null
}
},
"required": [
"status"
],
"required": ["status"],
"title": "HealthResponse",
"type": "object"
},
"McpStatus": {
"properties": {
"servers": {
"default": 0,
"title": "Servers",
"type": "integer"
},
"resources": {
"default": 0,
"title": "Resources",
"type": "integer"
},
"prompts": {
"default": 0,
"title": "Prompts",
"type": "integer"
}
},
"title": "McpStatus",
"type": "object"
},
"BackendStatus": {
"properties": {
"status": {
"examples": [
"up",
"down"
],
"examples": ["up", "down"],
"title": "Status",
"type": "string"
},
"circuit_state": {
"examples": [
"closed",
"open",
"half_open"
],
"examples": ["closed", "open", "half_open"],
"title": "Circuit State",
"type": "string"
}
},
"required": [
"status",
"circuit_state"
],
"required": ["status", "circuit_state"],
"title": "BackendStatus",
"type": "object"
},
+212 -3
View File
@@ -1,16 +1,40 @@
import { BaseClient, type ClientOptions } from "./base.js";
import type { ClusterEvent } from "./events.js";
import type {
AuditQueryOptions,
AuditResponse,
AuthLoginResponse,
AuthSetupResponse,
AuthStatusResponse,
ClusterNodesResponse,
ClusterOverviewResponse,
ClusterSnapshotResponse,
ClusterWorkstreamsResponse,
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreatePolicyOptions,
CreateRoleOptions,
CreateScheduleRequest,
CreateTemplateOptions,
ListScheduleRunsResponse,
ListSchedulesResponse,
NodeDetailResponse,
NodesOptions,
OrgInfo,
PromptTemplateInfo,
RoleInfo,
ScheduleInfo,
StatusResponse,
ToolPolicyInfo,
UpdateOrgOptions,
UpdatePolicyOptions,
UpdateRoleOptions,
UpdateScheduleRequest,
UpdateTemplateOptions,
UsageQueryOptions,
UsageResponse,
UserRoleInfo,
WorkstreamsOptions,
} from "./types.js";
@@ -26,6 +50,10 @@ export class TurnstoneConsole extends BaseClient {
return this.request("GET", "/v1/api/cluster/overview");
}
async snapshot(): Promise<ClusterSnapshotResponse> {
return this.request("GET", "/v1/api/cluster/snapshot");
}
async nodes(opts?: NodesOptions): Promise<ClusterNodesResponse> {
return this.request("GET", "/v1/api/cluster/nodes", {
params: {
@@ -70,9 +98,33 @@ export class TurnstoneConsole extends BaseClient {
// -- Auth -----------------------------------------------------------------
async login(token: string): Promise<AuthLoginResponse> {
return this.request("POST", "/v1/api/auth/login", {
json: { token },
async login(opts: {
token?: string;
username?: string;
password?: string;
}): Promise<AuthLoginResponse> {
const body =
opts.username && opts.password
? { username: opts.username, password: opts.password }
: { token: opts.token ?? "" };
return this.request("POST", "/v1/api/auth/login", { json: body });
}
async authStatus(): Promise<AuthStatusResponse> {
return this.request("GET", "/v1/api/auth/status");
}
async setup(opts: {
username: string;
displayName: string;
password: string;
}): Promise<AuthSetupResponse> {
return this.request("POST", "/v1/api/auth/setup", {
json: {
username: opts.username,
display_name: opts.displayName,
password: opts.password,
},
});
}
@@ -85,4 +137,161 @@ export class TurnstoneConsole extends BaseClient {
async health(): Promise<ConsoleHealthResponse> {
return this.request("GET", "/health");
}
// -- Schedules ------------------------------------------------------------
async listSchedules(): Promise<ListSchedulesResponse> {
return this.request("GET", "/v1/api/admin/schedules");
}
async createSchedule(opts: CreateScheduleRequest): Promise<ScheduleInfo> {
return this.request("POST", "/v1/api/admin/schedules", { json: opts });
}
async getSchedule(taskId: string): Promise<ScheduleInfo> {
return this.request("GET", `/v1/api/admin/schedules/${taskId}`);
}
async updateSchedule(
taskId: string,
opts: UpdateScheduleRequest,
): Promise<ScheduleInfo> {
return this.request("PUT", `/v1/api/admin/schedules/${taskId}`, {
json: opts,
});
}
async deleteSchedule(taskId: string): Promise<StatusResponse> {
return this.request("DELETE", `/v1/api/admin/schedules/${taskId}`);
}
async listScheduleRuns(
taskId: string,
opts?: { limit?: number },
): Promise<ListScheduleRunsResponse> {
return this.request("GET", `/v1/api/admin/schedules/${taskId}/runs`, {
params: { limit: opts?.limit ?? 50 },
});
}
// -- Governance: Roles ------------------------------------------------------
async listRoles(): Promise<{ roles: RoleInfo[] }> {
return this.request("GET", "/v1/api/admin/roles");
}
async createRole(opts: CreateRoleOptions): Promise<RoleInfo> {
return this.request("POST", "/v1/api/admin/roles", { json: opts });
}
async updateRole(roleId: string, opts: UpdateRoleOptions): Promise<RoleInfo> {
return this.request("PUT", `/v1/api/admin/roles/${roleId}`, {
json: opts,
});
}
async deleteRole(roleId: string): Promise<StatusResponse> {
return this.request("DELETE", `/v1/api/admin/roles/${roleId}`);
}
async listUserRoles(userId: string): Promise<{ roles: UserRoleInfo[] }> {
return this.request("GET", `/v1/api/admin/users/${userId}/roles`);
}
async assignRole(userId: string, roleId: string): Promise<StatusResponse> {
return this.request("POST", `/v1/api/admin/users/${userId}/roles`, {
json: { role_id: roleId },
});
}
async unassignRole(userId: string, roleId: string): Promise<StatusResponse> {
return this.request(
"DELETE",
`/v1/api/admin/users/${userId}/roles/${roleId}`,
);
}
// -- Governance: Organizations ----------------------------------------------
async listOrgs(): Promise<{ orgs: OrgInfo[] }> {
return this.request("GET", "/v1/api/admin/orgs");
}
async getOrg(orgId: string): Promise<OrgInfo> {
return this.request("GET", `/v1/api/admin/orgs/${orgId}`);
}
async updateOrg(orgId: string, opts: UpdateOrgOptions): Promise<OrgInfo> {
return this.request("PUT", `/v1/api/admin/orgs/${orgId}`, { json: opts });
}
// -- Governance: Tool Policies ----------------------------------------------
async listPolicies(): Promise<{ policies: ToolPolicyInfo[] }> {
return this.request("GET", "/v1/api/admin/policies");
}
async createPolicy(opts: CreatePolicyOptions): Promise<ToolPolicyInfo> {
return this.request("POST", "/v1/api/admin/policies", { json: opts });
}
async updatePolicy(
policyId: string,
opts: UpdatePolicyOptions,
): Promise<ToolPolicyInfo> {
return this.request("PUT", `/v1/api/admin/policies/${policyId}`, {
json: opts,
});
}
async deletePolicy(policyId: string): Promise<StatusResponse> {
return this.request("DELETE", `/v1/api/admin/policies/${policyId}`);
}
// -- Governance: Prompt Templates -------------------------------------------
async listTemplates(): Promise<{ templates: PromptTemplateInfo[] }> {
return this.request("GET", "/v1/api/admin/templates");
}
async createTemplate(
opts: CreateTemplateOptions,
): Promise<PromptTemplateInfo> {
return this.request("POST", "/v1/api/admin/templates", { json: opts });
}
async updateTemplate(
templateId: string,
opts: UpdateTemplateOptions,
): Promise<PromptTemplateInfo> {
return this.request("PUT", `/v1/api/admin/templates/${templateId}`, {
json: opts,
});
}
async deleteTemplate(templateId: string): Promise<StatusResponse> {
return this.request("DELETE", `/v1/api/admin/templates/${templateId}`);
}
// -- Governance: Usage & Audit ----------------------------------------------
async getUsage(opts: UsageQueryOptions): Promise<UsageResponse> {
const params: Record<string, string> = { since: opts.since };
if (opts.until) params.until = opts.until;
if (opts.user_id) params.user_id = opts.user_id;
if (opts.model) params.model = opts.model;
if (opts.group_by) params.group_by = opts.group_by;
return this.request("GET", "/v1/api/admin/usage", { params });
}
async getAudit(opts?: AuditQueryOptions): Promise<AuditResponse> {
const params: Record<string, string> = {};
if (opts?.action) params.action = opts.action;
if (opts?.user_id) params.user_id = opts.user_id;
if (opts?.since) params.since = opts.since;
if (opts?.until) params.until = opts.until;
if (opts?.limit !== undefined) params.limit = String(opts.limit);
if (opts?.offset !== undefined) params.offset = String(opts.offset);
return this.request("GET", "/v1/api/admin/audit", { params });
}
}
+33 -1
View File
@@ -1,3 +1,5 @@
import type { ClusterOverviewResponse, ClusterSnapshotNode } from "./types.js";
// ---------------------------------------------------------------------------
// Server SSE events
// ---------------------------------------------------------------------------
@@ -46,6 +48,12 @@ export interface ApproveRequestEvent {
items: Array<Record<string, unknown>>;
}
export interface ApprovalResolvedEvent {
type: "approval_resolved";
approved: boolean;
feedback: string;
}
export interface ToolResultEvent {
type: "tool_result";
call_id: string;
@@ -93,6 +101,10 @@ export interface ClearUiEvent {
type: "clear_ui";
}
export interface CancelledEvent {
type: "cancelled";
}
// Global events
export interface WsStateEvent {
@@ -135,6 +147,7 @@ export type ServerEvent =
| StreamEndEvent
| ToolInfoEvent
| ApproveRequestEvent
| ApprovalResolvedEvent
| ToolResultEvent
| ToolOutputChunkEvent
| StatusEvent
@@ -143,6 +156,7 @@ export type ServerEvent =
| ErrorEvent
| BusyErrorEvent
| ClearUiEvent
| CancelledEvent
| WsStateEvent
| WsActivityEvent
| WsRenameEvent
@@ -191,6 +205,13 @@ export interface ClusterWsRenameEvent {
name: string;
}
export interface ClusterSnapshotEvent {
type: "snapshot";
nodes: ClusterSnapshotNode[];
overview: ClusterOverviewResponse;
timestamp: number;
}
/** Discriminated union of all console cluster SSE event types. */
export type ClusterEvent =
| NodeJoinedEvent
@@ -198,7 +219,8 @@ export type ClusterEvent =
| ClusterStateEvent
| ClusterWsCreatedEvent
| ClusterWsClosedEvent
| ClusterWsRenameEvent;
| ClusterWsRenameEvent
| ClusterSnapshotEvent;
// ---------------------------------------------------------------------------
// Type guards
@@ -234,6 +256,16 @@ export function isApproveRequestEvent(
return e.type === "approve_request";
}
export function isApprovalResolvedEvent(
e: ServerEvent,
): e is ApprovalResolvedEvent {
return e.type === "approval_resolved";
}
export function isPlanReviewEvent(e: ServerEvent): e is PlanReviewEvent {
return e.type === "plan_review";
}
export function isCancelledEvent(e: ServerEvent): e is CancelledEvent {
return e.type === "cancelled";
}
+36 -2
View File
@@ -37,6 +37,7 @@ export type {
StreamEndEvent,
ToolInfoEvent,
ApproveRequestEvent,
ApprovalResolvedEvent,
ToolResultEvent,
ToolOutputChunkEvent,
StatusEvent,
@@ -45,6 +46,7 @@ export type {
ErrorEvent,
BusyErrorEvent,
ClearUiEvent,
CancelledEvent,
WsStateEvent,
WsActivityEvent,
WsRenameEvent,
@@ -55,6 +57,7 @@ export type {
ClusterWsCreatedEvent,
ClusterWsClosedEvent,
ClusterWsRenameEvent,
ClusterSnapshotEvent,
} from "./events.js";
export {
@@ -65,7 +68,9 @@ export {
isToolResultEvent,
isWsStateEvent,
isApproveRequestEvent,
isApprovalResolvedEvent,
isPlanReviewEvent,
isCancelledEvent,
} from "./events.js";
// Request/response types
@@ -83,24 +88,53 @@ export type {
DashboardWorkstream,
DashboardAggregate,
DashboardResponse,
SessionInfo,
ListSessionsResponse,
SavedWorkstreamInfo,
ListSavedWorkstreamsResponse,
BackendStatus,
McpStatus,
WorkstreamCounts,
HealthResponse,
AuthLoginRequest,
AuthLoginResponse,
AuthStatusResponse,
AuthSetupResponse,
StatusResponse,
ErrorResponse,
ClusterOverviewResponse,
ClusterNodeInfo,
ClusterNodesResponse,
ClusterSnapshotNode,
ClusterSnapshotResponse,
ClusterWorkstreamInfo,
ClusterWorkstreamsResponse,
NodeDetailResponse,
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateScheduleRequest,
UpdateScheduleRequest,
ScheduleInfo,
ScheduleRunInfo,
ListSchedulesResponse,
ListScheduleRunsResponse,
RoleInfo,
CreateRoleOptions,
UpdateRoleOptions,
UserRoleInfo,
OrgInfo,
UpdateOrgOptions,
ToolPolicyInfo,
CreatePolicyOptions,
UpdatePolicyOptions,
PromptTemplateInfo,
CreateTemplateOptions,
UpdateTemplateOptions,
UsageBreakdownItem,
UsageResponse,
UsageQueryOptions,
AuditEventInfo,
AuditQueryOptions,
AuditResponse,
TurnResult,
SendAndWaitOptions,
NodesOptions,
+39 -7
View File
@@ -2,11 +2,13 @@ import { BaseClient, type ClientOptions } from "./base.js";
import type { ServerEvent } from "./events.js";
import type {
AuthLoginResponse,
AuthSetupResponse,
AuthStatusResponse,
CreateWorkstreamRequest,
CreateWorkstreamResponse,
DashboardResponse,
HealthResponse,
ListSessionsResponse,
ListSavedWorkstreamsResponse,
ListWorkstreamsResponse,
SendAndWaitOptions,
SendResponse,
@@ -84,6 +86,12 @@ export class TurnstoneServer extends BaseClient {
});
}
async cancel(wsId: string): Promise<StatusResponse> {
return this.request("POST", "/v1/api/cancel", {
json: { ws_id: wsId },
});
}
// -- Streaming ------------------------------------------------------------
async *streamEvents(wsId: string): AsyncIterableIterator<ServerEvent> {
@@ -176,17 +184,41 @@ export class TurnstoneServer extends BaseClient {
return result;
}
// -- Sessions -------------------------------------------------------------
// -- Saved workstreams ----------------------------------------------------
async listSessions(): Promise<ListSessionsResponse> {
return this.request("GET", "/v1/api/sessions");
async listSavedWorkstreams(): Promise<ListSavedWorkstreamsResponse> {
return this.request("GET", "/v1/api/workstreams/saved");
}
// -- Auth -----------------------------------------------------------------
async login(token: string): Promise<AuthLoginResponse> {
return this.request("POST", "/v1/api/auth/login", {
json: { token },
async login(opts: {
token?: string;
username?: string;
password?: string;
}): Promise<AuthLoginResponse> {
const body =
opts.username && opts.password
? { username: opts.username, password: opts.password }
: { token: opts.token ?? "" };
return this.request("POST", "/v1/api/auth/login", { json: body });
}
async authStatus(): Promise<AuthStatusResponse> {
return this.request("GET", "/v1/api/auth/status");
}
async setup(opts: {
username: string;
displayName: string;
password: string;
}): Promise<AuthSetupResponse> {
return this.request("POST", "/v1/api/auth/setup", {
json: {
username: opts.username,
display_name: opts.displayName,
password: opts.password,
},
});
}
+293 -7
View File
@@ -17,6 +17,24 @@ export interface AuthLoginRequest {
export interface AuthLoginResponse {
status: string;
role: string;
scopes?: string;
jwt?: string;
user_id?: string;
}
export interface AuthStatusResponse {
auth_enabled: boolean;
has_users: boolean;
setup_required: boolean;
}
export interface AuthSetupResponse {
status: string;
user_id: string;
username: string;
role: string;
scopes: string;
jwt?: string;
}
// ---------------------------------------------------------------------------
@@ -53,11 +71,15 @@ export interface CreateWorkstreamRequest {
name?: string;
model?: string;
auto_approve?: boolean;
resume_ws?: string;
template?: string;
}
export interface CreateWorkstreamResponse {
ws_id: string;
name: string;
resumed?: boolean;
message_count?: number;
}
export interface CloseWorkstreamRequest {
@@ -68,7 +90,6 @@ export interface WorkstreamInfo {
id: string;
name: string;
state: string;
session_id?: string | null;
}
export interface ListWorkstreamsResponse {
@@ -79,7 +100,6 @@ export interface DashboardWorkstream {
id: string;
name: string;
state: string;
session_id?: string | null;
title?: string;
tokens?: number;
context_ratio?: number;
@@ -106,11 +126,11 @@ export interface DashboardResponse {
}
// ---------------------------------------------------------------------------
// Server API — Sessions
// Server API — Saved workstreams
// ---------------------------------------------------------------------------
export interface SessionInfo {
session_id: string;
export interface SavedWorkstreamInfo {
ws_id: string;
alias?: string | null;
title?: string | null;
created: string;
@@ -118,8 +138,8 @@ export interface SessionInfo {
message_count: number;
}
export interface ListSessionsResponse {
sessions: SessionInfo[];
export interface ListSavedWorkstreamsResponse {
workstreams: SavedWorkstreamInfo[];
}
// ---------------------------------------------------------------------------
@@ -140,6 +160,12 @@ export interface WorkstreamCounts {
error?: number;
}
export interface McpStatus {
servers: number;
resources: number;
prompts: number;
}
export interface HealthResponse {
status: string;
version?: string;
@@ -147,6 +173,7 @@ export interface HealthResponse {
model?: string;
workstreams?: WorkstreamCounts;
backend?: BackendStatus | null;
mcp?: McpStatus | null;
}
// ---------------------------------------------------------------------------
@@ -225,11 +252,29 @@ export interface NodeDetailResponse {
aggregate: ClusterAggregate;
}
export interface ClusterSnapshotNode {
node_id: string;
server_url: string;
max_ws: number;
reachable: boolean;
version: string;
health: Record<string, string>;
aggregate: Record<string, number>;
workstreams: ClusterWorkstreamInfo[];
}
export interface ClusterSnapshotResponse {
nodes: ClusterSnapshotNode[];
overview: ClusterOverviewResponse;
timestamp: number;
}
export interface ConsoleCreateWsRequest {
node_id?: string;
name?: string;
model?: string;
initial_message?: string;
template?: string;
}
export interface ConsoleCreateWsResponse {
@@ -247,6 +292,247 @@ export interface ConsoleHealthResponse {
versions: string[];
}
// ---------------------------------------------------------------------------
// Console API — Schedules
// ---------------------------------------------------------------------------
export interface CreateScheduleRequest {
name: string;
schedule_type: string;
initial_message: string;
description?: string;
cron_expr?: string;
at_time?: string;
target_mode?: string;
model?: string;
auto_approve?: boolean;
auto_approve_tools?: string[];
enabled?: boolean;
}
export interface UpdateScheduleRequest {
name?: string;
description?: string;
schedule_type?: string;
cron_expr?: string;
at_time?: string;
target_mode?: string;
model?: string;
initial_message?: string;
auto_approve?: boolean;
auto_approve_tools?: string[];
enabled?: boolean;
}
export interface ScheduleInfo {
task_id: string;
name: string;
description: string;
schedule_type: string;
cron_expr: string;
at_time: string;
target_mode: string;
model: string;
initial_message: string;
auto_approve: boolean;
auto_approve_tools: string[];
enabled: boolean;
created_by: string;
last_run: string | null;
next_run: string | null;
created: string;
updated: string;
}
export interface ListSchedulesResponse {
schedules: ScheduleInfo[];
}
export interface ScheduleRunInfo {
run_id: string;
task_id: string;
node_id: string;
ws_id: string;
correlation_id: string;
started: string;
status: string;
error: string;
}
export interface ListScheduleRunsResponse {
runs: ScheduleRunInfo[];
}
// ---------------------------------------------------------------------------
// Console API — Governance: Roles
// ---------------------------------------------------------------------------
export interface RoleInfo {
role_id: string;
name: string;
display_name: string;
permissions: string;
builtin: boolean;
org_id: string;
created: string;
updated: string;
}
export interface CreateRoleOptions {
name: string;
display_name?: string;
permissions?: string;
}
export interface UpdateRoleOptions {
display_name?: string;
permissions?: string;
}
export interface UserRoleInfo extends RoleInfo {
assigned_by: string;
assignment_created: string;
}
// ---------------------------------------------------------------------------
// Console API — Governance: Orgs
// ---------------------------------------------------------------------------
export interface OrgInfo {
org_id: string;
name: string;
display_name: string;
settings: string;
created: string;
updated: string;
}
export interface UpdateOrgOptions {
display_name?: string;
settings?: string;
}
// ---------------------------------------------------------------------------
// Console API — Governance: Tool Policies
// ---------------------------------------------------------------------------
export interface ToolPolicyInfo {
policy_id: string;
name: string;
tool_pattern: string;
action: string;
priority: number;
org_id: string;
enabled: boolean;
created_by: string;
created: string;
updated: string;
}
export interface CreatePolicyOptions {
name: string;
tool_pattern: string;
action: string;
priority?: number;
org_id?: string;
enabled?: boolean;
}
export interface UpdatePolicyOptions {
name?: string;
tool_pattern?: string;
action?: string;
priority?: number;
enabled?: boolean;
}
// ---------------------------------------------------------------------------
// Console API — Governance: Prompt Templates
// ---------------------------------------------------------------------------
export interface PromptTemplateInfo {
template_id: string;
name: string;
category: string;
content: string;
variables: string;
is_default: boolean;
org_id: string;
created_by: string;
created: string;
updated: string;
origin: string;
mcp_server: string;
readonly: boolean;
}
export interface CreateTemplateOptions {
name: string;
content: string;
category?: string;
variables?: string;
is_default?: boolean;
org_id?: string;
}
export interface UpdateTemplateOptions {
name?: string;
content?: string;
category?: string;
variables?: string;
is_default?: boolean;
}
// ---------------------------------------------------------------------------
// Console API — Governance: Usage & Audit
// ---------------------------------------------------------------------------
export interface UsageBreakdownItem {
key?: string;
prompt_tokens: number;
completion_tokens: number;
tool_calls_count: number;
}
export interface UsageResponse {
summary: UsageBreakdownItem[];
breakdown: UsageBreakdownItem[];
}
export interface UsageQueryOptions {
since: string;
until?: string;
user_id?: string;
model?: string;
group_by?: string;
}
export interface AuditEventInfo {
event_id: string;
timestamp: string;
user_id: string;
action: string;
resource_type: string;
resource_id: string;
detail: string;
ip_address: string;
created: string;
}
export interface AuditQueryOptions {
action?: string;
user_id?: string;
since?: string;
until?: string;
limit?: number;
offset?: number;
}
export interface AuditResponse {
events: AuditEventInfo[];
total: number;
}
// ---------------------------------------------------------------------------
// SDK-specific types
// ---------------------------------------------------------------------------
+10
View File
@@ -6,6 +6,7 @@ import {
isToolResultEvent,
isWsStateEvent,
isApproveRequestEvent,
isApprovalResolvedEvent,
isPlanReviewEvent,
isReasoningEvent,
} from "../src/events.js";
@@ -62,6 +63,15 @@ describe("event type guards", () => {
expect(isApproveRequestEvent(e)).toBe(true);
});
it("isApprovalResolvedEvent", () => {
const e: ServerEvent = {
type: "approval_resolved",
approved: false,
feedback: "Approval timed out",
};
expect(isApprovalResolvedEvent(e)).toBe(true);
});
it("isPlanReviewEvent", () => {
const e: ServerEvent = { type: "plan_review", content: "## Plan" };
expect(isPlanReviewEvent(e)).toBe(true);
+20 -2
View File
@@ -59,7 +59,7 @@
"user_prompt": "Change the default port from 8000 to 9000 in both server.py and config.py",
"setup": {
"files": {
"server.py": "from config import PORT\n\ndef run():\n print(f'Listening on port {PORT}')\n",
"server.py": "import socket\n\ndef run():\n sock = socket.socket()\n sock.bind(('localhost', 8000))\n print('Server running on port 8000')\n",
"config.py": "PORT = 8000\nHOST = 'localhost'\n"
}
},
@@ -126,7 +126,7 @@
"app.py": "import sqlite3\nfrom flask import Flask, jsonify\n\napp = Flask(__name__)\nDB = 'data.db'\n\ndef get_db():\n return sqlite3.connect(DB)\n\n@app.route('/users')\ndef list_users():\n db = get_db()\n users = db.execute('SELECT * FROM users').fetchall()\n db.close()\n return jsonify(users)\n\n@app.route('/users/<int:uid>')\ndef get_user(uid):\n db = get_db()\n user = db.execute('SELECT * FROM users WHERE id=?', (uid,)).fetchone()\n db.close()\n return jsonify(user)\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
}
},
"expected_actions": [{ "tool": "plan" }],
"expected_actions": [{ "tool": "create_plan" }],
"match_mode": "subset"
},
{
@@ -175,6 +175,24 @@
{ "tool": "man", "args_pattern": { "page": "tar" } }
],
"match_mode": "subset"
},
{
"id": "math-calculation",
"description": "Use the math tool for precise calculations, not bash or mental math",
"user_prompt": "What is 2^64 - 1? Use the math tool to calculate it precisely.",
"expected_actions": [
{ "tool": "math", "args_pattern": { "code": "2.*64" } }
],
"match_mode": "subset"
},
{
"id": "web-search-query",
"description": "Use web_search for general knowledge lookups, not web_fetch",
"user_prompt": "Search the web for the current population of Tokyo",
"expected_actions": [
{ "tool": "web_search", "args_pattern": { "query": "Tokyo" } }
],
"match_mode": "subset"
}
]
}
+181
View File
@@ -0,0 +1,181 @@
"""Tests for turnstone.mq.async_broker.AsyncRedisBroker."""
from __future__ import annotations
import asyncio
import contextlib
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from turnstone.mq.async_broker import AsyncRedisBroker
@pytest.fixture
def broker() -> AsyncRedisBroker:
return AsyncRedisBroker(host="localhost", port=6379, db=0, prefix="test", response_ttl=120)
@pytest.fixture
def mock_redis() -> AsyncMock:
"""Return a mock Redis client with common async methods."""
r = AsyncMock()
r.rpush = AsyncMock()
r.publish = AsyncMock()
r.expire = AsyncMock()
r.get = AsyncMock(return_value=None)
r.set = AsyncMock()
r.delete = AsyncMock()
r.blpop = AsyncMock(return_value=None)
ps = AsyncMock()
ps.subscribe = AsyncMock()
ps.unsubscribe = AsyncMock()
ps.close = AsyncMock()
ps.get_message = AsyncMock(return_value=None)
r.pubsub = MagicMock(return_value=ps)
return r
def _inject_redis(broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
"""Inject a mock Redis client into the broker, simulating connect()."""
broker._redis = mock_redis
broker._pubsub = mock_redis.pubsub()
class TestConstructor:
def test_stores_config(self) -> None:
b = AsyncRedisBroker(host="h", port=1234, db=2, prefix="pfx", password="pw")
assert b._host == "h"
assert b._port == 1234
assert b._db == 2
assert b._prefix == "pfx"
assert b._password == "pw"
assert b._redis is None
def test_defaults(self) -> None:
b = AsyncRedisBroker()
assert b._host == "localhost"
assert b._port == 6379
assert b._prefix == "turnstone"
class TestConnect:
@pytest.mark.anyio
async def test_creates_connection(self) -> None:
b = AsyncRedisBroker()
mock_r = AsyncMock()
mock_r.pubsub = MagicMock(return_value=AsyncMock())
with patch("redis.asyncio.Redis", return_value=mock_r):
await b.connect()
assert b._redis is mock_r
assert b._pubsub is not None
@pytest.mark.anyio
async def test_connect_idempotent(
self, broker: AsyncRedisBroker, mock_redis: AsyncMock
) -> None:
_inject_redis(broker, mock_redis)
old = broker._redis
await broker.connect()
assert broker._redis is old
class TestPushInbound:
@pytest.mark.anyio
async def test_shared_queue(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.push_inbound('{"type":"send"}')
mock_redis.rpush.assert_awaited_once_with("test:inbound", '{"type":"send"}')
@pytest.mark.anyio
async def test_per_node_queue(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.push_inbound('{"type":"send"}', node_id="node-1")
mock_redis.rpush.assert_awaited_once_with("test:inbound:node-1", '{"type":"send"}')
class TestPublishOutbound:
@pytest.mark.anyio
async def test_publishes(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.publish_outbound("test:events:global", '{"event":"data"}')
mock_redis.publish.assert_awaited_once_with("test:events:global", '{"event":"data"}')
class TestPushResponse:
@pytest.mark.anyio
async def test_rpush_and_expire(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.push_response("req-123", '{"ok":true}')
mock_redis.rpush.assert_awaited_once_with("test:resp:req-123", '{"ok":true}')
mock_redis.expire.assert_awaited_once_with("test:resp:req-123", 120)
class TestSubscribe:
@pytest.mark.anyio
async def test_creates_task(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.subscribe("test:events:global", lambda msg: None)
assert "test:events:global" in broker._callbacks
assert broker._listener_task is not None
assert isinstance(broker._listener_task, asyncio.Task)
# Clean up.
broker._listener_task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await broker._listener_task
class TestUnsubscribe:
@pytest.mark.anyio
async def test_cancels_task(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.subscribe("test:events:ch", lambda msg: None)
assert "test:events:ch" in broker._callbacks
await broker.unsubscribe("test:events:ch")
assert "test:events:ch" not in broker._callbacks
class TestRoutingPrimitives:
@pytest.mark.anyio
async def test_get_ws_owner(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
mock_redis.get.return_value = "node-1"
result = await broker.get_ws_owner("ws-abc")
mock_redis.get.assert_awaited_once_with("test:ws:ws-abc")
assert result == "node-1"
@pytest.mark.anyio
async def test_set_ws_owner(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.set_ws_owner("ws-abc", "node-2")
mock_redis.set.assert_awaited_once_with("test:ws:ws-abc", "node-2")
@pytest.mark.anyio
async def test_set_ws_owner_with_ttl(
self, broker: AsyncRedisBroker, mock_redis: AsyncMock
) -> None:
_inject_redis(broker, mock_redis)
await broker.set_ws_owner("ws-abc", "node-2", ttl=300)
mock_redis.set.assert_awaited_once_with("test:ws:ws-abc", "node-2", ex=300)
@pytest.mark.anyio
async def test_del_ws_owner(self, broker: AsyncRedisBroker, mock_redis: AsyncMock) -> None:
_inject_redis(broker, mock_redis)
await broker.del_ws_owner("ws-abc")
mock_redis.delete.assert_awaited_once_with("test:ws:ws-abc")
class TestClose:
@pytest.mark.anyio
async def test_cancels_tasks_and_closes(
self, broker: AsyncRedisBroker, mock_redis: AsyncMock
) -> None:
_inject_redis(broker, mock_redis)
await broker.subscribe("ch1", lambda m: None)
assert len(broker._callbacks) == 1
assert broker._listener_task is not None
await broker.close()
assert len(broker._callbacks) == 0
assert broker._listener_task is None
assert broker._redis is None
assert broker._pubsub is None
+58
View File
@@ -0,0 +1,58 @@
"""Tests for turnstone.core.audit."""
import json
import pytest
from turnstone.core.audit import record_audit
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
path = str(tmp_path / "test.db")
backend = SQLiteBackend(path)
yield backend
backend.close()
def test_record_audit_basic(storage):
record_audit(
storage, "user-1", "user.create", "user", "u123", {"username": "alice"}, "127.0.0.1"
)
events = storage.list_audit_events()
assert len(events) == 1
ev = events[0]
assert ev["user_id"] == "user-1"
assert ev["action"] == "user.create"
assert ev["resource_type"] == "user"
assert ev["resource_id"] == "u123"
assert ev["ip_address"] == "127.0.0.1"
detail = json.loads(ev["detail"])
assert detail["username"] == "alice"
def test_record_audit_no_detail(storage):
record_audit(storage, "user-1", "token.revoke", "token", "t456")
events = storage.list_audit_events()
assert len(events) == 1
assert events[0]["detail"] == "{}"
def test_record_audit_silent_on_failure():
"""record_audit should not raise even if storage is broken."""
class BrokenStorage:
def record_audit_event(self, **kw):
raise RuntimeError("boom")
# Should not raise
record_audit(BrokenStorage(), "u1", "test.action")
def test_record_audit_generates_unique_ids(storage):
record_audit(storage, "u1", "a.one")
record_audit(storage, "u1", "a.two")
events = storage.list_audit_events()
assert len(events) == 2
assert events[0]["event_id"] != events[1]["event_id"]
+388 -79
View File
@@ -1,7 +1,9 @@
"""Tests for turnstone.core.auth — bearer token authentication and cookies."""
import os
from unittest.mock import patch
import queue
import threading
from unittest.mock import MagicMock, patch
import pytest
@@ -15,7 +17,7 @@ from turnstone.core.auth import (
load_auth_config,
make_clear_cookie,
make_set_cookie,
required_role,
required_scope,
)
# ---------------------------------------------------------------------------
@@ -87,69 +89,59 @@ class TestIsPublicPath:
# ---------------------------------------------------------------------------
class TestRequiredRole:
class TestRequiredScope:
def test_get_api_needs_read(self):
assert required_role("GET", "/api/workstreams") == "read"
assert required_scope("GET", "/api/workstreams") == "read"
def test_get_events_needs_read(self):
assert required_role("GET", "/api/events") == "read"
assert required_scope("GET", "/api/events") == "read"
def test_get_dashboard_needs_read(self):
assert required_role("GET", "/api/dashboard") == "read"
def test_post_send_needs_write(self):
assert required_scope("POST", "/api/send") == "write"
def test_post_send_needs_full(self):
assert required_role("POST", "/api/send") == "full"
def test_post_approve_needs_approve(self):
assert required_scope("POST", "/api/approve") == "approve"
def test_post_approve_needs_full(self):
assert required_role("POST", "/api/approve") == "full"
def test_post_plan_needs_write(self):
assert required_scope("POST", "/api/plan") == "write"
def test_post_plan_needs_full(self):
assert required_role("POST", "/api/plan") == "full"
def test_post_command_needs_write(self):
assert required_scope("POST", "/api/command") == "write"
def test_post_command_needs_full(self):
assert required_role("POST", "/api/command") == "full"
def test_post_workstreams_new_needs_write(self):
assert required_scope("POST", "/api/workstreams/new") == "write"
def test_post_workstreams_new_needs_full(self):
assert required_role("POST", "/api/workstreams/new") == "full"
def test_post_workstreams_close_needs_write(self):
assert required_scope("POST", "/api/workstreams/close") == "write"
def test_post_workstreams_close_needs_full(self):
assert required_role("POST", "/api/workstreams/close") == "full"
def test_all_write_paths_need_full(self):
def test_all_write_paths_need_write(self):
for path in WRITE_PATHS:
assert required_role("POST", path) == "full"
scope = required_scope("POST", path)
assert scope in ("write", "approve"), f"{path} should need write or approve"
def test_post_unknown_path_needs_read(self):
assert required_role("POST", "/api/unknown") == "read"
assert required_scope("POST", "/api/unknown") == "read"
def test_v1_post_send_needs_full(self):
assert required_role("POST", "/v1/api/send") == "full"
def test_v1_post_send_needs_write(self):
assert required_scope("POST", "/v1/api/send") == "write"
def test_v1_post_approve_needs_full(self):
assert required_role("POST", "/v1/api/approve") == "full"
def test_v1_post_approve_needs_approve(self):
assert required_scope("POST", "/v1/api/approve") == "approve"
def test_v1_get_workstreams_needs_read(self):
assert required_role("GET", "/v1/api/workstreams") == "read"
assert required_scope("GET", "/v1/api/workstreams") == "read"
def test_v1_post_cluster_ws_new_needs_full(self):
assert required_role("POST", "/v1/api/cluster/workstreams/new") == "full"
def test_v1_post_cluster_ws_new_needs_write(self):
assert required_scope("POST", "/v1/api/cluster/workstreams/new") == "write"
def test_v1_all_write_paths_need_full(self):
for path in WRITE_PATHS:
v1_path = "/v1" + path
assert required_role("POST", v1_path) == "full", f"{v1_path} should need full"
def test_proxy_v1_send_needs_write(self):
assert required_scope("POST", "/node/node-a/v1/api/send") == "write"
def test_proxy_v1_send_needs_full(self):
assert required_role("POST", "/node/node-a/v1/api/send") == "full"
def test_proxy_v1_approve_needs_full(self):
assert required_role("POST", "/node/node-a/v1/api/approve") == "full"
def test_proxy_v1_cluster_ws_new_needs_full(self):
assert required_role("POST", "/node/node-a/v1/api/cluster/workstreams/new") == "full"
def test_proxy_v1_approve_needs_approve(self):
assert required_scope("POST", "/node/node-a/v1/api/approve") == "approve"
def test_proxy_v1_read_endpoint_needs_read(self):
assert required_role("GET", "/node/node-a/v1/api/workstreams") == "read"
assert required_scope("GET", "/node/node-a/v1/api/workstreams") == "read"
# ---------------------------------------------------------------------------
@@ -268,12 +260,24 @@ class TestMakeSetCookie:
def test_max_age_default(self):
val = make_set_cookie("tok_abc")
assert "Max-Age=2592000" in val # 30 days
assert "Max-Age=86400" in val # 24 hours (matches JWT expiry)
def test_max_age_custom(self):
val = make_set_cookie("tok_abc", max_age=3600)
assert "Max-Age=3600" in val
def test_secure_default(self):
val = make_set_cookie("tok_abc")
assert "; Secure" in val
def test_secure_false(self):
val = make_set_cookie("tok_abc", secure=False)
assert "; Secure" not in val
def test_secure_true(self):
val = make_set_cookie("tok_abc", secure=True)
assert "; Secure" in val
class TestMakeClearCookie:
def test_max_age_zero(self):
@@ -306,68 +310,78 @@ class TestCheckRequest:
)
def test_disabled_allows_all(self, disabled):
allowed, status, msg = check_request(disabled, "POST", "/api/send", None)
allowed, status, msg, _result = check_request(disabled, "POST", "/api/send", None)
assert allowed is True
assert status == 200
def test_disabled_allows_no_header(self, disabled):
allowed, status, msg = check_request(disabled, "GET", "/api/workstreams", None)
allowed, status, msg, _result = check_request(disabled, "GET", "/api/workstreams", None)
assert allowed is True
def test_public_path_no_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/health", None)
allowed, status, msg, _result = check_request(enabled, "GET", "/health", None)
assert allowed is True
assert status == 200
def test_public_root_no_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/", None)
allowed, status, msg, _result = check_request(enabled, "GET", "/", None)
assert allowed is True
def test_public_static_no_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/static/style.css", None)
allowed, status, msg, _result = check_request(enabled, "GET", "/static/style.css", None)
assert allowed is True
def test_api_no_token_401(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/api/workstreams", None)
allowed, status, msg, _result = check_request(enabled, "GET", "/api/workstreams", None)
assert allowed is False
assert status == 401
assert "Unauthorized" in msg
def test_api_invalid_token_401(self, enabled):
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "GET", "/api/workstreams", "Bearer wrong_token"
)
assert allowed is False
assert status == 401
def test_api_read_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/api/workstreams", "Bearer tok_read")
allowed, status, msg, _result = check_request(
enabled, "GET", "/api/workstreams", "Bearer tok_read"
)
assert allowed is True
assert status == 200
def test_api_full_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/api/workstreams", "Bearer tok_full")
allowed, status, msg, _result = check_request(
enabled, "GET", "/api/workstreams", "Bearer tok_full"
)
assert allowed is True
def test_write_read_token_403(self, enabled):
allowed, status, msg = check_request(enabled, "POST", "/api/send", "Bearer tok_read")
allowed, status, msg, _result = check_request(
enabled, "POST", "/api/send", "Bearer tok_read"
)
assert allowed is False
assert status == 403
assert "Forbidden" in msg
def test_write_full_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "POST", "/api/send", "Bearer tok_full")
allowed, status, msg, _result = check_request(
enabled, "POST", "/api/send", "Bearer tok_full"
)
assert allowed is True
assert status == 200
def test_approve_read_token_403(self, enabled):
allowed, status, msg = check_request(enabled, "POST", "/api/approve", "Bearer tok_read")
allowed, status, msg, _result = check_request(
enabled, "POST", "/api/approve", "Bearer tok_read"
)
assert allowed is False
assert status == 403
def test_proxy_write_read_token_403(self, enabled):
"""Read tokens cannot escalate to write ops via proxy routes."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "POST", "/node/node-a/api/send", "Bearer tok_read"
)
assert allowed is False
@@ -375,7 +389,7 @@ class TestCheckRequest:
def test_proxy_write_trailing_slash_read_token_403(self, enabled):
"""Trailing slash must not bypass write-role check on proxy routes."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "POST", "/node/node-a/api/send/", "Bearer tok_read"
)
assert allowed is False
@@ -383,20 +397,22 @@ class TestCheckRequest:
def test_direct_write_trailing_slash_read_token_403(self, enabled):
"""Trailing slash must not bypass write-role check on direct routes."""
allowed, status, msg = check_request(enabled, "POST", "/api/send/", "Bearer tok_read")
allowed, status, msg, _result = check_request(
enabled, "POST", "/api/send/", "Bearer tok_read"
)
assert allowed is False
assert status == 403
def test_proxy_write_full_token_ok(self, enabled):
"""Full tokens pass through proxy write routes."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "POST", "/node/node-a/api/send", "Bearer tok_full"
)
assert allowed is True
def test_proxy_v1_write_read_token_403(self, enabled):
"""Read tokens cannot escalate to write ops via v1 proxy routes."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "POST", "/node/node-a/v1/api/send", "Bearer tok_read"
)
assert allowed is False
@@ -404,14 +420,14 @@ class TestCheckRequest:
def test_proxy_v1_write_full_token_ok(self, enabled):
"""Full tokens pass through v1 proxy write routes."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "POST", "/node/node-a/v1/api/send", "Bearer tok_full"
)
assert allowed is True
def test_proxy_v1_cluster_ws_new_read_403(self, enabled):
"""Read tokens cannot create workstreams via v1 proxy."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled,
"POST",
"/node/node-a/v1/api/cluster/workstreams/new",
@@ -422,25 +438,27 @@ class TestCheckRequest:
def test_proxy_read_endpoint_read_token_ok(self, enabled):
"""Read tokens can access proxy read endpoints."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "GET", "/node/node-a/api/workstreams", "Bearer tok_read"
)
assert allowed is True
def test_console_create_ws_read_token_403(self, enabled):
"""Read tokens cannot create workstreams."""
allowed, status, msg = check_request(
allowed, status, msg, _result = check_request(
enabled, "POST", "/api/cluster/workstreams/new", "Bearer tok_read"
)
assert allowed is False
assert status == 403
def test_approve_full_token_ok(self, enabled):
allowed, status, msg = check_request(enabled, "POST", "/api/approve", "Bearer tok_full")
allowed, status, msg, _result = check_request(
enabled, "POST", "/api/approve", "Bearer tok_full"
)
assert allowed is True
def test_no_auth_header_string(self, enabled):
allowed, status, msg = check_request(enabled, "GET", "/api/dashboard", "")
allowed, status, msg, _result = check_request(enabled, "GET", "/api/dashboard", "")
assert allowed is False
assert status == 401
@@ -461,7 +479,7 @@ class TestCheckRequestWithCookie:
)
def test_cookie_fallback_when_no_bearer(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"GET",
"/api/workstreams",
@@ -473,7 +491,7 @@ class TestCheckRequestWithCookie:
def test_bearer_takes_precedence_over_cookie(self, enabled):
# Bearer is full, cookie is read — Bearer should win
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"POST",
"/api/send",
@@ -483,7 +501,7 @@ class TestCheckRequestWithCookie:
assert allowed is True
def test_invalid_cookie_401(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"GET",
"/api/workstreams",
@@ -494,7 +512,7 @@ class TestCheckRequestWithCookie:
assert status == 401
def test_cookie_read_on_write_403(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"POST",
"/api/send",
@@ -505,7 +523,7 @@ class TestCheckRequestWithCookie:
assert status == 403
def test_cookie_full_on_write_ok(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"POST",
"/api/send",
@@ -515,7 +533,7 @@ class TestCheckRequestWithCookie:
assert allowed is True
def test_no_cookie_no_bearer_401(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"GET",
"/api/workstreams",
@@ -526,7 +544,7 @@ class TestCheckRequestWithCookie:
assert status == 401
def test_login_path_public(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"POST",
"/api/auth/login",
@@ -535,7 +553,7 @@ class TestCheckRequestWithCookie:
assert allowed is True
def test_logout_path_public(self, enabled):
allowed, status, _ = check_request(
allowed, status, _, _r = check_request(
enabled,
"POST",
"/api/auth/logout",
@@ -552,12 +570,28 @@ class TestCheckRequestWithCookie:
class TestLoadAuthConfig:
"""Tests for load_auth_config with mocked config + env vars."""
def test_default_disabled(self):
def test_default_enabled(self):
with patch("turnstone.core.config.load_config", return_value={}):
cfg = load_auth_config()
assert cfg.enabled is False
assert cfg.enabled is True
assert cfg.tokens == {}
def test_explicit_disable(self):
with (
patch("turnstone.core.config.load_config", return_value={"enabled": False}),
patch.dict(os.environ, {}, clear=True),
):
cfg = load_auth_config()
assert cfg.enabled is False
def test_env_disable(self):
with (
patch("turnstone.core.config.load_config", return_value={}),
patch.dict(os.environ, {"TURNSTONE_AUTH_ENABLED": "0"}, clear=True),
):
cfg = load_auth_config()
assert cfg.enabled is False
def test_config_file_tokens(self):
mock_cfg = {
"enabled": True,
@@ -685,7 +719,7 @@ class TestServerAuth:
srv_mod._metrics.model = "test-model"
mock_session = MagicMock()
mock_session.session_id = "test-session-id"
mock_session.ws_id = "test-session-id"
mock_ws = MagicMock()
mock_ws.id = "test-ws"
@@ -705,6 +739,7 @@ class TestServerAuth:
enabled=True,
tokens={"tok_full": "full", "tok_read": "read"},
),
cors_origins=["*"],
)
cls.client = TestClient(app, raise_server_exceptions=False)
@@ -902,7 +937,7 @@ class TestServerLogin:
srv_mod._metrics.model = "test-model"
mock_session = MagicMock()
mock_session.session_id = "test-session-id"
mock_session.ws_id = "test-session-id"
mock_ws = MagicMock()
mock_ws.id = "test-ws"
@@ -1040,3 +1075,277 @@ class TestConsoleLogin:
self.test_client.post("/v1/api/auth/logout")
resp = self.test_client.get("/v1/api/cluster/overview")
assert resp.status_code == 401
# ---------------------------------------------------------------------------
# Security hardening tests
# ---------------------------------------------------------------------------
class TestLoginRateLimiter:
def test_allows_under_limit(self):
from turnstone.core.auth import LoginRateLimiter
limiter = LoginRateLimiter(max_attempts=3, window_seconds=60)
for _ in range(3):
ok, _ = limiter.check("ip:1.2.3.4")
assert ok
limiter.record("ip:1.2.3.4")
# 4th should be blocked (3 recorded)
ok, retry = limiter.check("ip:1.2.3.4")
assert not ok
assert retry > 0
def test_different_keys_independent(self):
from turnstone.core.auth import LoginRateLimiter
limiter = LoginRateLimiter(max_attempts=2, window_seconds=60)
limiter.record("ip:a")
limiter.record("ip:a")
ok_a, _ = limiter.check("ip:a")
ok_b, _ = limiter.check("ip:b")
assert not ok_a
assert ok_b
def test_cleanup(self):
from turnstone.core.auth import LoginRateLimiter
limiter = LoginRateLimiter(max_attempts=1, window_seconds=60)
limiter.record("ip:old")
removed = limiter.cleanup(max_age=0.0)
assert removed == 1
ok, _ = limiter.check("ip:old")
assert ok
def test_max_keys_protection(self):
from turnstone.core.auth import LoginRateLimiter
limiter = LoginRateLimiter(max_attempts=5, window_seconds=60)
limiter.MAX_KEYS = 2
limiter.record("a")
limiter.record("b")
limiter.record("c") # should be silently dropped (at capacity)
assert "c" not in limiter._attempts
class TestJWTAudienceIssuer:
SECRET = "test-secret-that-is-at-least-32-chars"
def test_create_jwt_includes_iss(self):
import jwt as pyjwt
from turnstone.core.auth import JWT_ISSUER, create_jwt
token = create_jwt("user1", frozenset({"read"}), "test", self.SECRET)
payload = pyjwt.decode(
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
)
assert payload["iss"] == JWT_ISSUER
def test_create_jwt_with_audience(self):
import jwt as pyjwt
from turnstone.core.auth import JWT_AUD_SERVER, create_jwt
token = create_jwt(
"user1", frozenset({"read"}), "test", self.SECRET, audience=JWT_AUD_SERVER
)
payload = pyjwt.decode(token, self.SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
assert payload["aud"] == JWT_AUD_SERVER
def test_validate_jwt_wrong_audience_rejected(self):
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, create_jwt, validate_jwt
token = create_jwt(
"user1", frozenset({"read"}), "test", self.SECRET, audience=JWT_AUD_SERVER
)
result = validate_jwt(token, self.SECRET, audience=JWT_AUD_CONSOLE)
assert result is None
def test_validate_jwt_correct_audience_accepted(self):
from turnstone.core.auth import JWT_AUD_SERVER, create_jwt, validate_jwt
token = create_jwt(
"user1", frozenset({"read"}), "test", self.SECRET, audience=JWT_AUD_SERVER
)
result = validate_jwt(token, self.SECRET, audience=JWT_AUD_SERVER)
assert result is not None
assert result.user_id == "user1"
def test_validate_jwt_no_audience_backward_compat(self):
from turnstone.core.auth import create_jwt, validate_jwt
# Token without aud claim should be accepted when audience="" (backward compat)
token = create_jwt("user1", frozenset({"read"}), "test", self.SECRET)
result = validate_jwt(token, self.SECRET, audience="")
assert result is not None
class TestServiceTokenManager:
SECRET = "test-secret-that-is-at-least-32-chars"
def test_auto_mints_on_first_access(self):
from turnstone.core.auth import ServiceTokenManager
mgr = ServiceTokenManager(
user_id="svc",
scopes=frozenset({"read"}),
source="test",
secret=self.SECRET,
)
token = mgr.token
assert token # non-empty
assert isinstance(token, str)
def test_bearer_header_format(self):
from turnstone.core.auth import ServiceTokenManager
mgr = ServiceTokenManager(
user_id="svc",
scopes=frozenset({"read"}),
source="test",
secret=self.SECRET,
)
header = mgr.bearer_header
assert "Authorization" in header
assert header["Authorization"].startswith("Bearer ")
def test_token_stable_within_window(self):
from turnstone.core.auth import ServiceTokenManager
mgr = ServiceTokenManager(
user_id="svc",
scopes=frozenset({"read"}),
source="test",
secret=self.SECRET,
expiry_hours=1,
)
t1 = mgr.token
t2 = mgr.token
assert t1 == t2
def test_token_rotates_near_expiry(self):
from turnstone.core.auth import ServiceTokenManager
mgr = ServiceTokenManager(
user_id="svc",
scopes=frozenset({"read"}),
source="test",
secret=self.SECRET,
expiry_hours=1,
)
_ = mgr.token # initial mint
# Simulate expiry by backdating _expires_at
mgr._expires_at = 0.0
t2 = mgr.token
# Token was re-minted (even if payload matches within same second,
# the internal state was refreshed)
assert t2 # non-empty, valid token
assert mgr._expires_at > 0.0 # was refreshed
def test_audience_included(self):
import jwt as pyjwt
from turnstone.core.auth import JWT_AUD_SERVER, ServiceTokenManager
mgr = ServiceTokenManager(
user_id="svc",
scopes=frozenset({"read"}),
source="test",
secret=self.SECRET,
audience=JWT_AUD_SERVER,
)
payload = pyjwt.decode(
mgr.token, self.SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER
)
assert payload["aud"] == JWT_AUD_SERVER
class TestIsSecureRequest:
def test_https_scheme(self):
from turnstone.core.auth import is_secure_request
assert is_secure_request({}, scheme="https") is True
def test_http_scheme(self):
from turnstone.core.auth import is_secure_request
assert is_secure_request({}, scheme="http") is False
def test_x_forwarded_proto_https(self):
from turnstone.core.auth import is_secure_request
assert is_secure_request({"x-forwarded-proto": "https"}, scheme="http") is True
def test_x_forwarded_proto_http(self):
from turnstone.core.auth import is_secure_request
assert is_secure_request({"x-forwarded-proto": "http"}, scheme="http") is False
class TestSecretStrength:
def test_short_secret_warns(self, caplog):
import logging
from turnstone.core.auth import _MIN_SECRET_LENGTH
with caplog.at_level(logging.WARNING, logger="turnstone.core.auth"):
import turnstone.core.auth as auth_mod
old = os.environ.get("TURNSTONE_JWT_SECRET", "")
os.environ["TURNSTONE_JWT_SECRET"] = "short"
try:
secret = auth_mod.load_jwt_secret()
assert secret == "short"
assert any(str(_MIN_SECRET_LENGTH) in r.message for r in caplog.records)
finally:
if old:
os.environ["TURNSTONE_JWT_SECRET"] = old
else:
os.environ.pop("TURNSTONE_JWT_SECRET", None)
class TestCorsConfigurable:
"""Verify CORS middleware is only added when origins are configured."""
def test_no_cors_origins_no_cors_headers(self):
"""Without cors_origins, no Access-Control headers."""
from starlette.testclient import TestClient
import turnstone.server as srv_mod
app = srv_mod.create_app(
workstreams=MagicMock(),
global_queue=queue.Queue(),
global_listeners=[],
global_listeners_lock=threading.Lock(),
skip_permissions=False,
auth_config=AuthConfig(enabled=False),
)
client = TestClient(app)
resp = client.get("/health", headers={"Origin": "http://evil.com"})
assert "Access-Control-Allow-Origin" not in resp.headers
client.close()
def test_cors_origins_set(self):
"""With cors_origins, CORS headers are present."""
from starlette.testclient import TestClient
import turnstone.server as srv_mod
app = srv_mod.create_app(
workstreams=MagicMock(),
global_queue=queue.Queue(),
global_listeners=[],
global_listeners_lock=threading.Lock(),
skip_permissions=False,
auth_config=AuthConfig(enabled=False),
cors_origins=["http://example.com"],
)
client = TestClient(app)
resp = client.get(
"/health",
headers={"Origin": "http://example.com"},
)
assert resp.headers.get("Access-Control-Allow-Origin") == "http://example.com"
client.close()
+357
View File
@@ -0,0 +1,357 @@
"""Tests for user identity, API tokens, JWT, and scoped auth."""
from __future__ import annotations
import time
import pytest
from turnstone.core.auth import (
AuthConfig,
AuthResult,
_authenticate_token,
check_request,
create_jwt,
generate_token,
hash_password,
hash_token,
parse_scopes,
required_scope,
token_prefix,
validate_jwt,
verify_password,
)
# ---------------------------------------------------------------------------
# AuthResult
# ---------------------------------------------------------------------------
class TestAuthResult:
def test_frozen(self):
r = AuthResult(user_id="u1", scopes=frozenset({"read"}), token_source="config")
with pytest.raises(AttributeError):
r.user_id = "u2" # type: ignore[misc]
def test_has_scope(self):
r = AuthResult(user_id="", scopes=frozenset({"read", "write"}), token_source="config")
assert r.has_scope("read")
assert r.has_scope("write")
assert not r.has_scope("approve")
def test_empty_scopes(self):
r = AuthResult(user_id="", scopes=frozenset(), token_source="config")
assert not r.has_scope("read")
# ---------------------------------------------------------------------------
# Token generation and hashing
# ---------------------------------------------------------------------------
class TestTokenHelpers:
def test_generate_token_format(self):
tok = generate_token()
assert tok.startswith("ts_")
assert len(tok) == 3 + 64 # ts_ + 64 hex chars
def test_generate_token_unique(self):
tokens = {generate_token() for _ in range(10)}
assert len(tokens) == 10
def test_hash_token_deterministic(self):
assert hash_token("ts_abc") == hash_token("ts_abc")
def test_hash_token_hex(self):
h = hash_token("test")
assert len(h) == 64 # SHA-256 hex
int(h, 16) # valid hex
def test_token_prefix(self):
assert token_prefix("ts_abcdefgh1234") == "ts_abcde"
# ---------------------------------------------------------------------------
# Password hashing (bcrypt)
# ---------------------------------------------------------------------------
class TestPasswordHashing:
def test_hash_and_verify(self):
pw = "hunter2"
hashed = hash_password(pw)
assert verify_password(pw, hashed)
def test_wrong_password(self):
hashed = hash_password("correct")
assert not verify_password("wrong", hashed)
def test_hash_is_different_each_time(self):
h1 = hash_password("same")
h2 = hash_password("same")
assert h1 != h2 # different salts
# ---------------------------------------------------------------------------
# Scope parsing
# ---------------------------------------------------------------------------
class TestParseScopes:
def test_single_scope(self):
assert parse_scopes("read") == frozenset({"read"})
def test_hierarchy_write(self):
assert parse_scopes("write") == frozenset({"read", "write"})
def test_hierarchy_approve(self):
assert parse_scopes("approve") == frozenset({"read", "write", "approve"})
def test_comma_separated(self):
assert parse_scopes("read,write") == frozenset({"read", "write"})
def test_redundant_scopes(self):
# approve already includes read,write
assert parse_scopes("read,approve") == frozenset({"read", "write", "approve"})
def test_empty_string(self):
assert parse_scopes("") == frozenset()
def test_invalid_scope_filtered(self):
assert parse_scopes("bogus") == frozenset()
def test_mixed_valid_invalid(self):
assert parse_scopes("read,bogus,approve") == frozenset({"read", "write", "approve"})
# ---------------------------------------------------------------------------
# JWT create / validate
# ---------------------------------------------------------------------------
class TestJWT:
SECRET = "test-secret-key-for-jwt"
def test_round_trip(self):
scopes = frozenset({"read", "write"})
token = create_jwt("user123", scopes, "database", self.SECRET, expiry_hours=1)
result = validate_jwt(token, self.SECRET)
assert result is not None
assert result.user_id == "user123"
assert result.scopes == frozenset({"read", "write"})
def test_expired_token(self):
import jwt
payload = {
"sub": "user1",
"scopes": "read",
"src": "database",
"iat": int(time.time()) - 7200,
"exp": int(time.time()) - 3600,
}
token = jwt.encode(payload, self.SECRET, algorithm="HS256")
assert validate_jwt(token, self.SECRET) is None
def test_invalid_signature(self):
token = create_jwt("user1", frozenset({"read"}), "db", self.SECRET)
assert validate_jwt(token, "wrong-secret") is None
def test_malformed_token(self):
assert validate_jwt("not.a.jwt", self.SECRET) is None
def test_contains_dots(self):
"""JWTs contain dots, used for detection."""
token = create_jwt("u1", frozenset({"read"}), "db", self.SECRET)
assert "." in token
# ---------------------------------------------------------------------------
# required_scope
# ---------------------------------------------------------------------------
class TestRequiredScope:
def test_get_read(self):
assert required_scope("GET", "/api/workstreams") == "read"
def test_post_write(self):
assert required_scope("POST", "/api/send") == "write"
def test_post_approve(self):
assert required_scope("POST", "/api/approve") == "approve"
def test_admin_prefix(self):
assert required_scope("GET", "/api/admin/users") == "approve"
assert required_scope("POST", "/api/admin/users") == "approve"
assert required_scope("DELETE", "/api/admin/users/abc") == "approve"
def test_versioned_path(self):
assert required_scope("POST", "/v1/api/send") == "write"
assert required_scope("POST", "/v1/api/approve") == "approve"
def test_proxy_write(self):
assert required_scope("POST", "/node/n1/api/send") == "write"
def test_proxy_approve(self):
assert required_scope("POST", "/node/n1/api/approve") == "approve"
# ---------------------------------------------------------------------------
# _authenticate_token
# ---------------------------------------------------------------------------
class TestAuthenticateToken:
def test_config_token_read(self):
cfg = AuthConfig(enabled=True, tokens={"tok_read": "read"})
result = _authenticate_token("tok_read", cfg)
assert result is not None
assert result.scopes == frozenset({"read"})
assert result.token_source == "config"
def test_config_token_full(self):
cfg = AuthConfig(enabled=True, tokens={"tok_full": "full"})
result = _authenticate_token("tok_full", cfg)
assert result is not None
assert result.scopes == frozenset({"read", "write", "approve"})
def test_jwt_token(self):
secret = "test-secret"
jwt_tok = create_jwt("user1", frozenset({"read", "write"}), "db", secret)
cfg = AuthConfig(enabled=True)
result = _authenticate_token(jwt_tok, cfg, jwt_secret=secret)
assert result is not None
assert result.user_id == "user1"
assert result.token_source == "db"
def test_api_token_with_storage(self):
"""API tokens are looked up by hash in storage."""
raw = generate_token()
class MockStorage:
def get_api_token_by_hash(self, token_hash):
expected = hash_token(raw)
if token_hash == expected:
return {
"token_id": "tid",
"token_prefix": "ts_abcde",
"user_id": "user1",
"name": "test",
"scopes": "read,write",
"created": "2026-01-01T00:00:00",
}
return None
cfg = AuthConfig(enabled=True)
result = _authenticate_token(raw, cfg, storage=MockStorage())
assert result is not None
assert result.user_id == "user1"
assert result.has_scope("write")
assert result.token_source == "database"
def test_api_token_expired(self):
"""Expired API tokens are rejected."""
raw = generate_token()
class MockStorage:
def get_api_token_by_hash(self, token_hash):
return {
"token_id": "tid",
"token_prefix": "ts_abcde",
"user_id": "user1",
"name": "test",
"scopes": "read",
"created": "2020-01-01T00:00:00",
"expires": "2020-01-02T00:00:00",
}
cfg = AuthConfig(enabled=True)
result = _authenticate_token(raw, cfg, storage=MockStorage())
assert result is None
def test_unknown_token(self):
cfg = AuthConfig(enabled=True, tokens={"tok": "full"})
result = _authenticate_token("unknown", cfg)
assert result is None
# ---------------------------------------------------------------------------
# check_request with scopes
# ---------------------------------------------------------------------------
class TestCheckRequestScopes:
def test_config_read_on_write_403(self):
cfg = AuthConfig(enabled=True, tokens={"tok_read": "read"})
allowed, status, msg, _ = check_request(cfg, "POST", "/api/send", "Bearer tok_read")
assert not allowed
assert status == 403
assert "write" in msg
def test_config_read_on_approve_403(self):
cfg = AuthConfig(enabled=True, tokens={"tok_read": "read"})
allowed, status, msg, _ = check_request(cfg, "POST", "/api/approve", "Bearer tok_read")
assert not allowed
assert status == 403
assert "approve" in msg
def test_config_full_on_approve_ok(self):
cfg = AuthConfig(enabled=True, tokens={"tok_full": "full"})
allowed, status, msg, result = check_request(cfg, "POST", "/api/approve", "Bearer tok_full")
assert allowed
assert result is not None
assert result.has_scope("approve")
def test_jwt_with_scopes(self):
secret = "test"
jwt_tok = create_jwt("u1", frozenset({"read", "write"}), "db", secret)
cfg = AuthConfig(enabled=True)
allowed, status, msg, result = check_request(
cfg,
"POST",
"/api/send",
f"Bearer {jwt_tok}",
jwt_secret=secret,
)
assert allowed
assert result is not None
assert result.user_id == "u1"
def test_jwt_insufficient_scope(self):
secret = "test"
jwt_tok = create_jwt("u1", frozenset({"read"}), "db", secret)
cfg = AuthConfig(enabled=True)
allowed, status, msg, _ = check_request(
cfg,
"POST",
"/api/send",
f"Bearer {jwt_tok}",
jwt_secret=secret,
)
assert not allowed
assert status == 403
def test_admin_path_requires_approve(self):
cfg = AuthConfig(enabled=True, tokens={"tok_read": "read"})
allowed, status, msg, _ = check_request(
cfg,
"GET",
"/v1/api/admin/users",
"Bearer tok_read",
)
assert not allowed
assert status == 403
def test_backward_compat_role_full(self):
"""Config tokens with role='full' get all scopes."""
cfg = AuthConfig(enabled=True, tokens={"tok_full": "full"})
allowed, _, _, result = check_request(
cfg,
"GET",
"/v1/api/admin/users",
"Bearer tok_full",
)
assert allowed
assert result is not None
assert result.has_scope("approve")
+630
View File
@@ -0,0 +1,630 @@
"""Tests for the bootstrap wizard module."""
from __future__ import annotations
import os
import socket
from pathlib import Path
from unittest.mock import MagicMock, patch
from turnstone.bootstrap import (
SYSTEM_PROMPT,
TOOLS,
_BootstrapLLM,
_FinishError,
_mask_secrets,
_tool_check_docker,
_tool_check_port,
_tool_finish,
_tool_generate_secret,
_tool_read_file,
_tool_validate_api_key,
_tool_write_file,
execute_tool,
)
# ---------------------------------------------------------------------------
# Tool function tests
# ---------------------------------------------------------------------------
class TestReadFile:
def test_existing_file(self, tmp_path: Path) -> None:
f = tmp_path / "test.txt"
f.write_text("hello world")
result = _tool_read_file(tmp_path, {"path": "test.txt"})
assert result == "hello world"
def test_missing_file(self, tmp_path: Path) -> None:
result = _tool_read_file(tmp_path, {"path": "nope.txt"})
assert "Error: file not found" in result
def test_nested_path(self, tmp_path: Path) -> None:
sub = tmp_path / "sub"
sub.mkdir()
f = sub / "nested.txt"
f.write_text("nested content")
result = _tool_read_file(tmp_path, {"path": "sub/nested.txt"})
assert result == "nested content"
def test_path_traversal_blocked(self, tmp_path: Path) -> None:
result = _tool_read_file(tmp_path, {"path": "../../etc/passwd"})
assert "escapes project directory" in result
def test_absolute_path_blocked(self, tmp_path: Path) -> None:
result = _tool_read_file(tmp_path, {"path": "/etc/passwd"})
assert "escapes project directory" in result
class TestWriteFile:
def test_write_confirmed(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="y"):
result = _tool_write_file(tmp_path, {"path": "out.txt", "content": "data\n"})
assert "written successfully" in result
assert (tmp_path / "out.txt").read_text() == "data\n"
def test_write_declined(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="n"):
result = _tool_write_file(tmp_path, {"path": "out.txt", "content": "data\n"})
assert "declined" in result
assert not (tmp_path / "out.txt").exists()
def test_write_creates_parent_dirs(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="y"):
result = _tool_write_file(tmp_path, {"path": "a/b/c.txt", "content": "deep\n"})
assert "written successfully" in result
assert (tmp_path / "a" / "b" / "c.txt").read_text() == "deep\n"
def test_sh_files_are_executable(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="y"):
_tool_write_file(tmp_path, {"path": "setup.sh", "content": "#!/bin/bash\n"})
mode = (tmp_path / "setup.sh").stat().st_mode
assert mode & 0o110 # user + group executable, not world
def test_path_traversal_blocked(self, tmp_path: Path) -> None:
result = _tool_write_file(tmp_path, {"path": "../../escape.txt", "content": "bad\n"})
assert "escapes project directory" in result
def test_default_enter_confirms(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value=""):
result = _tool_write_file(tmp_path, {"path": "ok.txt", "content": "ok\n"})
assert "written successfully" in result
def test_duplicate_write_skipped(self, tmp_path: Path) -> None:
(tmp_path / "dup.txt").write_text("same\n")
result = _tool_write_file(tmp_path, {"path": "dup.txt", "content": "same\n"})
assert "already exists" in result
def test_different_content_still_prompts(self, tmp_path: Path) -> None:
(tmp_path / "changed.txt").write_text("old\n")
with patch("builtins.input", return_value="y"):
result = _tool_write_file(tmp_path, {"path": "changed.txt", "content": "new\n"})
assert "written successfully" in result
assert (tmp_path / "changed.txt").read_text() == "new\n"
class TestGenerateSecret:
def test_default_length(self) -> None:
secret = _tool_generate_secret({})
assert len(secret) == 64 # 32 bytes -> 64 hex chars
def test_custom_length(self) -> None:
secret = _tool_generate_secret({"length": 16})
assert len(secret) == 32
def test_uniqueness(self) -> None:
s1 = _tool_generate_secret({})
s2 = _tool_generate_secret({})
assert s1 != s2
def test_invalid_length_fallback(self) -> None:
secret = _tool_generate_secret({"length": -1})
assert len(secret) == 64 # falls back to 32 bytes
def test_excessive_length_capped(self) -> None:
secret = _tool_generate_secret({"length": 99999})
assert len(secret) == 64 # falls back to 32 bytes
class TestCheckPort:
def test_available_port(self) -> None:
# Pick a random high port that's likely free
result = _tool_check_port({"port": 59123})
assert "AVAILABLE" in result or "IN USE" in result
def test_in_use_port(self) -> None:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("127.0.0.1", 0))
port = sock.getsockname()[1]
sock.listen(1)
result = _tool_check_port({"port": port})
assert "IN USE" in result
def test_invalid_port(self) -> None:
result = _tool_check_port({"port": -1})
assert "Error" in result
def test_port_zero(self) -> None:
result = _tool_check_port({"port": 0})
assert "Error" in result
class TestCheckDocker:
def test_docker_installed(self) -> None:
mock_docker = MagicMock()
mock_docker.returncode = 0
mock_docker.stdout = "24.0.7"
mock_compose = MagicMock()
mock_compose.returncode = 0
mock_compose.stdout = "2.24.5"
with patch("subprocess.run", side_effect=[mock_docker, mock_compose]):
result = _tool_check_docker({})
assert "Docker: installed" in result
assert "Docker Compose: installed" in result
def test_docker_not_installed(self) -> None:
with patch("subprocess.run", side_effect=FileNotFoundError):
result = _tool_check_docker({})
assert "NOT installed" in result or "NOT available" in result
def test_docker_daemon_not_running(self) -> None:
mock_docker = MagicMock()
mock_docker.returncode = 1
mock_docker.stderr = "Cannot connect to the Docker daemon"
mock_compose = MagicMock()
mock_compose.returncode = 1
with patch("subprocess.run", side_effect=[mock_docker, mock_compose]):
result = _tool_check_docker({})
assert "NOT running" in result
class TestValidateApiKey:
def test_openai_success(self) -> None:
mock_client = MagicMock()
mock_client.models.list.return_value = []
with patch("openai.OpenAI", return_value=mock_client):
result = _tool_validate_api_key({"provider": "openai", "api_key": "sk-test"})
assert "Success" in result
def test_openai_failure(self) -> None:
with patch("openai.OpenAI") as mock_cls:
mock_cls.return_value.models.list.side_effect = Exception("Invalid key")
result = _tool_validate_api_key({"provider": "openai", "api_key": "bad"})
assert "Failed" in result
def test_unknown_provider(self) -> None:
result = _tool_validate_api_key({"provider": "unknown", "api_key": "x"})
assert "unknown" in result
class TestExecuteTool:
def test_unknown_tool(self, tmp_path: Path) -> None:
result = execute_tool("nonexistent", {}, tmp_path)
assert "unknown tool" in result
def test_dispatches_correctly(self, tmp_path: Path) -> None:
f = tmp_path / "hello.txt"
f.write_text("hi")
result = execute_tool("read_file", {"path": "hello.txt"}, tmp_path)
assert result == "hi"
def test_finish_raises(self, tmp_path: Path) -> None:
import pytest
with pytest.raises(_FinishError, match="All done"):
execute_tool("finish", {"summary": "All done"}, tmp_path)
class TestFinishTool:
def test_raises_with_summary(self) -> None:
import pytest
with pytest.raises(_FinishError) as exc_info:
_tool_finish({"summary": "Configured production deployment."})
assert exc_info.value.summary == "Configured production deployment."
def test_default_summary(self) -> None:
import pytest
with pytest.raises(_FinishError) as exc_info:
_tool_finish({})
assert exc_info.value.summary == "Setup complete."
# ---------------------------------------------------------------------------
# Secret masking tests
# ---------------------------------------------------------------------------
class TestMaskSecrets:
def test_masks_api_key(self) -> None:
text = "OPENAI_API_KEY=sk-1234567890abcdef"
result = _mask_secrets(text)
assert "sk-1" in result
assert "cdef" in result
assert "1234567890abcde" not in result
def test_preserves_comments(self) -> None:
text = "# OPENAI_API_KEY=sk-1234567890abcdef"
result = _mask_secrets(text)
assert result == text
def test_preserves_short_values(self) -> None:
text = "TOKEN=short"
result = _mask_secrets(text)
assert result == text
def test_preserves_non_sensitive(self) -> None:
text = "MODEL=gpt-5.4"
result = _mask_secrets(text)
assert result == text
# ---------------------------------------------------------------------------
# Message conversion tests (Anthropic)
# ---------------------------------------------------------------------------
class TestAnthropicConversion:
"""Test the Anthropic message/tool conversion inside _BootstrapLLM."""
def _make_llm(self) -> _BootstrapLLM:
return _BootstrapLLM("anthropic", MagicMock(), "test-model")
def test_tool_format_conversion(self) -> None:
"""OpenAI tool format should convert to Anthropic format."""
llm = self._make_llm()
# The conversion happens inside _complete_anthropic; we test indirectly
# by checking the tools passed to the mock client
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="hello")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
llm.complete(
[{"role": "system", "content": "sys"}, {"role": "user", "content": "hi"}],
TOOLS[:1], # Just read_file
)
call_kwargs = llm.client.messages.create.call_args[1]
api_tools = call_kwargs["tools"]
assert len(api_tools) == 1
assert api_tools[0]["name"] == "read_file"
assert "input_schema" in api_tools[0]
assert "description" in api_tools[0]
def test_system_message_extraction(self) -> None:
"""System message should be extracted to system parameter."""
llm = self._make_llm()
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="ok")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
llm.complete(
[{"role": "system", "content": "test system"}, {"role": "user", "content": "hi"}],
[],
)
call_kwargs = llm.client.messages.create.call_args[1]
assert call_kwargs["system"] == "test system"
# System should NOT appear in messages
for msg in call_kwargs["messages"]:
assert msg["role"] != "system"
def test_tool_result_conversion(self) -> None:
"""OpenAI tool result messages should convert to Anthropic format."""
llm = self._make_llm()
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="got it")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
messages = [
{"role": "system", "content": "sys"},
{"role": "user", "content": "hi"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "tc_1",
"type": "function",
"function": {"name": "check_docker", "arguments": "{}"},
}
],
},
{
"role": "tool",
"tool_call_id": "tc_1",
"content": "Docker: installed",
},
]
llm.complete(messages, TOOLS)
call_kwargs = llm.client.messages.create.call_args[1]
api_messages = call_kwargs["messages"]
# Find the tool_result message
tool_result_found = False
for msg in api_messages:
if msg["role"] == "user" and isinstance(msg.get("content"), list):
for block in msg["content"]:
if isinstance(block, dict) and block.get("type") == "tool_result":
assert block["tool_use_id"] == "tc_1"
assert block["content"] == "Docker: installed"
tool_result_found = True
assert tool_result_found
def test_tool_use_blocks_in_assistant(self) -> None:
"""Assistant messages with tool_calls should convert to content blocks."""
llm = self._make_llm()
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="ok")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
messages = [
{"role": "system", "content": "sys"},
{"role": "user", "content": "hi"},
{
"role": "assistant",
"content": "Let me check",
"tool_calls": [
{
"id": "tc_1",
"type": "function",
"function": {"name": "check_docker", "arguments": "{}"},
}
],
},
{"role": "tool", "tool_call_id": "tc_1", "content": "ok"},
]
llm.complete(messages, TOOLS)
call_kwargs = llm.client.messages.create.call_args[1]
api_messages = call_kwargs["messages"]
# First message should be user "hi"
assert api_messages[0]["role"] == "user"
# Second should be assistant with content blocks
assistant_msg = api_messages[1]
assert assistant_msg["role"] == "assistant"
assert isinstance(assistant_msg["content"], list)
# Should have text block + tool_use block
types = [b["type"] for b in assistant_msg["content"]]
assert "text" in types
assert "tool_use" in types
class TestOpenAICompletion:
"""Test the OpenAI path of _BootstrapLLM."""
def test_text_response(self) -> None:
llm = _BootstrapLLM("openai", MagicMock(), "gpt-5.4")
mock_choice = MagicMock()
mock_choice.message.content = "Hello!"
mock_choice.message.tool_calls = None
mock_choice.finish_reason = "stop"
llm.client.chat.completions.create.return_value = MagicMock(choices=[mock_choice])
content, tool_calls, reason = llm.complete([{"role": "user", "content": "hi"}], TOOLS)
assert content == "Hello!"
assert tool_calls is None
assert reason == "stop"
def test_tool_call_response(self) -> None:
llm = _BootstrapLLM("openai", MagicMock(), "gpt-5.4")
mock_tc = MagicMock()
mock_tc.id = "call_123"
mock_tc.function.name = "check_docker"
mock_tc.function.arguments = "{}"
mock_choice = MagicMock()
mock_choice.message.content = ""
mock_choice.message.tool_calls = [mock_tc]
mock_choice.finish_reason = "tool_calls"
llm.client.chat.completions.create.return_value = MagicMock(choices=[mock_choice])
content, tool_calls, reason = llm.complete(
[{"role": "user", "content": "check docker"}], TOOLS
)
assert tool_calls is not None
assert len(tool_calls) == 1
assert tool_calls[0]["function"]["name"] == "check_docker"
assert tool_calls[0]["id"] == "call_123"
def test_no_content(self) -> None:
llm = _BootstrapLLM("openai", MagicMock(), "gpt-5.4")
mock_choice = MagicMock()
mock_choice.message.content = None
mock_choice.message.tool_calls = None
mock_choice.finish_reason = "stop"
llm.client.chat.completions.create.return_value = MagicMock(choices=[mock_choice])
content, tool_calls, reason = llm.complete([{"role": "user", "content": "hi"}], [])
assert content == ""
assert tool_calls is None
# ---------------------------------------------------------------------------
# Conversation loop tests
# ---------------------------------------------------------------------------
class TestConversationLoop:
def test_quit_exits(self) -> None:
"""User typing 'quit' should exit the loop."""
llm = MagicMock(spec=_BootstrapLLM)
llm.complete.return_value = ("What would you like?", None, "stop")
with patch("builtins.input", return_value="quit"):
from turnstone.bootstrap import _run_conversation
_run_conversation(llm, Path("/tmp"))
def test_tool_calls_executed(self, tmp_path: Path) -> None:
"""Tool calls should be executed and results fed back."""
llm = MagicMock(spec=_BootstrapLLM)
# First call: LLM returns a tool call
llm.complete.side_effect = [
(
"",
[
{
"id": "tc_1",
"type": "function",
"function": {"name": "generate_secret", "arguments": "{}"},
}
],
"tool_calls",
),
# Second call: LLM responds with text after seeing tool result
("Here's your secret!", None, "stop"),
]
with patch("builtins.input", return_value="quit"):
from turnstone.bootstrap import _run_conversation
_run_conversation(llm, tmp_path)
# Verify two calls were made
assert llm.complete.call_count == 2
# Verify tool result was fed back in second call's messages
second_call_messages = llm.complete.call_args_list[1][0][0]
tool_results = [m for m in second_call_messages if m.get("role") == "tool"]
assert len(tool_results) == 1
assert tool_results[0]["tool_call_id"] == "tc_1"
# Result should be a 64-char hex string
assert len(tool_results[0]["content"]) == 64
def test_empty_input_skipped(self) -> None:
"""Empty user input should be skipped."""
llm = MagicMock(spec=_BootstrapLLM)
llm.complete.return_value = ("Ask me something.", None, "stop")
call_count = 0
def mock_input(prompt: str = "") -> str:
nonlocal call_count
call_count += 1
if call_count <= 2:
return "" # Empty inputs
return "quit"
with patch("builtins.input", side_effect=mock_input):
from turnstone.bootstrap import _run_conversation
_run_conversation(llm, Path("/tmp"))
def test_finish_tool_exits_loop(self, tmp_path: Path) -> None:
"""LLM calling finish tool should exit the conversation cleanly."""
llm = MagicMock(spec=_BootstrapLLM)
llm.complete.return_value = (
"",
[
{
"id": "tc_fin",
"type": "function",
"function": {
"name": "finish",
"arguments": '{"summary": "All configured."}',
},
}
],
"tool_calls",
)
from turnstone.bootstrap import _run_conversation
# Should return without needing user input
_run_conversation(llm, tmp_path)
assert llm.complete.call_count == 1
# ---------------------------------------------------------------------------
# Interactive startup tests
# ---------------------------------------------------------------------------
class TestProviderDefaults:
def test_openai_default_model(self) -> None:
from turnstone.bootstrap import _DEFAULT_MODELS
assert _DEFAULT_MODELS["openai"] == "gpt-5.4"
def test_anthropic_default_model(self) -> None:
from turnstone.bootstrap import _DEFAULT_MODELS
assert _DEFAULT_MODELS["anthropic"] == "claude-sonnet-4-6"
class TestSelectProvider:
def test_openai_selection(self) -> None:
"""Selecting '1' should set up OpenAI."""
mock_client = MagicMock()
with (
patch("builtins.input", side_effect=["1", ""]),
patch("getpass.getpass", return_value="sk-test"),
patch("openai.OpenAI", return_value=mock_client),
):
from turnstone.bootstrap import _select_provider
provider, client, model = _select_provider()
assert provider == "openai"
assert model == "gpt-5.4"
def test_local_selection(self) -> None:
"""Selecting '3' should set up local/vLLM."""
mock_client = MagicMock()
# Ensure OPENAI_API_KEY is not in env so we hit the getpass path
env = {k: v for k, v in os.environ.items() if k != "OPENAI_API_KEY"}
with (
patch.dict("os.environ", env, clear=True),
patch("builtins.input", side_effect=["3", "http://localhost:8000/v1", "my-model"]),
patch("getpass.getpass", return_value="none"),
patch("openai.OpenAI", return_value=mock_client),
):
from turnstone.bootstrap import _select_provider
provider, client, model = _select_provider()
assert provider == "openai"
assert model == "my-model"
# ---------------------------------------------------------------------------
# System prompt and tools sanity checks
# ---------------------------------------------------------------------------
class TestConstants:
def test_system_prompt_not_empty(self) -> None:
assert len(SYSTEM_PROMPT) > 500
def test_system_prompt_mentions_turnstone(self) -> None:
assert "Turnstone" in SYSTEM_PROMPT
def test_all_tools_have_required_fields(self) -> None:
for tool in TOOLS:
assert tool["type"] == "function"
func = tool["function"]
assert "name" in func
assert "description" in func
assert "parameters" in func
assert func["parameters"]["type"] == "object"
def test_tool_count(self) -> None:
assert len(TOOLS) == 7
def test_all_tools_have_implementations(self) -> None:
from turnstone.bootstrap import TOOL_FUNCTIONS
for tool in TOOLS:
name = tool["function"]["name"]
assert name in TOOL_FUNCTIONS, f"Missing implementation for tool: {name}"
+69
View File
@@ -0,0 +1,69 @@
"""Tests for bridge event publishing — TurnCompleteEvent on idle transitions."""
from unittest.mock import MagicMock, patch
from turnstone.mq.bridge import Bridge
from turnstone.mq.protocol import StateChangeEvent, TurnCompleteEvent
def _make_bridge():
"""Create a Bridge with a mock broker (no Redis or HTTP needed)."""
broker = MagicMock()
bridge = Bridge(server_url="http://localhost:8080", broker=broker, node_id="test-node")
return bridge
class TestIdleTurnComplete:
"""TurnCompleteEvent should be emitted on every idle transition."""
def test_idle_emits_turn_complete_with_correlation_id(self):
"""Bridge-initiated turn: TurnCompleteEvent has the correlation_id."""
bridge = _make_bridge()
bridge._active_sends["ws-1"] = "cid-abc"
published = []
with patch.object(
bridge, "_publish_ws", side_effect=lambda ws, ev: published.append((ws, ev))
):
bridge._handle_global_event({"type": "ws_state", "ws_id": "ws-1", "state": "idle"})
turn_completes = [(ws, ev) for ws, ev in published if isinstance(ev, TurnCompleteEvent)]
assert len(turn_completes) == 1
ws, ev = turn_completes[0]
assert ws == "ws-1"
assert ev.correlation_id == "cid-abc"
# correlation_id should be removed from _active_sends
assert "ws-1" not in bridge._active_sends
def test_idle_emits_turn_complete_without_correlation_id(self):
"""Server-UI-initiated turn: TurnCompleteEvent has empty correlation_id."""
bridge = _make_bridge()
# No entry in _active_sends for this workstream
published = []
with patch.object(
bridge, "_publish_ws", side_effect=lambda ws, ev: published.append((ws, ev))
):
bridge._handle_global_event({"type": "ws_state", "ws_id": "ws-2", "state": "idle"})
turn_completes = [(ws, ev) for ws, ev in published if isinstance(ev, TurnCompleteEvent)]
assert len(turn_completes) == 1
ws, ev = turn_completes[0]
assert ws == "ws-2"
assert ev.correlation_id == ""
def test_non_idle_state_does_not_emit_turn_complete(self):
"""Non-idle state transitions should emit StateChangeEvent but not TurnCompleteEvent."""
bridge = _make_bridge()
published = []
with patch.object(
bridge, "_publish_ws", side_effect=lambda ws, ev: published.append((ws, ev))
):
bridge._handle_global_event({"type": "ws_state", "ws_id": "ws-3", "state": "thinking"})
state_changes = [ev for _, ev in published if isinstance(ev, StateChangeEvent)]
turn_completes = [ev for _, ev in published if isinstance(ev, TurnCompleteEvent)]
assert len(state_changes) == 1
assert state_changes[0].state == "thinking"
assert len(turn_completes) == 0
+406
View File
@@ -0,0 +1,406 @@
"""Tests for generation cancellation (cooperative cancel via threading.Event)."""
import threading
import time
from dataclasses import dataclass, field
from unittest.mock import MagicMock, patch
import pytest
from turnstone.core.session import ChatSession, GenerationCancelled
class NullUI:
"""UI adapter that records state changes and discards other output."""
def __init__(self):
self.states = []
self.infos = []
self.stream_ends = 0
def on_thinking_start(self):
pass
def on_thinking_stop(self):
pass
def on_reasoning_token(self, text):
pass
def on_content_token(self, text):
pass
def on_stream_end(self):
self.stream_ends += 1
def approve_tools(self, items):
return True, None
def on_tool_result(self, call_id, name, output):
pass
def on_tool_output_chunk(self, call_id, chunk):
pass
def on_status(self, usage, context_window, effort):
pass
def on_plan_review(self, content):
return ""
def on_info(self, message):
self.infos.append(message)
def on_error(self, message):
pass
def on_state_change(self, state):
self.states.append(state)
def on_rename(self, name):
pass
def _make_session(ui=None, **kwargs):
"""Helper to construct a ChatSession with minimal setup."""
defaults = dict(
client=MagicMock(),
model="test-model",
ui=ui or NullUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
defaults.update(kwargs)
return ChatSession(**defaults)
class TestCancelEvent:
"""Basic cancel event mechanics."""
def test_cancel_sets_event(self, tmp_db):
session = _make_session()
assert not session._cancel_event.is_set()
session.cancel()
assert session._cancel_event.is_set()
def test_check_cancelled_raises_when_set(self, tmp_db):
session = _make_session()
session.cancel()
with pytest.raises(GenerationCancelled):
session._check_cancelled()
def test_check_cancelled_noop_when_clear(self, tmp_db):
session = _make_session()
session._check_cancelled() # Should not raise
def test_cancel_is_idempotent(self, tmp_db):
session = _make_session()
session.cancel()
session.cancel() # Double call is harmless
assert session._cancel_event.is_set()
def test_cancel_event_cleared_on_send_start(self, tmp_db):
"""send() clears a stale cancel flag before starting."""
ui = NullUI()
session = _make_session(ui=ui)
session.cancel() # Set stale flag
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = "stop"
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
fake_stream = iter([FakeChunk(content_delta="Hello", finish_reason="stop")])
with (
patch.object(session, "_create_stream_with_retry", return_value=fake_stream),
patch.object(session, "_full_messages", return_value=[]),
):
session.send("test")
# Should complete normally — cancel flag was cleared
assert "idle" in ui.states
class TestCancelDuringStreaming:
"""Cancel while _stream_response is iterating chunks."""
def test_preserves_partial_content(self, tmp_db):
"""Partial content already streamed should be preserved in messages."""
ui = NullUI()
session = _make_session(ui=ui)
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = ""
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
def cancelling_stream():
"""Yield a few chunks then cancel."""
yield FakeChunk(content_delta="Hello ")
yield FakeChunk(content_delta="world")
session.cancel()
yield FakeChunk(content_delta=" — this should not appear")
with (
patch.object(session, "_create_stream_with_retry", return_value=cancelling_stream()),
patch.object(session, "_full_messages", return_value=[]),
):
session.send("test")
# Session should be idle (not error)
assert ui.states[-1] == "idle"
# Check that "[Generation cancelled]" was emitted
assert any("cancelled" in i.lower() for i in ui.infos)
# The partial content should be preserved as an assistant message
assistant_msgs = [m for m in session.messages if m["role"] == "assistant"]
assert len(assistant_msgs) == 1
assert assistant_msgs[0]["content"] == "Hello world"
# No tool_calls in the partial message
assert "tool_calls" not in assistant_msgs[0]
class TestCancelDuringToolExecution:
"""Cancel while tools are being executed."""
def test_rollback_incomplete_tool_results(self, tmp_db):
"""When cancelled during tool execution, incomplete results are rolled back."""
ui = NullUI()
session = _make_session(ui=ui)
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = ""
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
@dataclass
class FakeToolDelta:
index: int = 0
id: str = ""
name: str = ""
arguments_delta: str = ""
# First call: return content with a tool call
def stream_with_tool():
yield FakeChunk(
tool_call_deltas=[FakeToolDelta(index=0, id="tc_1", name="bash")],
finish_reason="",
)
yield FakeChunk(
tool_call_deltas=[FakeToolDelta(index=0, arguments_delta='{"command":"echo hi"}')],
finish_reason="tool_calls",
)
call_count = 0
def fake_create_stream(msgs):
nonlocal call_count
call_count += 1
if call_count == 1:
return stream_with_tool()
# Should not be called a second time since cancel happens before phase 3
raise AssertionError("Should not stream again after cancel")
def cancel_before_execute(tool_calls):
"""Simulate cancel happening before tool execution."""
session.cancel()
raise GenerationCancelled()
with (
patch.object(session, "_create_stream_with_retry", side_effect=fake_create_stream),
patch.object(session, "_full_messages", return_value=[]),
patch.object(session, "_execute_tools", side_effect=cancel_before_execute),
):
session.send("run something")
# Session should be idle
assert ui.states[-1] == "idle"
# No tool result messages should remain (rolled back)
roles = [m["role"] for m in session.messages]
assert "tool" not in roles
# The assistant message with tool_calls should also be rolled back
for m in session.messages:
if m["role"] == "assistant":
assert "tool_calls" not in m or not m["tool_calls"]
class TestCancelWhenIdle:
"""Cancelling when no generation is active is harmless."""
def test_cancel_when_idle_is_noop(self, tmp_db):
session = _make_session()
session.cancel()
# Next send should work normally (cancel cleared at start)
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = "stop"
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
fake_stream = iter([FakeChunk(content_delta="ok", finish_reason="stop")])
with (
patch.object(session, "_create_stream_with_retry", return_value=fake_stream),
patch.object(session, "_full_messages", return_value=[]),
):
session.send("hello")
# Should complete normally
assistant_msgs = [m for m in session.messages if m["role"] == "assistant"]
assert len(assistant_msgs) == 1
assert assistant_msgs[0]["content"] == "ok"
class TestCancelThreadSafety:
"""Cancel from a different thread while generation is running."""
def test_cancel_from_another_thread(self, tmp_db):
ui = NullUI()
session = _make_session(ui=ui)
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = ""
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
barrier = threading.Event()
def slow_stream():
yield FakeChunk(content_delta="Start")
barrier.set() # Signal that streaming has started
time.sleep(2) # Simulate slow streaming
yield FakeChunk(content_delta=" end", finish_reason="stop")
with (
patch.object(session, "_create_stream_with_retry", return_value=slow_stream()),
patch.object(session, "_full_messages", return_value=[]),
):
# Run send() in a thread
error = []
def run():
try:
session.send("test")
except Exception as e:
error.append(e)
t = threading.Thread(target=run)
t.start()
barrier.wait(timeout=5)
# Cancel from main thread
session.cancel()
t.join(timeout=5)
assert not error
assert ui.states[-1] == "idle"
assert any("cancelled" in i.lower() for i in ui.infos)
class TestGenerationCancelledException:
"""GenerationCancelled is a BaseException, not Exception."""
def test_is_base_exception(self):
assert issubclass(GenerationCancelled, BaseException)
def test_not_caught_by_except_exception(self):
"""Verify GenerationCancelled is NOT caught by except Exception."""
with pytest.raises(GenerationCancelled):
try:
raise GenerationCancelled()
except Exception:
pytest.fail("GenerationCancelled was caught by except Exception")
class TestStreamFlushBeforeToolCalls:
"""Content pending buffer must be flushed before tool call processing."""
def test_pending_content_flushed_before_tool_calls(self, tmp_db):
"""All content tokens arrive via on_content_token before tool calls."""
events: list[tuple[str, ...]] = []
class TrackingUI(NullUI):
def on_content_token(self, text):
events.append(("content", text))
def on_stream_end(self):
events.append(("stream_end",))
super().on_stream_end()
ui = TrackingUI()
session = _make_session(ui=ui)
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = ""
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
@dataclass
class FakeToolDelta:
index: int = 0
id: str = ""
name: str = ""
arguments_delta: str = ""
def stream_content_then_tool():
# Content long enough to leave chars in pending buffer
# (_MAX_TAG_LEN = 13, so _drain_pending retains last 13 chars)
yield FakeChunk(content_delta="Hello world, this is a test message")
yield FakeChunk(
tool_call_deltas=[FakeToolDelta(index=0, id="tc_1", name="bash")],
)
yield FakeChunk(
tool_call_deltas=[FakeToolDelta(index=0, arguments_delta='{"command":"echo hi"}')],
finish_reason="tool_calls",
)
with (
patch.object(
session,
"_create_stream_with_retry",
return_value=stream_content_then_tool(),
),
patch.object(session, "_full_messages", return_value=[]),
# Prevent real tool execution (e.g., bash) during this test.
patch.object(session, "_execute_tools", return_value=([], None)),
):
session.send("test")
# All content should have been emitted
total = "".join(e[1] for e in events if e[0] == "content")
assert total == "Hello world, this is a test message"
# No content events after stream_end
stream_end_idx = next(i for i, e in enumerate(events) if e[0] == "stream_end")
late_content = [e for e in events[stream_end_idx + 1 :] if e[0] == "content"]
assert late_content == [], f"Content after stream_end: {late_content}"
+381
View File
@@ -0,0 +1,381 @@
"""Tests for the Discord channel adapter (bot, cog, views, config, CLI)."""
from __future__ import annotations
import asyncio
import sys
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
discord = pytest.importorskip("discord")
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _run(coro):
"""Run an async coroutine in a fresh event loop (no pytest-asyncio needed)."""
return asyncio.run(coro)
def _make_message(*, bot=False, guild=True, content="hello", channel=None):
"""Build a mock ``discord.Message``."""
msg = MagicMock(spec=discord.Message)
msg.author = MagicMock()
msg.author.bot = bot
msg.author.id = 12345
msg.content = content
msg.guild = MagicMock() if guild else None
msg.channel = channel or MagicMock()
msg.mentions = []
return msg
def _make_interaction(*, footer_text=None, has_embeds=True):
"""Build a mock ``discord.Interaction``."""
interaction = MagicMock(spec=discord.Interaction)
interaction.user = MagicMock()
interaction.user.id = 67890
interaction.response = MagicMock()
interaction.response.send_message = AsyncMock()
if has_embeds and footer_text is not None:
embed = MagicMock()
embed.footer.text = footer_text
interaction.message = MagicMock()
interaction.message.embeds = [embed]
elif not has_embeds:
interaction.message = MagicMock()
interaction.message.embeds = []
else:
interaction.message = None
return interaction
# ---------------------------------------------------------------------------
# DiscordConfig
# ---------------------------------------------------------------------------
class TestDiscordConfig:
"""Tests for DiscordConfig default and custom values."""
def test_defaults(self):
from turnstone.channels.discord.config import DiscordConfig
cfg = DiscordConfig()
assert cfg.bot_token == ""
assert cfg.guild_id == 0
assert cfg.allowed_channels == []
assert cfg.thread_auto_archive == 1440
assert cfg.max_message_length == 2000
assert cfg.streaming_edit_interval == 1.5
# Inherited from ChannelConfig
assert cfg.redis_host == "localhost"
assert cfg.redis_port == 6379
assert cfg.model == ""
assert cfg.auto_approve is False
def test_custom_values(self):
from turnstone.channels.discord.config import DiscordConfig
cfg = DiscordConfig(
bot_token="tok_123",
guild_id=999,
allowed_channels=[1, 2, 3],
thread_auto_archive=60,
max_message_length=4000,
streaming_edit_interval=0.5,
model="gpt-5",
auto_approve=True,
)
assert cfg.bot_token == "tok_123"
assert cfg.guild_id == 999
assert cfg.allowed_channels == [1, 2, 3]
assert cfg.thread_auto_archive == 60
assert cfg.max_message_length == 4000
assert cfg.streaming_edit_interval == 0.5
assert cfg.model == "gpt-5"
assert cfg.auto_approve is True
# ---------------------------------------------------------------------------
# StreamingMessage
# ---------------------------------------------------------------------------
class TestStreamingMessage:
"""Tests for the StreamingMessage helper in bot.py."""
def test_append_accumulates(self):
from turnstone.channels.discord.bot import StreamingMessage
channel = MagicMock()
channel.send = AsyncMock()
sm = StreamingMessage(channel=channel, edit_interval=999.0)
_run(sm.append("hello "))
_run(sm.append("world"))
assert "".join(sm._buffer) == "hello world"
def test_finalize_sends_when_no_prior_message(self):
from turnstone.channels.discord.bot import StreamingMessage
channel = MagicMock()
channel.send = AsyncMock()
sm = StreamingMessage(channel=channel, edit_interval=999.0)
_run(sm.append("hello"))
_run(sm.finalize())
channel.send.assert_awaited_once_with("hello")
def test_finalize_edits_existing_message(self):
from turnstone.channels.discord.bot import StreamingMessage
channel = MagicMock()
sent_msg = MagicMock()
sent_msg.edit = AsyncMock()
channel.send = AsyncMock(return_value=sent_msg)
sm = StreamingMessage(channel=channel, edit_interval=0.0)
# First append triggers flush (interval=0) which creates the message.
_run(sm.append("hi"))
assert sm._message is sent_msg
_run(sm.append(" there"))
_run(sm.finalize())
# finalize edits the existing message with full content.
sent_msg.edit.assert_awaited_with(content="hi there")
def test_finalize_chunks_long_content(self):
from turnstone.channels.discord.bot import StreamingMessage
channel = MagicMock()
channel.send = AsyncMock()
sm = StreamingMessage(channel=channel, max_length=10, edit_interval=999.0)
# Content longer than max_length should be chunked on finalize.
_run(sm.append("a" * 25))
_run(sm.finalize())
# Should have sent multiple chunks via channel.send.
assert channel.send.await_count >= 2
def test_finalize_empty_is_noop(self):
from turnstone.channels.discord.bot import StreamingMessage
channel = MagicMock()
channel.send = AsyncMock()
sm = StreamingMessage(channel=channel)
_run(sm.finalize())
channel.send.assert_not_awaited()
# ---------------------------------------------------------------------------
# MessageCog._on_message
# ---------------------------------------------------------------------------
class TestMessageCog:
"""Tests for the MessageCog on_message filtering logic."""
def _make_cog(self):
"""Build a MessageCog with a fully mocked bot and TurnstoneBot."""
from turnstone.channels.discord.cog import MessageCog
bot = MagicMock()
bot.user = MagicMock()
bot.user.id = 99999
bot.user.mentioned_in = MagicMock(return_value=False)
ts = MagicMock()
ts._is_allowed_channel = MagicMock(return_value=True)
ts.storage = MagicMock()
ts.router = MagicMock()
ts.router.resolve_user = AsyncMock(return_value="u_abc")
ts.router.send_message = AsyncMock()
ts.config = MagicMock()
ts._ws_tasks = {}
bot.turnstone = ts
cog = MessageCog(bot)
return cog, ts, bot
def test_ignores_bot_messages(self):
cog, ts, _bot = self._make_cog()
msg = _make_message(bot=True)
_run(cog._on_message(msg))
# No router interaction means the message was ignored.
ts.router.send_message.assert_not_awaited()
def test_ignores_own_messages(self):
cog, ts, bot = self._make_cog()
msg = _make_message(bot=False)
msg.author = bot.user # message from ourselves
_run(cog._on_message(msg))
ts.router.send_message.assert_not_awaited()
def test_ignores_dms(self):
cog, ts, _bot = self._make_cog()
msg = _make_message(guild=False)
_run(cog._on_message(msg))
ts.router.send_message.assert_not_awaited()
def test_ignores_non_allowed_channels(self):
cog, ts, _bot = self._make_cog()
ts._is_allowed_channel = MagicMock(return_value=False)
thread = MagicMock(spec=discord.Thread)
thread.id = 111
thread.parent_id = 222
msg = _make_message(channel=thread)
_run(cog._on_message(msg))
ts.router.send_message.assert_not_awaited()
# ---------------------------------------------------------------------------
# _parse_footer (views.py)
# ---------------------------------------------------------------------------
class TestParseFooter:
"""Tests for _parse_footer in views.py."""
def test_valid_footer(self):
from turnstone.channels.discord.views import _parse_footer
interaction = _make_interaction(footer_text="ws_abc|corr_123")
result = _parse_footer(interaction)
assert result == ("ws_abc", "corr_123")
def test_footer_with_pipe_in_correlation(self):
from turnstone.channels.discord.views import _parse_footer
interaction = _make_interaction(footer_text="ws_abc|corr|extra")
result = _parse_footer(interaction)
# split("|", 1) means the second part includes everything after first pipe.
assert result == ("ws_abc", "corr|extra")
def test_no_message_returns_none(self):
from turnstone.channels.discord.views import _parse_footer
interaction = MagicMock()
interaction.message = None
assert _parse_footer(interaction) is None
def test_no_embeds_returns_none(self):
from turnstone.channels.discord.views import _parse_footer
interaction = _make_interaction(has_embeds=False)
assert _parse_footer(interaction) is None
def test_empty_footer_returns_none(self):
from turnstone.channels.discord.views import _parse_footer
# Build an interaction whose embed has footer.text = None.
interaction = MagicMock(spec=discord.Interaction)
embed = MagicMock()
embed.footer.text = None
interaction.message = MagicMock()
interaction.message.embeds = [embed]
assert _parse_footer(interaction) is None
def test_footer_without_pipe_returns_none(self):
from turnstone.channels.discord.views import _parse_footer
interaction = _make_interaction(footer_text="no_pipe_here")
# footer text has no "|" separator
embed = MagicMock()
embed.footer.text = "no_pipe_here"
interaction.message.embeds = [embed]
assert _parse_footer(interaction) is None
# ---------------------------------------------------------------------------
# CLI main() — no adapter configured
# ---------------------------------------------------------------------------
class TestWsEventFinalization:
"""TurnCompleteEvent should finalize streaming messages in the Discord bot."""
def test_turn_complete_finalizes_streaming(self):
"""ContentEvent + TurnCompleteEvent(correlation_id='') finalizes the message."""
from turnstone.channels.discord.bot import TurnstoneBot
from turnstone.mq.protocol import ContentEvent, TurnCompleteEvent
bot = MagicMock(spec=TurnstoneBot)
bot.config = MagicMock()
bot.config.max_message_length = 2000
bot.config.streaming_edit_interval = 1.5
bot.config.auto_approve = False
bot.config.auto_approve_tools = []
bot._streaming = {}
# Use the real _on_ws_event method
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
thread = AsyncMock()
# Feed content event
content_raw = ContentEvent(ws_id="ws-1", text="Hello world").to_json()
_run(bot._on_ws_event("ws-1", thread, content_raw))
# StreamingMessage should exist
assert "ws-1" in bot._streaming
# Feed turn complete with empty correlation_id (server-UI-initiated)
complete_raw = TurnCompleteEvent(ws_id="ws-1", correlation_id="").to_json()
_run(bot._on_ws_event("ws-1", thread, complete_raw))
# StreamingMessage should be removed and finalized
assert "ws-1" not in bot._streaming
def test_turn_complete_no_streaming_is_noop(self):
"""TurnCompleteEvent without prior content should not error."""
from turnstone.channels.discord.bot import TurnstoneBot
from turnstone.mq.protocol import TurnCompleteEvent
bot = MagicMock(spec=TurnstoneBot)
bot._streaming = {}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
thread = AsyncMock()
complete_raw = TurnCompleteEvent(ws_id="ws-1", correlation_id="").to_json()
_run(bot._on_ws_event("ws-1", thread, complete_raw))
# No error, no streaming message
assert "ws-1" not in bot._streaming
class TestChannelCLI:
"""Tests for the channel CLI entry point."""
def test_exits_without_adapter_token(self):
from turnstone.channels.cli import main
with (
patch.object(sys, "argv", ["turnstone-channel"]),
patch.dict("os.environ", {}, clear=True),
pytest.raises(SystemExit) as exc_info,
):
main()
assert exc_info.value.code == 1
+203
View File
@@ -0,0 +1,203 @@
"""Tests for turnstone.channels._protocol and turnstone.channels._formatter."""
from __future__ import annotations
from turnstone.channels._formatter import (
chunk_message,
format_approval_request,
format_plan_review,
truncate,
)
from turnstone.channels._protocol import ChannelEvent
# ---------------------------------------------------------------------------
# ChannelEvent
# ---------------------------------------------------------------------------
class TestChannelEvent:
def test_construction(self) -> None:
evt = ChannelEvent(
channel_type="discord",
channel_id="ch-1",
channel_user_id="u-42",
message="hello",
parent_channel_id="parent",
metadata={"key": "val"},
)
assert evt.channel_type == "discord"
assert evt.channel_id == "ch-1"
assert evt.channel_user_id == "u-42"
assert evt.message == "hello"
assert evt.parent_channel_id == "parent"
assert evt.metadata == {"key": "val"}
def test_defaults(self) -> None:
evt = ChannelEvent(
channel_type="slack",
channel_id="ch-2",
channel_user_id="u-7",
message="hi",
)
assert evt.parent_channel_id == ""
assert evt.metadata == {}
def test_metadata_independence(self) -> None:
"""Default metadata dicts are independent across instances."""
a = ChannelEvent(channel_type="x", channel_id="1", channel_user_id="u", message="m")
b = ChannelEvent(channel_type="x", channel_id="2", channel_user_id="u", message="m")
a.metadata["key"] = "val"
assert "key" not in b.metadata
# ---------------------------------------------------------------------------
# chunk_message
# ---------------------------------------------------------------------------
class TestChunkMessage:
def test_empty_string(self) -> None:
assert chunk_message("") == [""]
def test_under_limit(self) -> None:
assert chunk_message("short text", max_length=100) == ["short text"]
def test_exactly_at_limit(self) -> None:
text = "a" * 50
assert chunk_message(text, max_length=50) == [text]
def test_splits_at_newline(self) -> None:
text = "line one\nline two\nline three"
chunks = chunk_message(text, max_length=18)
assert len(chunks) >= 2
# The split should happen at a newline boundary within the text.
# Reassembled chunks (with newline separators) should cover all content.
rejoined = "\n".join(chunks)
assert "line one" in rejoined
assert "line three" in rejoined
def test_splits_at_word_boundary(self) -> None:
text = "word1 word2 word3 word4"
chunks = chunk_message(text, max_length=12)
assert len(chunks) >= 2
# No chunk should start with a space (lstrip handles newlines).
for chunk in chunks:
assert not chunk.startswith("\n")
def test_hard_splits(self) -> None:
text = "a" * 30
chunks = chunk_message(text, max_length=10)
assert len(chunks) == 3
assert "".join(chunks) == text
def test_code_block_spanning_boundary(self) -> None:
text = "before\n```\ncode line 1\ncode line 2\ncode line 3\n```\nafter"
chunks = chunk_message(text, max_length=30)
assert len(chunks) >= 2
# If a chunk opens a code block without closing it, the chunker
# should close it and reopen in the next chunk.
for chunk in chunks:
fence_count = chunk.count("```")
assert fence_count % 2 == 0, f"Unmatched code fence in chunk: {chunk!r}"
def test_multiple_code_blocks(self) -> None:
text = "```\nblock1\n```\ntext\n```\nblock2\n```"
chunks = chunk_message(text, max_length=20)
for chunk in chunks:
fence_count = chunk.count("```")
assert fence_count % 2 == 0, f"Unmatched code fence in chunk: {chunk!r}"
def test_custom_max_length(self) -> None:
text = "hello world"
chunks = chunk_message(text, max_length=5)
assert len(chunks) >= 2
assert chunks[0] == "hello"
def test_very_long_single_line(self) -> None:
text = "x" * 5000
chunks = chunk_message(text, max_length=2000)
assert len(chunks) == 3
total = "".join(chunks)
assert total == text
# ---------------------------------------------------------------------------
# format_approval_request
# ---------------------------------------------------------------------------
class TestFormatApprovalRequest:
def test_single_tool(self) -> None:
items = [{"function": {"name": "read_file", "arguments": "/etc/hosts"}}]
result = format_approval_request(items)
assert "Tool approval required" in result
assert "`read_file`" in result
def test_multiple_tools(self) -> None:
items = [
{"function": {"name": "tool_a", "arguments": "arg1"}},
{"function": {"name": "tool_b", "arguments": "arg2"}},
]
result = format_approval_request(items)
assert "`tool_a`" in result
assert "`tool_b`" in result
def test_long_arguments_truncated(self) -> None:
long_args = "x" * 500
items = [{"function": {"name": "fn", "arguments": long_args}}]
result = format_approval_request(items)
# The result should be shorter than the original args.
assert len(result) < 500
def test_server_sse_format(self) -> None:
"""Items from the server SSE use func_name/preview, not function.name."""
items = [
{
"call_id": "c1",
"func_name": "bash",
"preview": "ls -la",
"header": "Execute: ls -la",
"needs_approval": True,
}
]
result = format_approval_request(items)
assert "`bash`" in result
assert "Execute: ls -la" in result
def test_server_sse_format_no_header(self) -> None:
items = [{"func_name": "read_file", "preview": "/etc/hosts"}]
result = format_approval_request(items)
assert "`read_file`" in result
assert "/etc/hosts" in result
# ---------------------------------------------------------------------------
# format_plan_review
# ---------------------------------------------------------------------------
class TestFormatPlanReview:
def test_format(self) -> None:
result = format_plan_review("Step 1: do stuff")
assert result.startswith("**Plan review requested:**")
assert "Step 1: do stuff" in result
# ---------------------------------------------------------------------------
# truncate
# ---------------------------------------------------------------------------
class TestTruncate:
def test_short_text_unchanged(self) -> None:
assert truncate("hello", max_length=200) == "hello"
def test_long_text_truncated(self) -> None:
text = "a" * 300
result = truncate(text, max_length=200)
assert len(result) == 200
assert result.endswith("\u2026")
def test_exactly_at_limit(self) -> None:
text = "b" * 200
assert truncate(text, max_length=200) == text
+107
View File
@@ -0,0 +1,107 @@
"""Tests for turnstone.channels._routing.ChannelRouter."""
from __future__ import annotations
import json
from unittest.mock import AsyncMock, MagicMock
import pytest
from turnstone.channels._routing import ChannelRouter
@pytest.fixture
def mock_broker() -> AsyncMock:
"""Return a mock AsyncRedisBroker."""
broker = AsyncMock()
broker._prefix = "test"
broker.push_inbound = AsyncMock()
broker.push_response = AsyncMock()
broker.subscribe = AsyncMock()
broker.unsubscribe = AsyncMock()
return broker
@pytest.fixture
def mock_storage() -> MagicMock:
"""Return a mock StorageBackend."""
storage = MagicMock()
storage.get_channel_user = MagicMock(return_value=None)
storage.get_channel_route = MagicMock(return_value=None)
storage.get_channel_route_by_ws = MagicMock(return_value=None)
storage.create_channel_route = MagicMock()
storage.delete_channel_route = MagicMock(return_value=True)
return storage
@pytest.fixture
def router(mock_broker: AsyncMock, mock_storage: MagicMock) -> ChannelRouter:
return ChannelRouter(broker=mock_broker, storage=mock_storage)
class TestResolveUser:
@pytest.mark.anyio
async def test_linked_user(self, router: ChannelRouter, mock_storage: MagicMock) -> None:
mock_storage.get_channel_user.return_value = {"user_id": "usr-1", "channel_user_id": "d-42"}
result = await router.resolve_user("discord", "d-42")
assert result == "usr-1"
mock_storage.get_channel_user.assert_called_once_with("discord", "d-42")
@pytest.mark.anyio
async def test_unlinked_user(self, router: ChannelRouter, mock_storage: MagicMock) -> None:
mock_storage.get_channel_user.return_value = None
result = await router.resolve_user("slack", "s-99")
assert result is None
class TestSendMessage:
@pytest.mark.anyio
async def test_pushes_send_message(self, router: ChannelRouter, mock_broker: AsyncMock) -> None:
cid = await router.send_message("ws-1", "hello world")
assert isinstance(cid, str)
assert len(cid) > 0
mock_broker.push_inbound.assert_awaited_once()
raw = mock_broker.push_inbound.call_args[0][0]
payload = json.loads(raw)
assert payload["type"] == "send"
assert payload["ws_id"] == "ws-1"
assert payload["message"] == "hello world"
assert payload["correlation_id"] == cid
class TestSendApproval:
@pytest.mark.anyio
async def test_pushes_to_response_queue(
self, router: ChannelRouter, mock_broker: AsyncMock
) -> None:
await router.send_approval("ws-1", "corr-abc", approved=True, feedback="ok")
mock_broker.push_response.assert_awaited_once()
queue_name = mock_broker.push_response.call_args[0][0]
assert queue_name == "corr-abc"
raw = mock_broker.push_response.call_args[0][1]
payload = json.loads(raw)
assert payload["type"] == "approve"
assert payload["approved"] is True
assert payload["ws_id"] == "ws-1"
class TestSendPlanFeedback:
@pytest.mark.anyio
async def test_pushes_to_response_queue(
self, router: ChannelRouter, mock_broker: AsyncMock
) -> None:
await router.send_plan_feedback("ws-2", "corr-xyz", "looks good")
mock_broker.push_response.assert_awaited_once()
raw = mock_broker.push_response.call_args[0][1]
payload = json.loads(raw)
assert payload["type"] == "plan_feedback"
assert payload["feedback"] == "looks good"
class TestDeleteRoute:
@pytest.mark.anyio
async def test_calls_storage_delete(
self, router: ChannelRouter, mock_storage: MagicMock
) -> None:
await router.delete_route("discord", "ch-123")
mock_storage.delete_channel_route.assert_called_once_with("discord", "ch-123")
+136
View File
@@ -0,0 +1,136 @@
"""Tests for channel_users and channel_routes storage CRUD."""
from __future__ import annotations
import pytest
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def db(tmp_path):
"""Fresh SQLite backend for each test."""
backend = SQLiteBackend(str(tmp_path / "test.db"))
return backend
class TestChannelUserCRUD:
"""Tests for channel_users table operations."""
def test_create_and_get(self, db):
db.create_channel_user("discord", "12345", "u_abc")
result = db.get_channel_user("discord", "12345")
assert result is not None
assert result["channel_type"] == "discord"
assert result["channel_user_id"] == "12345"
assert result["user_id"] == "u_abc"
assert "created" in result
def test_get_nonexistent(self, db):
assert db.get_channel_user("discord", "99999") is None
def test_create_duplicate_noop(self, db):
db.create_channel_user("discord", "12345", "u_abc")
db.create_channel_user("discord", "12345", "u_different")
result = db.get_channel_user("discord", "12345")
assert result is not None
assert result["user_id"] == "u_abc" # first write wins
def test_same_user_different_channels(self, db):
db.create_channel_user("discord", "d_123", "u_abc")
db.create_channel_user("slack", "s_456", "u_abc")
d = db.get_channel_user("discord", "d_123")
s = db.get_channel_user("slack", "s_456")
assert d is not None and d["user_id"] == "u_abc"
assert s is not None and s["user_id"] == "u_abc"
def test_list_by_user(self, db):
db.create_channel_user("discord", "d_123", "u_abc")
db.create_channel_user("slack", "s_456", "u_abc")
db.create_channel_user("discord", "d_999", "u_other")
results = db.list_channel_users_by_user("u_abc")
assert len(results) == 2
types = {r["channel_type"] for r in results}
assert types == {"discord", "slack"}
def test_list_by_user_empty(self, db):
assert db.list_channel_users_by_user("u_nobody") == []
def test_delete(self, db):
db.create_channel_user("discord", "12345", "u_abc")
assert db.delete_channel_user("discord", "12345") is True
assert db.get_channel_user("discord", "12345") is None
def test_delete_nonexistent(self, db):
assert db.delete_channel_user("discord", "99999") is False
def test_delete_user_cascades_channel_users(self, db):
"""Deleting a turnstone user should cascade to channel_users."""
db.create_user("u_abc", "admin", "Admin", "hash123")
db.create_channel_user("discord", "12345", "u_abc")
db.delete_user("u_abc")
assert db.get_channel_user("discord", "12345") is None
class TestChannelRouteCRUD:
"""Tests for channel_routes table operations."""
def test_create_and_get(self, db):
db.create_channel_route("discord", "thread_123", "ws_abc", "node_1")
result = db.get_channel_route("discord", "thread_123")
assert result is not None
assert result["channel_type"] == "discord"
assert result["channel_id"] == "thread_123"
assert result["ws_id"] == "ws_abc"
assert result["node_id"] == "node_1"
assert "created" in result
def test_get_nonexistent(self, db):
assert db.get_channel_route("discord", "thread_999") is None
def test_create_duplicate_noop(self, db):
db.create_channel_route("discord", "thread_123", "ws_abc")
db.create_channel_route("discord", "thread_123", "ws_different")
result = db.get_channel_route("discord", "thread_123")
assert result is not None
assert result["ws_id"] == "ws_abc" # first write wins
def test_default_empty_node_id(self, db):
db.create_channel_route("discord", "thread_123", "ws_abc")
result = db.get_channel_route("discord", "thread_123")
assert result is not None
assert result["node_id"] == ""
def test_get_by_ws(self, db):
db.create_channel_route("discord", "thread_123", "ws_abc", "node_1")
result = db.get_channel_route_by_ws("ws_abc")
assert result is not None
assert result["channel_id"] == "thread_123"
assert result["ws_id"] == "ws_abc"
def test_get_by_ws_nonexistent(self, db):
assert db.get_channel_route_by_ws("ws_nobody") is None
def test_delete(self, db):
db.create_channel_route("discord", "thread_123", "ws_abc")
assert db.delete_channel_route("discord", "thread_123") is True
assert db.get_channel_route("discord", "thread_123") is None
def test_delete_nonexistent(self, db):
assert db.delete_channel_route("discord", "thread_999") is False
def test_multiple_channels_same_type(self, db):
db.create_channel_route("discord", "thread_1", "ws_1")
db.create_channel_route("discord", "thread_2", "ws_2")
r1 = db.get_channel_route("discord", "thread_1")
r2 = db.get_channel_route("discord", "thread_2")
assert r1 is not None and r1["ws_id"] == "ws_1"
assert r2 is not None and r2["ws_id"] == "ws_2"
def test_different_channel_types(self, db):
db.create_channel_route("discord", "thread_1", "ws_1")
db.create_channel_route("slack", "channel_1", "ws_2")
d = db.get_channel_route("discord", "thread_1")
s = db.get_channel_route("slack", "channel_1")
assert d is not None and d["ws_id"] == "ws_1"
assert s is not None and s["ws_id"] == "ws_2"
+376
View File
@@ -1,5 +1,6 @@
"""Tests for turnstone.console — collector and HTTP server."""
import asyncio
import json
import queue
from unittest.mock import MagicMock
@@ -201,6 +202,68 @@ class TestCollectorPolling:
# Should not raise
c._apply_poll("unknown", _dashboard_response(), {})
def test_apply_poll_emits_ws_created_for_new_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
dashboard = _dashboard_response(
workstreams=[{"id": "ws1", "name": "new-task", "state": "idle"}]
)
c._apply_poll("node-a", dashboard, {})
event = q.get_nowait()
assert event["type"] == "ws_created"
assert event["ws_id"] == "ws1"
assert event["name"] == "new-task"
assert event["node_id"] == "node-a"
def test_apply_poll_emits_ws_closed_for_removed_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "old", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
c._apply_poll("node-a", _dashboard_response(), {})
event = q.get_nowait()
assert event["type"] == "ws_closed"
assert event["ws_id"] == "ws1"
def test_apply_poll_no_events_when_unchanged(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"ws1": {"id": "ws1", "name": "same", "state": "idle"}},
)
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
dashboard = _dashboard_response(
workstreams=[{"id": "ws1", "name": "same", "state": "running"}]
)
c._apply_poll("node-a", dashboard, {})
assert q.empty()
def test_apply_poll_skips_empty_id_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
q: queue.Queue[dict] = queue.Queue()
c.register_listener(q)
dashboard = _dashboard_response(workstreams=[{"name": "no-id", "state": "idle"}])
c._apply_poll("node-a", dashboard, {})
assert q.empty()
assert len(c._nodes["node-a"].workstreams) == 0
class TestCollectorEvents:
"""Real-time event handling from cluster channel."""
@@ -445,6 +508,44 @@ class TestCollectorQueries:
def test_get_node_detail_not_found(self, populated_collector):
assert populated_collector.get_node_detail("nonexistent") is None
def test_get_snapshot_empty(self):
c = _make_collector()
snap = c.get_snapshot()
assert snap["nodes"] == []
assert snap["overview"]["nodes"] == 0
assert snap["overview"]["workstreams"] == 0
assert snap["overview"]["states"]["running"] == 0
assert "timestamp" in snap
def test_get_snapshot_with_nodes(self, populated_collector):
snap = populated_collector.get_snapshot()
assert len(snap["nodes"]) == 2
assert snap["overview"]["nodes"] == 2
assert snap["overview"]["workstreams"] == 3
assert snap["overview"]["states"]["running"] == 1
assert snap["overview"]["states"]["attention"] == 1
assert snap["overview"]["states"]["idle"] == 1
assert snap["overview"]["aggregate"]["total_tokens"] == 17000
assert snap["timestamp"] > 0
# Each node should embed its workstreams
node_ids = {n["node_id"] for n in snap["nodes"]}
assert node_ids == {"node-a", "node-b"}
for n in snap["nodes"]:
if n["node_id"] == "node-a":
assert len(n["workstreams"]) == 2
elif n["node_id"] == "node-b":
assert len(n["workstreams"]) == 1
def test_get_snapshot_consistency(self, populated_collector):
"""Snapshot overview should match get_overview()."""
snap = populated_collector.get_snapshot()
overview = populated_collector.get_overview()
assert snap["overview"]["nodes"] == overview["nodes"]
assert snap["overview"]["workstreams"] == overview["workstreams"]
assert snap["overview"]["states"] == overview["states"]
assert snap["overview"]["aggregate"] == overview["aggregate"]
assert snap["overview"]["version_drift"] == overview["version_drift"]
# ---------------------------------------------------------------------------
# ClusterStateEvent protocol tests
@@ -535,6 +636,31 @@ class TestConsoleHTTPEndpoints:
"workstreams": [],
"aggregate": {},
}
collector.get_snapshot.return_value = {
"nodes": [
{
"node_id": "node-a",
"server_url": "http://a:8080",
"max_ws": 10,
"reachable": True,
"version": "0.5.0",
"health": {},
"aggregate": {"total_tokens": 50000, "total_tool_calls": 200},
"workstreams": [
{"id": "ws1", "name": "test", "state": "running", "node": "node-a"},
],
},
],
"overview": {
"nodes": 3,
"workstreams": 15,
"states": {"running": 5, "thinking": 2, "attention": 1, "idle": 6, "error": 1},
"aggregate": {"total_tokens": 50000, "total_tool_calls": 200},
"version_drift": False,
"versions": ["0.5.0"],
},
"timestamp": 1234567890.0,
}
return collector
@pytest.fixture()
@@ -614,6 +740,16 @@ class TestConsoleHTTPEndpoints:
assert status == 404
assert "error" in data
def test_get_snapshot(self, client, mock_collector):
status, data = self._get(client, "/v1/api/cluster/snapshot")
assert status == 200
assert len(data["nodes"]) == 1
assert data["nodes"][0]["node_id"] == "node-a"
assert data["overview"]["nodes"] == 3
assert data["overview"]["workstreams"] == 15
assert data["timestamp"] == 1234567890.0
mock_collector.get_snapshot.assert_called_once()
def test_health_endpoint(self, client, mock_collector):
status, data = self._get(client, "/health")
assert status == 200
@@ -1299,3 +1435,243 @@ class TestProxySharedStatic:
resp = client.get("/node/unknown/shared/base.css")
assert resp.status_code == 404
client.close()
# ---------------------------------------------------------------------------
# SSE proxy — raw byte passthrough
# ---------------------------------------------------------------------------
class TestSSEProxy:
"""Verify _proxy_sse forwards raw bytes including ping comments."""
def test_proxy_sse_preserves_pings_and_events(self):
"""SSE proxy should forward ping comments and events verbatim."""
from turnstone.console.server import _proxy_sse
# Simulate an upstream SSE response with a ping comment and a real event
sse_payload = b': ping - 2026-03-08T12:00:00Z\n\nevent: message\ndata: {"type": "test"}\n\n'
class FakeResponse:
status_code = 200
headers = {"content-type": "text/event-stream"}
async def aiter_bytes(self):
yield sse_payload
async def aclose(self):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeClient:
def stream(self, method, url, **kwargs):
return FakeResponse()
class FakeRequest:
class url: # noqa: N801
query = "ws_id=test123"
class app: # noqa: N801
class state: # noqa: N801
proxy_sse_client = FakeClient()
proxy_auth_token = ""
headers = {}
async def is_disconnected(self):
return False
async def _run():
response = await _proxy_sse(
FakeRequest(), "http://fake:8080", "events", api_prefix="v1/api"
)
assert response.media_type == "text/event-stream"
# Collect the streamed bytes
chunks: list[bytes] = []
async for chunk in response.body_iterator:
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
body = b"".join(chunks)
# Ping comment must be preserved (not filtered)
assert b": ping" in body
# Real event must be preserved
assert b"event: message" in body
assert b'"type": "test"' in body
asyncio.run(_run())
def test_proxy_sse_upstream_error_status(self):
"""Non-200 upstream status should yield an error event."""
from turnstone.console.server import _proxy_sse
class FakeResponse:
status_code = 502
async def aiter_bytes(self):
return
yield # make it an async generator
async def aclose(self):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeClient:
def stream(self, method, url, **kwargs):
return FakeResponse()
class FakeRequest:
class url: # noqa: N801
query = ""
class app: # noqa: N801
class state: # noqa: N801
proxy_sse_client = FakeClient()
proxy_auth_token = ""
headers = {}
async def is_disconnected(self):
return False
async def _run():
response = await _proxy_sse(FakeRequest(), "http://fake:8080", "events")
chunks: list[bytes] = []
async for chunk in response.body_iterator:
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
body = b"".join(chunks)
assert b"event: error" in body
assert b"502" in body
asyncio.run(_run())
def test_proxy_sse_disconnect_handling(self):
"""Proxy should stop when browser disconnects."""
from turnstone.console.server import _proxy_sse
class FakeResponse:
status_code = 200
async def aiter_bytes(self):
yield b"data: chunk1\n\n"
yield b"data: chunk2\n\n" # should not be reached
yield b"data: chunk3\n\n"
async def aclose(self):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeClient:
def stream(self, method, url, **kwargs):
return FakeResponse()
call_count = 0
class FakeRequest:
class url: # noqa: N801
query = ""
class app: # noqa: N801
class state: # noqa: N801
proxy_sse_client = FakeClient()
proxy_auth_token = ""
headers = {}
async def is_disconnected(self):
nonlocal call_count
call_count += 1
return call_count > 1 # disconnect after first chunk
async def _run():
response = await _proxy_sse(FakeRequest(), "http://fake:8080", "events")
chunks: list[bytes] = []
async for chunk in response.body_iterator:
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
body = b"".join(chunks)
assert b"chunk1" in body
# Should have stopped before chunk3
assert b"chunk3" not in body
asyncio.run(_run())
# ---------------------------------------------------------------------------
# Collector — MCP aggregation in get_overview()
# ---------------------------------------------------------------------------
class TestCollectorMCPAggregation:
"""Verify MCP server/resource/prompt aggregation in overview and snapshot."""
def test_overview_mcp_aggregation(self):
"""Two nodes with MCP data produce correct sums in the overview."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
health={"mcp": {"servers": 2, "resources": 5, "prompts": 3}},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
health={"mcp": {"servers": 1, "resources": 4, "prompts": 2}},
)
overview = c.get_overview()
assert overview["mcp_servers"] == 3
assert overview["mcp_resources"] == 9
assert overview["mcp_prompts"] == 5
def test_overview_mcp_absent_when_zero(self):
"""Nodes without MCP data produce no mcp_servers key in the overview."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
health={"status": "ok"},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
health={},
)
overview = c.get_overview()
assert "mcp_servers" not in overview
assert "mcp_resources" not in overview
assert "mcp_prompts" not in overview
def test_overview_mcp_mixed_nodes(self):
"""One node with MCP, one without — only the MCP node contributes."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
health={"mcp": {"servers": 3, "resources": 10, "prompts": 7}},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
health={"status": "ok"},
)
overview = c.get_overview()
assert overview["mcp_servers"] == 3
assert overview["mcp_resources"] == 10
assert overview["mcp_prompts"] == 7
+761
View File
@@ -0,0 +1,761 @@
"""Tests for governance admin API endpoints (roles, orgs, policies, templates, usage, audit)."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
from turnstone.console.server import (
admin_assign_role,
admin_audit,
admin_create_policy,
admin_create_role,
admin_create_template,
admin_delete_policy,
admin_delete_role,
admin_delete_template,
admin_delete_user,
admin_get_org,
admin_list_orgs,
admin_list_policies,
admin_list_roles,
admin_list_templates,
admin_list_user_roles,
admin_unassign_role,
admin_update_org,
admin_update_policy,
admin_update_role,
admin_update_template,
admin_usage,
)
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Auth bypass middleware — injects a full-access AuthResult on every request.
# ---------------------------------------------------------------------------
class _InjectAuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="test-admin",
scopes=frozenset({"approve"}),
token_source="config",
permissions=frozenset(
{
"read",
"write",
"approve",
"admin.roles",
"admin.users",
"admin.orgs",
"admin.policies",
"admin.templates",
"admin.usage",
"admin.audit",
"admin.schedules",
"admin.watches",
"tools.approve",
"workstreams.create",
"workstreams.close",
}
),
)
return await call_next(request)
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture
def storage(tmp_path):
"""Fresh SQLite backend for each test, seeded with test users."""
backend = SQLiteBackend(str(tmp_path / "test.db"))
# Seed users required by role assignment tests
backend.create_user("test-admin", "testadmin", "Test Admin", "hash")
backend.create_user("user-1", "user1", "User One", "hash")
return backend
@pytest.fixture
def client(storage):
"""TestClient with storage and auth bypassed."""
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
# Roles
Route("/api/admin/roles", admin_list_roles),
Route("/api/admin/roles", admin_create_role, methods=["POST"]),
Route("/api/admin/roles/{role_id}", admin_update_role, methods=["PUT"]),
Route("/api/admin/roles/{role_id}", admin_delete_role, methods=["DELETE"]),
# Users
Route(
"/api/admin/users/{user_id}",
admin_delete_user,
methods=["DELETE"],
),
# User-role assignments
Route("/api/admin/users/{user_id}/roles", admin_list_user_roles),
Route(
"/api/admin/users/{user_id}/roles",
admin_assign_role,
methods=["POST"],
),
Route(
"/api/admin/users/{user_id}/roles/{role_id}",
admin_unassign_role,
methods=["DELETE"],
),
# Orgs
Route("/api/admin/orgs", admin_list_orgs),
Route("/api/admin/orgs/{org_id}", admin_get_org),
Route("/api/admin/orgs/{org_id}", admin_update_org, methods=["PUT"]),
# Policies
Route("/api/admin/policies", admin_list_policies),
Route("/api/admin/policies", admin_create_policy, methods=["POST"]),
Route(
"/api/admin/policies/{policy_id}",
admin_update_policy,
methods=["PUT"],
),
Route(
"/api/admin/policies/{policy_id}",
admin_delete_policy,
methods=["DELETE"],
),
# Templates
Route("/api/admin/templates", admin_list_templates),
Route("/api/admin/templates", admin_create_template, methods=["POST"]),
Route(
"/api/admin/templates/{template_id}",
admin_update_template,
methods=["PUT"],
),
Route(
"/api/admin/templates/{template_id}",
admin_delete_template,
methods=["DELETE"],
),
# Usage & Audit
Route("/api/admin/usage", admin_usage),
Route("/api/admin/audit", admin_audit),
],
),
],
middleware=[Middleware(_InjectAuthMiddleware)],
)
app.state.auth_storage = storage
return TestClient(app)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _role_payload(**overrides: Any) -> dict[str, Any]:
defaults: dict[str, Any] = {
"name": "analyst",
"display_name": "Data Analyst",
"permissions": "read,write",
}
defaults.update(overrides)
return defaults
def _policy_payload(**overrides: Any) -> dict[str, Any]:
defaults: dict[str, Any] = {
"name": "Allow bash",
"tool_pattern": "bash_*",
"action": "allow",
"priority": 10,
}
defaults.update(overrides)
return defaults
def _template_payload(**overrides: Any) -> dict[str, Any]:
defaults: dict[str, Any] = {
"name": "Greeting",
"content": "Hello {{user}}, how can I help?",
"category": "system",
}
defaults.update(overrides)
return defaults
# ---------------------------------------------------------------------------
# Tests — Roles
# ---------------------------------------------------------------------------
class TestRoles:
def test_list_empty(self, client):
resp = client.get("/v1/api/admin/roles")
assert resp.status_code == 200
assert resp.json()["roles"] == []
def test_create_role(self, client):
resp = client.post("/v1/api/admin/roles", json=_role_payload())
assert resp.status_code == 200
role = resp.json()
assert role["name"] == "analyst"
assert role["display_name"] == "Data Analyst"
assert role["permissions"] == "read,write"
assert role["builtin"] is False
assert "role_id" in role
assert "created" in role
def test_create_role_missing_name(self, client):
resp = client.post("/v1/api/admin/roles", json=_role_payload(name=""))
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
def test_create_role_invalid_name(self, client):
resp = client.post("/v1/api/admin/roles", json=_role_payload(name="bad name!@#"))
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
def test_create_role_default_display_name(self, client):
resp = client.post(
"/v1/api/admin/roles",
json={"name": "ops", "permissions": ""},
)
assert resp.status_code == 200
role = resp.json()
# display_name defaults to name when not provided
assert role["display_name"] == "ops"
def test_list_after_create(self, client):
client.post("/v1/api/admin/roles", json=_role_payload())
resp = client.get("/v1/api/admin/roles")
assert resp.status_code == 200
roles = resp.json()["roles"]
assert len(roles) == 1
assert roles[0]["name"] == "analyst"
def test_update_role(self, client):
create_resp = client.post("/v1/api/admin/roles", json=_role_payload())
role_id = create_resp.json()["role_id"]
resp = client.put(
f"/v1/api/admin/roles/{role_id}",
json={"display_name": "Senior Analyst", "permissions": "read,write,approve"},
)
assert resp.status_code == 200
role = resp.json()
assert role["display_name"] == "Senior Analyst"
assert role["permissions"] == "read,write,approve"
def test_update_nonexistent_role(self, client):
resp = client.put(
"/v1/api/admin/roles/nonexistent",
json={"display_name": "Nope"},
)
assert resp.status_code == 404
def test_update_builtin_role_rejected(self, client, storage):
# Seed a builtin role directly via storage
storage.create_role(
role_id="builtin-admin",
name="admin",
display_name="Administrator",
permissions="*",
builtin=True,
)
resp = client.put(
"/v1/api/admin/roles/builtin-admin",
json={"display_name": "Hacked"},
)
assert resp.status_code == 400
assert "builtin" in resp.json()["error"].lower()
def test_delete_role(self, client):
create_resp = client.post("/v1/api/admin/roles", json=_role_payload())
role_id = create_resp.json()["role_id"]
resp = client.delete(f"/v1/api/admin/roles/{role_id}")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify gone from listing
list_resp = client.get("/v1/api/admin/roles")
assert list_resp.json()["roles"] == []
def test_delete_nonexistent_role(self, client):
resp = client.delete("/v1/api/admin/roles/nonexistent")
assert resp.status_code == 404
def test_delete_builtin_role_rejected(self, client, storage):
storage.create_role(
role_id="builtin-viewer",
name="viewer",
display_name="Viewer",
permissions="read",
builtin=True,
)
resp = client.delete("/v1/api/admin/roles/builtin-viewer")
assert resp.status_code == 400
assert "builtin" in resp.json()["error"].lower()
# ---------------------------------------------------------------------------
# Tests — Role assignments
# ---------------------------------------------------------------------------
class TestRoleAssignments:
def test_list_user_roles_empty(self, client):
resp = client.get("/v1/api/admin/users/user-1/roles")
assert resp.status_code == 200
assert resp.json()["roles"] == []
def test_assign_role(self, client):
create_resp = client.post("/v1/api/admin/roles", json=_role_payload())
role_id = create_resp.json()["role_id"]
resp = client.post(
"/v1/api/admin/users/user-1/roles",
json={"role_id": role_id},
)
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify listed
list_resp = client.get("/v1/api/admin/users/user-1/roles")
roles = list_resp.json()["roles"]
assert len(roles) >= 1
def test_assign_role_missing_role_id(self, client):
resp = client.post(
"/v1/api/admin/users/user-1/roles",
json={},
)
assert resp.status_code == 400
assert "role_id" in resp.json()["error"].lower()
def test_unassign_role(self, client):
create_resp = client.post("/v1/api/admin/roles", json=_role_payload())
role_id = create_resp.json()["role_id"]
# Assign first
client.post(
"/v1/api/admin/users/user-1/roles",
json={"role_id": role_id},
)
# Now unassign
resp = client.delete(f"/v1/api/admin/users/user-1/roles/{role_id}")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify removed
list_resp = client.get("/v1/api/admin/users/user-1/roles")
assert list_resp.json()["roles"] == []
def test_unassign_nonexistent(self, client):
resp = client.delete("/v1/api/admin/users/user-1/roles/nonexistent")
assert resp.status_code == 404
# ---------------------------------------------------------------------------
# Tests — Orgs
# ---------------------------------------------------------------------------
class TestOrgs:
def test_list_empty(self, client):
resp = client.get("/v1/api/admin/orgs")
assert resp.status_code == 200
assert resp.json()["orgs"] == []
def test_get_org(self, client, storage):
storage.create_org(
org_id="org-1",
name="acme",
display_name="Acme Corp",
settings='{"theme": "dark"}',
)
resp = client.get("/v1/api/admin/orgs/org-1")
assert resp.status_code == 200
org = resp.json()
assert org["org_id"] == "org-1"
assert org["name"] == "acme"
assert org["display_name"] == "Acme Corp"
def test_get_org_not_found(self, client):
resp = client.get("/v1/api/admin/orgs/nonexistent")
assert resp.status_code == 404
def test_update_org(self, client, storage):
storage.create_org(org_id="org-1", name="acme", display_name="Acme Corp")
resp = client.put(
"/v1/api/admin/orgs/org-1",
json={"display_name": "Acme Inc.", "settings": '{"theme": "light"}'},
)
assert resp.status_code == 200
org = resp.json()
assert org["display_name"] == "Acme Inc."
assert org["settings"] == '{"theme": "light"}'
def test_update_org_not_found(self, client):
resp = client.put(
"/v1/api/admin/orgs/nonexistent",
json={"display_name": "Nope"},
)
assert resp.status_code == 404
# ---------------------------------------------------------------------------
# Tests — Tool policies
# ---------------------------------------------------------------------------
class TestPolicies:
def test_list_empty(self, client):
resp = client.get("/v1/api/admin/policies")
assert resp.status_code == 200
assert resp.json()["policies"] == []
def test_create_policy(self, client):
resp = client.post("/v1/api/admin/policies", json=_policy_payload())
assert resp.status_code == 200
policy = resp.json()
assert policy["name"] == "Allow bash"
assert policy["tool_pattern"] == "bash_*"
assert policy["action"] == "allow"
assert policy["priority"] == 10
assert "policy_id" in policy
assert "created" in policy
def test_create_policy_missing_name(self, client):
resp = client.post("/v1/api/admin/policies", json=_policy_payload(name=""))
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
def test_create_policy_missing_tool_pattern(self, client):
resp = client.post(
"/v1/api/admin/policies",
json=_policy_payload(tool_pattern=""),
)
assert resp.status_code == 400
assert "tool_pattern" in resp.json()["error"].lower()
def test_create_policy_invalid_action(self, client):
resp = client.post(
"/v1/api/admin/policies",
json=_policy_payload(action="yolo"),
)
assert resp.status_code == 400
assert "action" in resp.json()["error"].lower()
def test_list_after_create(self, client):
client.post("/v1/api/admin/policies", json=_policy_payload())
resp = client.get("/v1/api/admin/policies")
assert resp.status_code == 200
policies = resp.json()["policies"]
assert len(policies) == 1
assert policies[0]["name"] == "Allow bash"
def test_update_policy(self, client):
create_resp = client.post("/v1/api/admin/policies", json=_policy_payload())
policy_id = create_resp.json()["policy_id"]
resp = client.put(
f"/v1/api/admin/policies/{policy_id}",
json={"name": "Deny bash", "action": "deny", "priority": 20},
)
assert resp.status_code == 200
policy = resp.json()
assert policy["name"] == "Deny bash"
assert policy["action"] == "deny"
assert policy["priority"] == 20
def test_update_policy_invalid_action(self, client):
create_resp = client.post("/v1/api/admin/policies", json=_policy_payload())
policy_id = create_resp.json()["policy_id"]
resp = client.put(
f"/v1/api/admin/policies/{policy_id}",
json={"action": "nope"},
)
assert resp.status_code == 400
assert "action" in resp.json()["error"].lower()
def test_update_policy_not_found(self, client):
resp = client.put(
"/v1/api/admin/policies/nonexistent",
json={"name": "Nope"},
)
assert resp.status_code == 404
def test_delete_policy(self, client):
create_resp = client.post("/v1/api/admin/policies", json=_policy_payload())
policy_id = create_resp.json()["policy_id"]
resp = client.delete(f"/v1/api/admin/policies/{policy_id}")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify gone
list_resp = client.get("/v1/api/admin/policies")
assert list_resp.json()["policies"] == []
def test_delete_policy_not_found(self, client):
resp = client.delete("/v1/api/admin/policies/nonexistent")
assert resp.status_code == 404
# ---------------------------------------------------------------------------
# Tests — Prompt templates
# ---------------------------------------------------------------------------
class TestTemplates:
def test_list_empty(self, client):
resp = client.get("/v1/api/admin/templates")
assert resp.status_code == 200
assert resp.json()["templates"] == []
def test_create_template(self, client):
resp = client.post("/v1/api/admin/templates", json=_template_payload())
assert resp.status_code == 200
tmpl = resp.json()
assert tmpl["name"] == "Greeting"
assert "{{user}}" in tmpl["content"]
assert tmpl["category"] == "system"
assert "template_id" in tmpl
assert "created" in tmpl
def test_create_template_missing_name(self, client):
resp = client.post(
"/v1/api/admin/templates",
json=_template_payload(name=""),
)
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
def test_create_template_missing_content(self, client):
resp = client.post(
"/v1/api/admin/templates",
json=_template_payload(content=""),
)
assert resp.status_code == 400
assert "content" in resp.json()["error"].lower()
def test_list_after_create(self, client):
client.post("/v1/api/admin/templates", json=_template_payload())
resp = client.get("/v1/api/admin/templates")
assert resp.status_code == 200
templates = resp.json()["templates"]
assert len(templates) == 1
assert templates[0]["name"] == "Greeting"
def test_update_template(self, client):
create_resp = client.post("/v1/api/admin/templates", json=_template_payload())
template_id = create_resp.json()["template_id"]
resp = client.put(
f"/v1/api/admin/templates/{template_id}",
json={"name": "Welcome", "content": "Welcome, {{user}}!", "is_default": True},
)
assert resp.status_code == 200
tmpl = resp.json()
assert tmpl["name"] == "Welcome"
assert tmpl["content"] == "Welcome, {{user}}!"
assert tmpl["is_default"] is True
def test_update_template_not_found(self, client):
resp = client.put(
"/v1/api/admin/templates/nonexistent",
json={"name": "Nope"},
)
assert resp.status_code == 404
def test_delete_template(self, client):
create_resp = client.post("/v1/api/admin/templates", json=_template_payload())
template_id = create_resp.json()["template_id"]
resp = client.delete(f"/v1/api/admin/templates/{template_id}")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify gone
list_resp = client.get("/v1/api/admin/templates")
assert list_resp.json()["templates"] == []
def test_delete_template_not_found(self, client):
resp = client.delete("/v1/api/admin/templates/nonexistent")
assert resp.status_code == 404
# ---------------------------------------------------------------------------
# Tests — Usage
# ---------------------------------------------------------------------------
class TestUsage:
def test_usage_defaults(self, client):
"""Query usage with no params — should return summary and breakdown."""
resp = client.get("/v1/api/admin/usage")
assert resp.status_code == 200
data = resp.json()
assert "summary" in data
assert "breakdown" in data
# Summary is a list with at least one row
assert isinstance(data["summary"], list)
assert len(data["summary"]) >= 1
# All-zeros when no data
assert data["summary"][0]["prompt_tokens"] == 0
def test_usage_with_data(self, client, storage):
"""Seed usage events and verify they appear in the query."""
storage.record_usage_event(
event_id="evt-1",
user_id="user-1",
model="gpt-5",
prompt_tokens=100,
completion_tokens=50,
tool_calls_count=2,
)
storage.record_usage_event(
event_id="evt-2",
user_id="user-1",
model="gpt-5",
prompt_tokens=200,
completion_tokens=75,
tool_calls_count=1,
)
resp = client.get("/v1/api/admin/usage")
assert resp.status_code == 200
summary = resp.json()["summary"]
assert summary[0]["prompt_tokens"] == 300
assert summary[0]["completion_tokens"] == 125
assert summary[0]["tool_calls_count"] == 3
def test_usage_with_filters(self, client, storage):
storage.record_usage_event(
event_id="evt-f1",
user_id="user-a",
model="gpt-5",
prompt_tokens=100,
completion_tokens=10,
)
storage.record_usage_event(
event_id="evt-f2",
user_id="user-b",
model="claude-4",
prompt_tokens=200,
completion_tokens=20,
)
resp = client.get("/v1/api/admin/usage?user_id=user-a")
assert resp.status_code == 200
summary = resp.json()["summary"]
assert summary[0]["prompt_tokens"] == 100
resp2 = client.get("/v1/api/admin/usage?model=claude-4")
assert resp2.status_code == 200
summary2 = resp2.json()["summary"]
assert summary2[0]["prompt_tokens"] == 200
# ---------------------------------------------------------------------------
# Tests — Audit
# ---------------------------------------------------------------------------
class TestAudit:
def test_audit_empty(self, client):
resp = client.get("/v1/api/admin/audit")
assert resp.status_code == 200
data = resp.json()
assert data["events"] == []
assert data["total"] == 0
def test_audit_populated_by_mutations(self, client):
"""Creating a role should produce an audit event."""
client.post("/v1/api/admin/roles", json=_role_payload())
resp = client.get("/v1/api/admin/audit")
assert resp.status_code == 200
data = resp.json()
assert data["total"] >= 1
actions = [e["action"] for e in data["events"]]
assert "role.create" in actions
def test_audit_filter_by_action(self, client):
# Create a role and a policy to produce different audit actions
client.post("/v1/api/admin/roles", json=_role_payload())
client.post("/v1/api/admin/policies", json=_policy_payload())
resp = client.get("/v1/api/admin/audit?action=policy.create")
assert resp.status_code == 200
data = resp.json()
assert data["total"] >= 1
assert all(e["action"] == "policy.create" for e in data["events"])
def test_audit_filter_by_user_id(self, client):
client.post("/v1/api/admin/roles", json=_role_payload())
resp = client.get("/v1/api/admin/audit?user_id=test-admin")
assert resp.status_code == 200
data = resp.json()
assert data["total"] >= 1
assert all(e["user_id"] == "test-admin" for e in data["events"])
def test_audit_pagination(self, client):
# Create several resources to produce multiple audit events
for i in range(5):
client.post(
"/v1/api/admin/roles",
json=_role_payload(name=f"role-{i}"),
)
resp = client.get("/v1/api/admin/audit?limit=2&offset=0")
assert resp.status_code == 200
data = resp.json()
assert len(data["events"]) == 2
assert data["total"] >= 5
resp2 = client.get("/v1/api/admin/audit?limit=2&offset=2")
assert resp2.status_code == 200
data2 = resp2.json()
assert len(data2["events"]) == 2
# The two pages should not overlap
ids_page1 = {e["event_id"] for e in data["events"]}
ids_page2 = {e["event_id"] for e in data2["events"]}
assert ids_page1.isdisjoint(ids_page2)
# ---------------------------------------------------------------------------
# Tests — User self-deletion guard
# ---------------------------------------------------------------------------
class TestUserSelfDeletion:
def test_cannot_delete_self(self, client):
"""Admin should not be able to delete their own account."""
resp = client.delete("/v1/api/admin/users/test-admin")
assert resp.status_code == 400
assert "own account" in resp.json()["error"].lower()
def test_can_delete_other_user(self, client):
resp = client.delete("/v1/api/admin/users/user-1")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
+808
View File
@@ -0,0 +1,808 @@
"""Tests for governance storage operations (SQLite backend).
Covers RBAC roles, organizations, tool policies, prompt templates,
usage events, and audit events.
"""
from __future__ import annotations
from datetime import UTC, datetime
import pytest
import sqlalchemy as sa
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture()
def db(tmp_path):
"""Create a fresh SQLite backend for each test."""
return SQLiteBackend(str(tmp_path / "test.db"))
# ---------------------------------------------------------------------------
# Roles
# ---------------------------------------------------------------------------
class TestRoleCRUD:
def test_create_role(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
role = db.get_role("r1")
assert role is not None
assert role["role_id"] == "r1"
assert role["name"] == "editor"
assert role["display_name"] == "Editor"
assert role["permissions"] == "read,write"
assert role["builtin"] is False
assert role["org_id"] == ""
assert "created" in role
assert "updated" in role
def test_create_role_idempotent(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
# Second insert with same role_id should be silently ignored.
db.create_role("r1", "editor2", "Editor 2", "read", builtin=True, org_id="org1")
role = db.get_role("r1")
assert role is not None
# Original values preserved.
assert role["name"] == "editor"
assert role["display_name"] == "Editor"
def test_get_role_by_name(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
role = db.get_role_by_name("editor")
assert role is not None
assert role["role_id"] == "r1"
def test_get_role_by_name_nonexistent(self, db):
assert db.get_role_by_name("nope") is None
def test_list_roles(self, db):
db.create_role("r2", "beta", "Beta Role", "read", builtin=False, org_id="")
db.create_role("r1", "alpha", "Alpha Role", "write", builtin=False, org_id="")
roles = db.list_roles()
assert len(roles) == 2
# Ordered by name ascending.
assert roles[0]["name"] == "alpha"
assert roles[1]["name"] == "beta"
def test_list_roles_filter_org(self, db):
db.create_role("r1", "role_a", "A", "read", builtin=False, org_id="org1")
db.create_role("r2", "role_b", "B", "read", builtin=False, org_id="org2")
db.create_role("r3", "role_c", "C", "read", builtin=False, org_id="org1")
result = db.list_roles(org_id="org1")
assert len(result) == 2
assert {r["role_id"] for r in result} == {"r1", "r3"}
def test_update_role(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
ok = db.update_role("r1", permissions="read,write,approve", display_name="Senior Editor")
assert ok is True
role = db.get_role("r1")
assert role is not None
assert role["permissions"] == "read,write,approve"
assert role["display_name"] == "Senior Editor"
def test_update_role_nonexistent(self, db):
assert db.update_role("missing", permissions="read") is False
def test_delete_role(self, db):
db.create_role("r1", "editor", "Editor", "read", builtin=False, org_id="")
db.create_user("u1", "alice", "Alice", "$2b$hash")
db.assign_role("u1", "r1")
# Verify assignment exists.
assert len(db.list_user_roles("u1")) == 1
ok = db.delete_role("r1")
assert ok is True
assert db.get_role("r1") is None
# Cascade: user_roles for this role should be gone.
assert len(db.list_user_roles("u1")) == 0
def test_delete_role_nonexistent(self, db):
assert db.delete_role("missing") is False
def test_assign_role(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
db.create_user("u1", "alice", "Alice", "$2b$hash")
db.assign_role("u1", "r1", assigned_by="admin")
roles = db.list_user_roles("u1")
assert len(roles) == 1
assert roles[0]["role_id"] == "r1"
assert roles[0]["assigned_by"] == "admin"
def test_assign_role_idempotent(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
db.create_user("u1", "alice", "Alice", "$2b$hash")
db.assign_role("u1", "r1")
# Second assign should not raise.
db.assign_role("u1", "r1")
roles = db.list_user_roles("u1")
assert len(roles) == 1
def test_unassign_role(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
db.create_user("u1", "alice", "Alice", "$2b$hash")
db.assign_role("u1", "r1")
ok = db.unassign_role("u1", "r1")
assert ok is True
assert len(db.list_user_roles("u1")) == 0
def test_unassign_role_nonexistent(self, db):
assert db.unassign_role("u1", "r1") is False
def test_list_user_roles(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
db.create_role("r2", "viewer", "Viewer", "read", builtin=True, org_id="")
db.create_user("u1", "alice", "Alice", "$2b$hash")
db.assign_role("u1", "r1", assigned_by="admin")
db.assign_role("u1", "r2", assigned_by="system")
roles = db.list_user_roles("u1")
assert len(roles) == 2
# Each entry should have joined role fields plus assignment metadata.
for r in roles:
assert "role_id" in r
assert "name" in r
assert "permissions" in r
assert "assigned_by" in r
assert "assignment_created" in r
def test_get_user_permissions(self, db):
db.create_role("r1", "editor", "Editor", "read,write", builtin=False, org_id="")
db.create_role("r2", "approver", "Approver", "approve,read", builtin=False, org_id="")
db.create_user("u1", "alice", "Alice", "$2b$hash")
db.assign_role("u1", "r1")
db.assign_role("u1", "r2")
perms = db.get_user_permissions("u1")
assert perms == {"read", "write", "approve"}
def test_get_user_permissions_no_roles(self, db):
db.create_user("u1", "alice", "Alice", "$2b$hash")
assert db.get_user_permissions("u1") == set()
# ---------------------------------------------------------------------------
# Organizations
# ---------------------------------------------------------------------------
class TestOrgCRUD:
def test_create_org(self, db):
db.create_org("org1", "acme", "Acme Corp", '{"plan":"pro"}')
org = db.get_org("org1")
assert org is not None
assert org["org_id"] == "org1"
assert org["name"] == "acme"
assert org["display_name"] == "Acme Corp"
assert org["settings"] == '{"plan":"pro"}'
assert "created" in org
assert "updated" in org
def test_get_org_nonexistent(self, db):
assert db.get_org("nope") is None
def test_create_org_idempotent(self, db):
db.create_org("org1", "acme", "Acme Corp")
db.create_org("org1", "acme2", "Acme 2")
org = db.get_org("org1")
assert org is not None
assert org["name"] == "acme"
def test_list_orgs(self, db):
db.create_org("o2", "beta", "Beta Inc")
db.create_org("o1", "alpha", "Alpha LLC")
orgs = db.list_orgs()
assert len(orgs) == 2
# Ordered by name ascending.
assert orgs[0]["name"] == "alpha"
assert orgs[1]["name"] == "beta"
def test_update_org(self, db):
db.create_org("org1", "acme", "Acme Corp")
ok = db.update_org(
"org1", display_name="Acme Corp Global", settings='{"plan":"enterprise"}'
)
assert ok is True
org = db.get_org("org1")
assert org is not None
assert org["display_name"] == "Acme Corp Global"
assert org["settings"] == '{"plan":"enterprise"}'
def test_update_org_nonexistent(self, db):
assert db.update_org("missing", display_name="X") is False
# ---------------------------------------------------------------------------
# Tool Policies
# ---------------------------------------------------------------------------
class TestToolPolicyCRUD:
def test_create_tool_policy(self, db):
db.create_tool_policy(
"p1",
"deny-bash",
"bash*",
"deny",
priority=100,
org_id="org1",
enabled=True,
created_by="admin",
)
pol = db.get_tool_policy("p1")
assert pol is not None
assert pol["policy_id"] == "p1"
assert pol["name"] == "deny-bash"
assert pol["tool_pattern"] == "bash*"
assert pol["action"] == "deny"
assert pol["priority"] == 100
assert pol["org_id"] == "org1"
assert pol["enabled"] is True
assert pol["created_by"] == "admin"
def test_get_tool_policy_nonexistent(self, db):
assert db.get_tool_policy("missing") is None
def test_list_tool_policies_ordered_by_priority(self, db):
db.create_tool_policy("p1", "low", "*", "allow", priority=10)
db.create_tool_policy("p2", "high", "*", "deny", priority=100)
db.create_tool_policy("p3", "mid", "*", "ask", priority=50)
policies = db.list_tool_policies()
assert len(policies) == 3
# DESC priority order.
assert policies[0]["priority"] == 100
assert policies[1]["priority"] == 50
assert policies[2]["priority"] == 10
def test_update_tool_policy(self, db):
db.create_tool_policy("p1", "deny-bash", "bash*", "deny", priority=100)
ok = db.update_tool_policy("p1", action="allow", priority=50)
assert ok is True
pol = db.get_tool_policy("p1")
assert pol is not None
assert pol["action"] == "allow"
assert pol["priority"] == 50
def test_update_tool_policy_nonexistent(self, db):
assert db.update_tool_policy("missing", action="deny") is False
def test_delete_tool_policy(self, db):
db.create_tool_policy("p1", "deny-bash", "bash*", "deny", priority=100)
ok = db.delete_tool_policy("p1")
assert ok is True
assert db.get_tool_policy("p1") is None
def test_delete_tool_policy_nonexistent(self, db):
assert db.delete_tool_policy("missing") is False
def test_enabled_as_bool(self, db):
db.create_tool_policy("p1", "on", "*", "allow", priority=0, enabled=True)
db.create_tool_policy("p2", "off", "*", "deny", priority=0, enabled=False)
p1 = db.get_tool_policy("p1")
p2 = db.get_tool_policy("p2")
assert p1 is not None
assert p2 is not None
assert p1["enabled"] is True
assert isinstance(p1["enabled"], bool)
assert p2["enabled"] is False
assert isinstance(p2["enabled"], bool)
def test_list_policies_filter_org(self, db):
db.create_tool_policy("p1", "a", "*", "allow", priority=0, org_id="org1")
db.create_tool_policy("p2", "b", "*", "deny", priority=0, org_id="org2")
db.create_tool_policy("p3", "c", "*", "ask", priority=0, org_id="org1")
result = db.list_tool_policies(org_id="org1")
assert len(result) == 2
assert {r["policy_id"] for r in result} == {"p1", "p3"}
# ---------------------------------------------------------------------------
# Prompt Templates
# ---------------------------------------------------------------------------
class TestPromptTemplateCRUD:
def test_create_prompt_template(self, db):
db.create_prompt_template(
"t1",
"greeting",
"general",
"Hello {{name}}!",
variables='["name"]',
is_default=True,
org_id="org1",
created_by="admin",
)
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["template_id"] == "t1"
assert tpl["name"] == "greeting"
assert tpl["category"] == "general"
assert tpl["content"] == "Hello {{name}}!"
assert tpl["variables"] == '["name"]'
assert tpl["is_default"] is True
assert tpl["org_id"] == "org1"
assert tpl["created_by"] == "admin"
def test_get_prompt_template_nonexistent(self, db):
assert db.get_prompt_template("missing") is None
def test_list_prompt_templates_ordered_by_name(self, db):
db.create_prompt_template("t2", "beta", "general", "B")
db.create_prompt_template("t1", "alpha", "general", "A")
templates = db.list_prompt_templates()
assert len(templates) == 2
assert templates[0]["name"] == "alpha"
assert templates[1]["name"] == "beta"
def test_list_prompt_templates_filter_org(self, db):
db.create_prompt_template("t1", "a", "general", "A", org_id="org1")
db.create_prompt_template("t2", "b", "general", "B", org_id="org2")
result = db.list_prompt_templates(org_id="org1")
assert len(result) == 1
assert result[0]["template_id"] == "t1"
def test_update_prompt_template(self, db):
db.create_prompt_template("t1", "greeting", "general", "Hello!")
ok = db.update_prompt_template("t1", content="Hi there!", category="custom")
assert ok is True
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["content"] == "Hi there!"
assert tpl["category"] == "custom"
def test_update_prompt_template_nonexistent(self, db):
assert db.update_prompt_template("missing", content="x") is False
def test_delete_prompt_template(self, db):
db.create_prompt_template("t1", "greeting", "general", "Hello!")
ok = db.delete_prompt_template("t1")
assert ok is True
assert db.get_prompt_template("t1") is None
def test_delete_prompt_template_nonexistent(self, db):
assert db.delete_prompt_template("missing") is False
def test_is_default_as_bool(self, db):
db.create_prompt_template("t1", "default_one", "general", "D", is_default=True)
db.create_prompt_template("t2", "not_default", "general", "N", is_default=False)
t1 = db.get_prompt_template("t1")
t2 = db.get_prompt_template("t2")
assert t1 is not None
assert t2 is not None
assert t1["is_default"] is True
assert isinstance(t1["is_default"], bool)
assert t2["is_default"] is False
assert isinstance(t2["is_default"], bool)
def test_create_with_mcp_origin(self, db):
db.create_prompt_template(
"t1",
"mcp__srv__prompt",
"mcp",
"content",
variables="[]",
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server="srv",
readonly=True,
)
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["origin"] == "mcp"
assert tpl["mcp_server"] == "srv"
assert tpl["readonly"] is True
assert isinstance(tpl["readonly"], bool)
def test_default_origin_values(self, db):
db.create_prompt_template("t1", "basic", "general", "Hello")
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["origin"] == "manual"
assert tpl["mcp_server"] == ""
assert tpl["readonly"] is False
def test_get_prompt_template_by_name(self, db):
db.create_prompt_template("t1", "greeting", "general", "Hello!")
tpl = db.get_prompt_template_by_name("greeting")
assert tpl is not None
assert tpl["template_id"] == "t1"
assert tpl["name"] == "greeting"
def test_get_prompt_template_by_name_nonexistent(self, db):
assert db.get_prompt_template_by_name("nope") is None
def test_list_default_templates(self, db):
db.create_prompt_template("t1", "alpha", "general", "A", is_default=True)
db.create_prompt_template("t2", "beta", "general", "B", is_default=False)
db.create_prompt_template("t3", "gamma", "general", "C", is_default=True)
result = db.list_default_templates()
assert len(result) == 2
assert result[0]["name"] == "alpha"
assert result[1]["name"] == "gamma"
def test_list_default_templates_empty(self, db):
db.create_prompt_template("t1", "alpha", "general", "A", is_default=False)
assert db.list_default_templates() == []
def test_list_prompt_templates_by_origin(self, db):
db.create_prompt_template("t1", "manual_one", "general", "A", origin="manual")
db.create_prompt_template("t2", "mcp_one", "mcp", "B", origin="mcp", mcp_server="srv1")
db.create_prompt_template("t3", "mcp_two", "mcp", "C", origin="mcp", mcp_server="srv2")
result = db.list_prompt_templates_by_origin("mcp")
assert len(result) == 2
names = [r["name"] for r in result]
assert "mcp_one" in names
assert "mcp_two" in names
# ---------------------------------------------------------------------------
# Usage Events
# ---------------------------------------------------------------------------
class TestUsageEvents:
def test_record_usage_event(self, db):
db.record_usage_event(
"ev1",
user_id="u1",
ws_id="ws1",
node_id="n1",
model="gpt-5",
prompt_tokens=100,
completion_tokens=50,
tool_calls_count=2,
)
# Verify via query_usage (no group_by returns summary).
result = db.query_usage(since="2000-01-01T00:00:00")
assert len(result) == 1
assert result[0]["prompt_tokens"] == 100
assert result[0]["completion_tokens"] == 50
assert result[0]["tool_calls_count"] == 2
def test_query_usage_summary(self, db):
db.record_usage_event("ev1", model="gpt-5", prompt_tokens=100, completion_tokens=50)
db.record_usage_event("ev2", model="gpt-5", prompt_tokens=200, completion_tokens=75)
result = db.query_usage(since="2000-01-01T00:00:00")
assert len(result) == 1
assert result[0]["prompt_tokens"] == 300
assert result[0]["completion_tokens"] == 125
def test_query_usage_by_day(self, db):
# Insert events with known timestamps by directly inserting rows.
from turnstone.core.storage._schema import usage_events
with db._engine.connect() as conn:
conn.execute(
sa.insert(usage_events),
[
{
"event_id": "e1",
"timestamp": "2026-03-01T10:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "gpt-5",
"prompt_tokens": 100,
"completion_tokens": 50,
"tool_calls_count": 0,
"created": "2026-03-01T10:00:00",
},
{
"event_id": "e2",
"timestamp": "2026-03-01T14:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "gpt-5",
"prompt_tokens": 50,
"completion_tokens": 25,
"tool_calls_count": 0,
"created": "2026-03-01T14:00:00",
},
{
"event_id": "e3",
"timestamp": "2026-03-02T08:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "gpt-5",
"prompt_tokens": 200,
"completion_tokens": 100,
"tool_calls_count": 0,
"created": "2026-03-02T08:00:00",
},
],
)
conn.commit()
result = db.query_usage(since="2026-03-01T00:00:00", group_by="day")
assert len(result) == 2
assert result[0]["key"] == "2026-03-01"
assert result[0]["prompt_tokens"] == 150
assert result[1]["key"] == "2026-03-02"
assert result[1]["prompt_tokens"] == 200
def test_query_usage_by_model(self, db):
from turnstone.core.storage._schema import usage_events
with db._engine.connect() as conn:
conn.execute(
sa.insert(usage_events),
[
{
"event_id": "e1",
"timestamp": "2026-03-01T10:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "gpt-5",
"prompt_tokens": 100,
"completion_tokens": 50,
"tool_calls_count": 0,
"created": "2026-03-01T10:00:00",
},
{
"event_id": "e2",
"timestamp": "2026-03-01T10:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "claude-4",
"prompt_tokens": 200,
"completion_tokens": 100,
"tool_calls_count": 1,
"created": "2026-03-01T10:00:00",
},
],
)
conn.commit()
result = db.query_usage(since="2026-03-01T00:00:00", group_by="model")
assert len(result) == 2
keys = [r["key"] for r in result]
assert "gpt-5" in keys
assert "claude-4" in keys
def test_query_usage_by_user(self, db):
from turnstone.core.storage._schema import usage_events
with db._engine.connect() as conn:
conn.execute(
sa.insert(usage_events),
[
{
"event_id": "e1",
"timestamp": "2026-03-01T10:00:00",
"user_id": "u1",
"ws_id": "",
"node_id": "",
"model": "",
"prompt_tokens": 100,
"completion_tokens": 50,
"tool_calls_count": 0,
"created": "2026-03-01T10:00:00",
},
{
"event_id": "e2",
"timestamp": "2026-03-01T10:00:00",
"user_id": "u2",
"ws_id": "",
"node_id": "",
"model": "",
"prompt_tokens": 300,
"completion_tokens": 150,
"tool_calls_count": 2,
"created": "2026-03-01T10:00:00",
},
],
)
conn.commit()
result = db.query_usage(since="2026-03-01T00:00:00", group_by="user")
assert len(result) == 2
by_key = {r["key"]: r for r in result}
assert by_key["u1"]["prompt_tokens"] == 100
assert by_key["u2"]["prompt_tokens"] == 300
def test_query_usage_filter_model(self, db):
from turnstone.core.storage._schema import usage_events
with db._engine.connect() as conn:
conn.execute(
sa.insert(usage_events),
[
{
"event_id": "e1",
"timestamp": "2026-03-01T10:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "gpt-5",
"prompt_tokens": 100,
"completion_tokens": 50,
"tool_calls_count": 0,
"created": "2026-03-01T10:00:00",
},
{
"event_id": "e2",
"timestamp": "2026-03-01T10:00:00",
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "claude-4",
"prompt_tokens": 200,
"completion_tokens": 100,
"tool_calls_count": 0,
"created": "2026-03-01T10:00:00",
},
],
)
conn.commit()
result = db.query_usage(since="2026-03-01T00:00:00", model="gpt-5")
assert len(result) == 1
assert result[0]["prompt_tokens"] == 100
def test_prune_usage_events(self, db):
from turnstone.core.storage._schema import usage_events
old_ts = "2020-01-01T00:00:00"
now_ts = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with db._engine.connect() as conn:
conn.execute(
sa.insert(usage_events),
[
{
"event_id": "old",
"timestamp": old_ts,
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "",
"prompt_tokens": 10,
"completion_tokens": 5,
"tool_calls_count": 0,
"created": old_ts,
},
{
"event_id": "new",
"timestamp": now_ts,
"user_id": "",
"ws_id": "",
"node_id": "",
"model": "",
"prompt_tokens": 20,
"completion_tokens": 10,
"tool_calls_count": 0,
"created": now_ts,
},
],
)
conn.commit()
pruned = db.prune_usage_events(retention_days=30)
assert pruned == 1
# Only the recent event should remain.
result = db.query_usage(since="2000-01-01T00:00:00")
assert result[0]["prompt_tokens"] == 20
# ---------------------------------------------------------------------------
# Audit Events
# ---------------------------------------------------------------------------
class TestAuditEvents:
def test_record_audit_event(self, db):
db.record_audit_event(
"a1",
user_id="u1",
action="role.create",
resource_type="role",
resource_id="r1",
detail='{"name":"editor"}',
ip_address="127.0.0.1",
)
events = db.list_audit_events()
assert len(events) == 1
ev = events[0]
assert ev["event_id"] == "a1"
assert ev["user_id"] == "u1"
assert ev["action"] == "role.create"
assert ev["resource_type"] == "role"
assert ev["resource_id"] == "r1"
assert ev["detail"] == '{"name":"editor"}'
assert ev["ip_address"] == "127.0.0.1"
def test_list_audit_events(self, db):
db.record_audit_event("a1", action="login")
db.record_audit_event("a2", action="logout")
events = db.list_audit_events()
assert len(events) == 2
# Ordered by timestamp DESC — most recent first.
# Both created in quick succession with same-second granularity,
# but the order should still be deterministic (DESC).
assert {e["event_id"] for e in events} == {"a1", "a2"}
def test_list_audit_events_filter_action(self, db):
db.record_audit_event("a1", action="login")
db.record_audit_event("a2", action="logout")
db.record_audit_event("a3", action="login")
events = db.list_audit_events(action="login")
assert len(events) == 2
assert all(e["action"] == "login" for e in events)
def test_list_audit_events_filter_user(self, db):
db.record_audit_event("a1", user_id="u1", action="login")
db.record_audit_event("a2", user_id="u2", action="login")
events = db.list_audit_events(user_id="u1")
assert len(events) == 1
assert events[0]["user_id"] == "u1"
def test_list_audit_events_pagination(self, db):
for i in range(5):
db.record_audit_event(f"a{i}", action="test")
page1 = db.list_audit_events(limit=2, offset=0)
page2 = db.list_audit_events(limit=2, offset=2)
page3 = db.list_audit_events(limit=2, offset=4)
assert len(page1) == 2
assert len(page2) == 2
assert len(page3) == 1
# No overlap.
ids = [e["event_id"] for e in page1 + page2 + page3]
assert len(set(ids)) == 5
def test_count_audit_events(self, db):
db.record_audit_event("a1", action="login")
db.record_audit_event("a2", action="logout")
db.record_audit_event("a3", action="login")
assert db.count_audit_events() == 3
assert db.count_audit_events(action="login") == 2
assert db.count_audit_events(action="logout") == 1
def test_count_audit_events_filter_user(self, db):
db.record_audit_event("a1", user_id="u1", action="login")
db.record_audit_event("a2", user_id="u2", action="login")
assert db.count_audit_events(user_id="u1") == 1
def test_prune_audit_events(self, db):
from turnstone.core.storage._schema import audit_events
old_ts = "2020-01-01T00:00:00"
now_ts = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with db._engine.connect() as conn:
conn.execute(
sa.insert(audit_events),
[
{
"event_id": "old",
"timestamp": old_ts,
"user_id": "",
"action": "test",
"resource_type": "",
"resource_id": "",
"detail": "{}",
"ip_address": "",
"created": old_ts,
},
{
"event_id": "new",
"timestamp": now_ts,
"user_id": "",
"action": "test",
"resource_type": "",
"resource_id": "",
"detail": "{}",
"ip_address": "",
"created": now_ts,
},
],
)
conn.commit()
pruned = db.prune_audit_events(retention_days=30)
assert pruned == 1
assert db.count_audit_events() == 1
+214
View File
@@ -0,0 +1,214 @@
"""Tests for turnstone.core.log — structured logging configuration."""
from __future__ import annotations
import json
import logging
import structlog
from turnstone.core.log import (
configure_logging,
ctx_node_id,
ctx_request_id,
ctx_user_id,
ctx_ws_id,
get_logger,
)
class TestConfigureLogging:
"""Test configure_logging() sets up handlers and formatters."""
def setup_method(self):
# Reset structlog and stdlib between tests
structlog.reset_defaults()
root = logging.getLogger()
root.handlers.clear()
root.setLevel(logging.WARNING)
# Reset context vars
for var in (ctx_node_id, ctx_ws_id, ctx_user_id, ctx_request_id):
var.set("")
def test_sets_root_handler(self):
configure_logging(level="INFO", json_output=False, service="test")
root = logging.getLogger()
assert len(root.handlers) == 1
assert root.level == logging.INFO
def test_level_debug(self):
configure_logging(level="DEBUG", json_output=False)
root = logging.getLogger()
assert root.level == logging.DEBUG
def test_level_warning(self):
configure_logging(level="WARNING", json_output=False)
root = logging.getLogger()
assert root.level == logging.WARNING
def test_json_output(self, capsys):
configure_logging(level="INFO", json_output=True, service="test-svc")
log = logging.getLogger("test.json_output")
log.info("hello world")
captured = capsys.readouterr()
# JSON goes to stderr
line = captured.err.strip()
data = json.loads(line)
assert data["event"] == "hello world"
assert data["level"] == "info"
assert data["service"] == "test-svc"
assert "timestamp" in data
def test_console_output(self, capsys):
configure_logging(level="INFO", json_output=False)
log = logging.getLogger("test.console_output")
log.info("console hello")
captured = capsys.readouterr()
assert "console hello" in captured.err
def test_quiet_third_party(self):
configure_logging(level="DEBUG", json_output=False)
for name in ("httpx", "httpcore", "openai", "anthropic", "uvicorn.access"):
assert logging.getLogger(name).level == logging.WARNING
def test_replaces_existing_handlers(self):
root = logging.getLogger()
# Count existing handlers (pytest may add its own)
before = len(root.handlers)
root.addHandler(logging.StreamHandler())
root.addHandler(logging.StreamHandler())
assert len(root.handlers) == before + 2
configure_logging(level="INFO", json_output=False)
# configure_logging clears all and adds exactly 1
assert len(root.handlers) == 1
def test_env_var_level_override(self, monkeypatch):
monkeypatch.setenv("TURNSTONE_LOG_LEVEL", "ERROR")
configure_logging(level="DEBUG", json_output=False)
root = logging.getLogger()
assert root.level == logging.ERROR
def test_env_var_format_json(self, monkeypatch, capsys):
monkeypatch.setenv("TURNSTONE_LOG_FORMAT", "json")
configure_logging(level="INFO", service="test")
log = logging.getLogger("test.env_json")
log.info("env json test")
captured = capsys.readouterr()
data = json.loads(captured.err.strip())
assert data["event"] == "env json test"
def test_env_var_format_text(self, monkeypatch, capsys):
monkeypatch.setenv("TURNSTONE_LOG_FORMAT", "text")
configure_logging(level="INFO", json_output=True) # json_output overridden by env
log = logging.getLogger("test.env_text")
log.info("env text test")
captured = capsys.readouterr()
# Should NOT be JSON
line = captured.err.strip()
assert "env text test" in line
# Verify it's not JSON
try:
json.loads(line)
is_json = True
except json.JSONDecodeError:
is_json = False
assert not is_json
class TestContextInjection:
"""Test that context variables appear in log output."""
def setup_method(self):
structlog.reset_defaults()
root = logging.getLogger()
root.handlers.clear()
root.setLevel(logging.WARNING)
for var in (ctx_node_id, ctx_ws_id, ctx_user_id, ctx_request_id):
var.set("")
def test_node_id_in_output(self, capsys):
configure_logging(level="INFO", json_output=True)
ctx_node_id.set("worker-01_a3f2")
log = logging.getLogger("test.ctx")
log.info("ctx test")
data = json.loads(capsys.readouterr().err.strip())
assert data["node_id"] == "worker-01_a3f2"
def test_ws_id_in_output(self, capsys):
configure_logging(level="INFO", json_output=True)
ctx_ws_id.set("abc123")
log = logging.getLogger("test.ctx")
log.info("ws test")
data = json.loads(capsys.readouterr().err.strip())
assert data["ws_id"] == "abc123"
def test_empty_context_omitted(self, capsys):
configure_logging(level="INFO", json_output=True)
# All context vars are empty string (default)
log = logging.getLogger("test.ctx")
log.info("empty ctx")
data = json.loads(capsys.readouterr().err.strip())
assert "node_id" not in data
assert "ws_id" not in data
assert "user_id" not in data
assert "request_id" not in data
def test_multiple_context_vars(self, capsys):
configure_logging(level="INFO", json_output=True)
ctx_node_id.set("node-1")
ctx_ws_id.set("ws-2")
ctx_request_id.set("req-3")
log = logging.getLogger("test.ctx")
log.info("multi ctx")
data = json.loads(capsys.readouterr().err.strip())
assert data["node_id"] == "node-1"
assert data["ws_id"] == "ws-2"
assert data["request_id"] == "req-3"
assert "user_id" not in data
class TestGetLogger:
"""Test get_logger() returns a usable bound logger."""
def setup_method(self):
structlog.reset_defaults()
root = logging.getLogger()
root.handlers.clear()
root.setLevel(logging.WARNING)
def test_get_logger_returns_bound_logger(self):
configure_logging(level="INFO", json_output=False)
log = get_logger("test.bound")
assert log is not None
def test_get_logger_outputs(self, capsys):
configure_logging(level="INFO", json_output=True)
log = get_logger("test.bound")
log.info("bound logger test", extra_key="extra_val")
data = json.loads(capsys.readouterr().err.strip())
assert data["event"] == "bound logger test"
assert data["extra_key"] == "extra_val"
class TestServiceField:
"""Test that service name is injected when configured."""
def setup_method(self):
structlog.reset_defaults()
root = logging.getLogger()
root.handlers.clear()
root.setLevel(logging.WARNING)
def test_service_present(self, capsys):
configure_logging(level="INFO", json_output=True, service="myservice")
log = logging.getLogger("test.svc")
log.info("svc test")
data = json.loads(capsys.readouterr().err.strip())
assert data["service"] == "myservice"
def test_no_service_when_empty(self, capsys):
configure_logging(level="INFO", json_output=True)
log = logging.getLogger("test.svc")
log.info("no svc")
data = json.loads(capsys.readouterr().err.strip())
assert "service" not in data
+958 -1
View File
@@ -2,10 +2,11 @@
from __future__ import annotations
import asyncio
import json
from contextlib import AsyncExitStack
from typing import Any
from unittest.mock import MagicMock, patch
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@@ -50,6 +51,83 @@ def _fake_openai_tool(name: str = "mcp__test__search") -> dict[str, Any]:
}
def _fake_mcp_resource(
uri: str = "file:///README.md",
name: str = "readme",
description: str = "Project readme",
mime_type: str = "text/plain",
) -> MagicMock:
"""Create a mock MCP Resource object matching the SDK's Resource type."""
res = MagicMock()
res.uri = uri
res.name = name
res.description = description
res.mimeType = mime_type
return res
def _fake_resource_dict(
uri: str = "file:///README.md",
name: str = "readme",
description: str = "Project readme",
mime_type: str = "text/plain",
server: str = "test",
) -> dict[str, Any]:
"""Create a fake resource dict as stored in per-server state."""
return {
"uri": uri,
"name": name,
"description": description,
"mimeType": mime_type,
"server": server,
}
def _fake_mcp_prompt(
name: str = "code_review",
description: str = "Generate a code review",
arguments: list[dict[str, Any]] | None = None,
) -> MagicMock:
"""Create a mock MCP Prompt object matching the SDK's Prompt type."""
prompt = MagicMock()
prompt.name = name
prompt.description = description
if arguments is None:
arg = MagicMock()
arg.name = "language"
arg.description = "Programming language"
arg.required = True
prompt.arguments = [arg]
else:
mock_args = []
for a in arguments:
arg = MagicMock()
arg.name = a["name"]
arg.description = a.get("description", "")
arg.required = a.get("required", False)
mock_args.append(arg)
prompt.arguments = mock_args
return prompt
def _fake_prompt_dict(
name: str = "mcp__test__code_review",
original_name: str = "code_review",
server: str = "test",
description: str = "Generate a code review",
) -> dict[str, Any]:
"""Create a fake prompt dict as stored in per-server state."""
return {
"name": name,
"original_name": original_name,
"server": server,
"description": description,
"arguments": [
{"name": "language", "description": "Programming language", "required": True}
],
}
# ---------------------------------------------------------------------------
# Schema conversion
# ---------------------------------------------------------------------------
@@ -411,3 +489,882 @@ class TestCreateMcpClient:
result = create_mcp_client()
assert result is None
# ---------------------------------------------------------------------------
# Tool refresh — _rebuild_tools, _refresh_server, listeners
# ---------------------------------------------------------------------------
class TestRebuildTools:
def test_rebuild_from_per_server(self):
mgr = MCPClientManager({})
mgr._per_server_tools = {
"github": [_fake_openai_tool("mcp__github__search")],
"slack": [_fake_openai_tool("mcp__slack__send")],
}
mgr._rebuild_tools()
assert len(mgr._tools) == 2
names = {t["function"]["name"] for t in mgr._tools}
assert names == {"mcp__github__search", "mcp__slack__send"}
assert mgr._tool_map["mcp__github__search"] == ("github", "search")
assert mgr._tool_map["mcp__slack__send"] == ("slack", "send")
def test_rebuild_copy_on_write(self):
mgr = MCPClientManager({})
mgr._per_server_tools = {"a": [_fake_openai_tool("mcp__a__x")]}
mgr._rebuild_tools()
old_tools = mgr._tools
old_map = mgr._tool_map
mgr._per_server_tools["b"] = [_fake_openai_tool("mcp__b__y")]
mgr._rebuild_tools()
assert mgr._tools is not old_tools
assert mgr._tool_map is not old_map
def test_rebuild_empty(self):
mgr = MCPClientManager({})
mgr._per_server_tools = {}
mgr._rebuild_tools()
assert mgr._tools == []
assert mgr._tool_map == {}
class TestRefreshServer:
@staticmethod
def _add_empty_resource_prompt_mocks(
mgr: MCPClientManager, server_name: str, mock_session: MagicMock
) -> None:
"""Add empty list_resources/list_prompts mocks so _refresh_server works."""
mgr._supports_resources[server_name] = True
mgr._supports_prompts[server_name] = True
empty_res = MagicMock()
empty_res.resources = []
mock_session.list_resources = AsyncMock(return_value=empty_res)
empty_tmpl = MagicMock()
empty_tmpl.resourceTemplates = []
mock_session.list_resource_templates = AsyncMock(return_value=empty_tmpl)
empty_prompts = MagicMock()
empty_prompts.prompts = []
mock_session.list_prompts = AsyncMock(return_value=empty_prompts)
def test_refresh_detects_added_tools(self):
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mock_result = MagicMock()
mock_result.tools = [
_fake_mcp_tool("search"),
_fake_mcp_tool("create"), # new tool
]
mock_session.list_tools = AsyncMock(return_value=mock_result)
self._add_empty_resource_prompt_mocks(mgr, "github", mock_session)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
added, removed = await mgr._refresh_server("github")
assert "mcp__github__create" in added
assert removed == []
assert len(mgr._tools) == 2
asyncio.run(_run())
def test_refresh_detects_removed_tools(self):
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mock_result = MagicMock()
mock_result.tools = [] # all tools removed
mock_session.list_tools = AsyncMock(return_value=mock_result)
self._add_empty_resource_prompt_mocks(mgr, "github", mock_session)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
added, removed = await mgr._refresh_server("github")
assert added == []
assert "mcp__github__search" in removed
assert mgr._tools == []
asyncio.run(_run())
def test_refresh_no_changes(self):
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mock_result = MagicMock()
mock_result.tools = [_fake_mcp_tool("search")]
mock_session.list_tools = AsyncMock(return_value=mock_result)
self._add_empty_resource_prompt_mocks(mgr, "github", mock_session)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
added, removed = await mgr._refresh_server("github")
assert added == []
assert removed == []
asyncio.run(_run())
def test_refresh_disconnected_raises(self):
async def _run() -> None:
mgr = MCPClientManager({})
with pytest.raises(RuntimeError, match="not connected"):
await mgr._refresh_server_tools("ghost")
asyncio.run(_run())
class TestListeners:
def test_add_and_notify(self):
mgr = MCPClientManager({})
calls: list[int] = []
mgr.add_listener(lambda: calls.append(1))
mgr._per_server_tools = {"a": [_fake_openai_tool("mcp__a__x")]}
mgr._rebuild_tools()
assert len(calls) == 1
def test_remove_listener(self):
mgr = MCPClientManager({})
calls: list[int] = []
cb = lambda: calls.append(1) # noqa: E731
mgr.add_listener(cb)
mgr.remove_listener(cb)
mgr._rebuild_tools()
assert calls == []
def test_remove_nonexistent_listener(self):
mgr = MCPClientManager({})
mgr.remove_listener(lambda: None) # should not raise
def test_listener_error_does_not_propagate(self):
mgr = MCPClientManager({})
mgr.add_listener(lambda: 1 / 0) # will raise ZeroDivisionError
mgr._rebuild_tools() # should not raise
class TestServerNames:
def test_server_names_property(self):
mgr = MCPClientManager({"github": {}, "slack": {}})
assert sorted(mgr.server_names) == ["github", "slack"]
def test_server_names_empty(self):
mgr = MCPClientManager({})
assert mgr.server_names == []
# ---------------------------------------------------------------------------
# Session integration — tool refresh propagation
# ---------------------------------------------------------------------------
class TestSessionRefresh:
@pytest.fixture()
def tmp_db(self, tmp_path):
from turnstone.core.storage import init_storage, reset_storage
reset_storage()
init_storage("sqlite", path=str(tmp_path / "test.db"), run_migrations=False)
yield
reset_storage()
def _make_session(self, mcp_client=None, **kwargs):
from turnstone.core.session import ChatSession
defaults: dict[str, Any] = dict(
client=MagicMock(),
model="test-model",
ui=MagicMock(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
mcp_client=mcp_client,
)
defaults.update(kwargs)
return ChatSession(**defaults)
def test_listener_registered_on_init(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = []
session = self._make_session(mcp_client=mock_mcp)
mock_mcp.add_listener.assert_called_once()
assert session._mcp_refresh_cb is not None
def test_no_listener_without_mcp(self, tmp_db):
session = self._make_session(mcp_client=None)
assert session._mcp_refresh_cb is None
def test_close_removes_listener(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = []
session = self._make_session(mcp_client=mock_mcp)
session.close()
mock_mcp.remove_listener.assert_called_once()
assert session._mcp_refresh_cb is None
def test_close_idempotent(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = []
session = self._make_session(mcp_client=mock_mcp)
session.close()
session.close() # should not raise
assert mock_mcp.remove_listener.call_count == 1
def test_on_mcp_tools_changed_rebuilds_tools(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool("mcp__test__a")]
session = self._make_session(mcp_client=mock_mcp)
initial_count = len(session._tools)
# Simulate a tool refresh — MCP now has 2 tools
mock_mcp.get_tools.return_value = [
_fake_openai_tool("mcp__test__a"),
_fake_openai_tool("mcp__test__b"),
]
session._on_mcp_tools_changed()
assert len(session._tools) == initial_count + 1
def test_tool_search_preserved_across_refresh(self, tmp_db):
# Create enough MCP tools to trigger tool search
mcp_tools = [_fake_openai_tool(f"mcp__srv__tool{i}") for i in range(25)]
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = mcp_tools
session = self._make_session(
mcp_client=mock_mcp,
tool_search="auto",
tool_search_threshold=20,
)
assert session._tool_search is not None
# Expand a tool
session._tool_search.expand_visible(["mcp__srv__tool0"])
assert "mcp__srv__tool0" in session._tool_search.get_expanded_names()
# Refresh with same tools
session._on_mcp_tools_changed()
assert session._tool_search is not None
assert "mcp__srv__tool0" in session._tool_search.get_expanded_names()
def test_tool_search_prunes_removed_from_expanded(self, tmp_db):
mcp_tools = [_fake_openai_tool(f"mcp__srv__tool{i}") for i in range(25)]
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = mcp_tools
session = self._make_session(
mcp_client=mock_mcp,
tool_search="auto",
tool_search_threshold=20,
)
session._tool_search.expand_visible(["mcp__srv__tool0"])
# Refresh with tool0 removed
new_tools = [_fake_openai_tool(f"mcp__srv__tool{i}") for i in range(1, 25)]
mock_mcp.get_tools.return_value = new_tools
session._on_mcp_tools_changed()
# tool0 was removed, so it should no longer be in expanded
expanded = session._tool_search.get_expanded_names()
assert "mcp__srv__tool0" not in expanded
def test_mcp_refresh_command(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["test"]
mock_mcp.refresh_sync.return_value = {"test": (["mcp__test__new"], [])}
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh")
mock_mcp.refresh_sync.assert_called_once_with(None)
session.ui.on_info.assert_called()
def test_mcp_refresh_specific_server(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["github", "slack"]
mock_mcp.refresh_sync.return_value = {"github": ([], [])}
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh github")
mock_mcp.refresh_sync.assert_called_once_with("github")
def test_mcp_refresh_unknown_server(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["github"]
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh nonexistent")
session.ui.on_error.assert_called_once()
assert "Unknown MCP server" in session.ui.on_error.call_args[0][0]
def test_mcp_refresh_error_handling(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["test"]
mock_mcp.refresh_sync.side_effect = TimeoutError("timed out")
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh")
session.ui.on_error.assert_called_once()
assert "MCP refresh failed" in session.ui.on_error.call_args[0][0]
# ---------------------------------------------------------------------------
# MCP Resources
# ---------------------------------------------------------------------------
class TestMCPResources:
def test_resource_discovery(self):
"""Mock list_resources() returning 2 resources, verify get_resources()."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"fs": [
_fake_resource_dict("file:///a.txt", "a", "File A", "text/plain", "fs"),
_fake_resource_dict("file:///b.txt", "b", "File B", "text/plain", "fs"),
],
}
mgr._rebuild_resources()
resources = mgr.get_resources()
assert len(resources) == 2
uris = {r["uri"] for r in resources}
assert uris == {"file:///a.txt", "file:///b.txt"}
assert all(r["server"] == "fs" for r in resources)
def test_rebuild_resources_copy_on_write(self):
"""Verify mutation safety — get_resources() returns independent copy."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"a": [_fake_resource_dict("file:///x", "x", "", "", "a")],
}
mgr._rebuild_resources()
old_resources = mgr._resources
old_map = mgr._resource_map
mgr._per_server_resources["b"] = [_fake_resource_dict("file:///y", "y", "", "", "b")]
mgr._rebuild_resources()
assert mgr._resources is not old_resources
assert mgr._resource_map is not old_map
def test_get_resources_returns_copy(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {
"a": [_fake_resource_dict("file:///x", "x", "", "", "a")],
}
mgr._rebuild_resources()
resources = mgr.get_resources()
assert len(resources) == 1
resources.clear()
assert len(mgr.get_resources()) == 1
def test_read_resource_sync(self):
"""Mock session.read_resource(), verify text extraction."""
mgr = MCPClientManager({})
mgr._resource_map = {"file:///readme": ("fs", "file:///readme")}
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._loop = asyncio.new_event_loop()
# Mock the read_resource result
text_content = MagicMock(spec=["text"])
text_content.text = "Hello, world!"
mock_result = MagicMock()
mock_result.contents = [text_content]
mock_session.read_resource = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
output = mgr.read_resource_sync("file:///readme", timeout=5)
assert output == "Hello, world!"
mock_session.read_resource.assert_awaited_once_with("file:///readme")
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_read_resource_sync_blob(self):
"""Verify base64 blob extraction."""
mgr = MCPClientManager({})
mgr._resource_map = {"file:///img.png": ("fs", "file:///img.png")}
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._loop = asyncio.new_event_loop()
blob_content = MagicMock(spec=["blob"])
blob_content.blob = "aGVsbG8="
mock_result = MagicMock()
mock_result.contents = [blob_content]
mock_session.read_resource = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
output = mgr.read_resource_sync("file:///img.png", timeout=5)
assert output == "aGVsbG8="
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_read_resource_sync_unknown_uri(self):
mgr = MCPClientManager({})
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("file:///nonexistent")
def test_read_resource_sync_disconnected(self):
mgr = MCPClientManager({})
mgr._resource_map = {"file:///x": ("dead", "file:///x")}
with pytest.raises(RuntimeError, match="not connected"):
mgr.read_resource_sync("file:///x")
def test_read_resource_sync_timeout(self):
"""Verify timeout handling."""
mgr = MCPClientManager({})
mgr._resource_map = {"file:///x": ("fs", "file:///x")}
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._loop = asyncio.new_event_loop()
async def _slow_read(_uri: str) -> None:
await asyncio.sleep(10)
mock_session.read_resource = _slow_read
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
with pytest.raises(TimeoutError):
mgr.read_resource_sync("file:///x", timeout=1)
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_resource_listener_notification(self):
"""Verify callback fires on rebuild."""
mgr = MCPClientManager({})
calls: list[int] = []
mgr.add_resource_listener(lambda: calls.append(1))
mgr._per_server_resources = {"a": [_fake_resource_dict()]}
mgr._rebuild_resources()
assert len(calls) == 1
def test_resource_listener_remove(self):
mgr = MCPClientManager({})
calls: list[int] = []
cb = lambda: calls.append(1) # noqa: E731
mgr.add_resource_listener(cb)
mgr.remove_resource_listener(cb)
mgr._rebuild_resources()
assert calls == []
def test_resource_listener_error_does_not_propagate(self):
mgr = MCPClientManager({})
mgr.add_resource_listener(lambda: 1 / 0)
mgr._rebuild_resources() # should not raise
def test_resource_refresh_on_notification(self):
"""Mock notification, verify re-fetch of resources."""
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._supports_resources["fs"] = True
# Initial state
mgr._per_server_resources["fs"] = [
_fake_resource_dict("file:///old", server="fs"),
]
mgr._rebuild_resources()
assert len(mgr.get_resources()) == 1
# Mock the re-fetch returning a new resource
new_res = _fake_mcp_resource("file:///new", "new")
mock_res_result = MagicMock()
mock_res_result.resources = [new_res]
mock_session.list_resources = AsyncMock(return_value=mock_res_result)
mock_tmpl_result = MagicMock()
mock_tmpl_result.resourceTemplates = []
mock_session.list_resource_templates = AsyncMock(return_value=mock_tmpl_result)
await mgr._refresh_server_resources("fs")
resources = mgr.get_resources()
assert len(resources) == 1
assert resources[0]["uri"] == "file:///new"
asyncio.run(_run())
def test_rebuild_resources_empty(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {}
mgr._rebuild_resources()
assert mgr._resources == []
assert mgr._resource_map == {}
def test_rebuild_resources_multi_server(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {
"fs": [_fake_resource_dict("file:///a", server="fs")],
"db": [_fake_resource_dict("db://table", name="table", server="db")],
}
mgr._rebuild_resources()
assert len(mgr._resources) == 2
assert mgr._resource_map["file:///a"] == ("fs", "file:///a")
assert mgr._resource_map["db://table"] == ("db", "db://table")
def test_template_prefix_matching(self):
"""Expanded URI matches template by prefix."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"db": [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "A row",
"mimeType": "application/json",
"server": "db",
"template": True,
},
],
}
mgr._rebuild_resources()
# Template should not be in resource_map
assert "db://tables/{table}/rows/{id}" not in mgr._resource_map
# But prefix matching should find it
result = mgr._match_template("db://tables/users/rows/1")
assert result is not None
server, template_uri = result
assert server == "db"
assert template_uri == "db://tables/{table}/rows/{id}"
def test_template_longest_prefix_wins(self):
"""When two templates have overlapping prefixes, the longer one wins."""
mgr = MCPClientManager({})
# Use templates with genuinely different prefix lengths:
# "db://data/" (6 chars after scheme) vs "db://data/tables/" (13 chars after scheme)
mgr._per_server_resources = {
"short": [
{
"uri": "db://data/{collection}",
"name": "collection",
"description": "",
"mimeType": "",
"server": "short",
"template": True,
},
],
"long": [
{
"uri": "db://data/tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "long",
"template": True,
},
],
}
mgr._rebuild_resources()
# "db://data/tables/users" matches both prefixes ("db://data/" and
# "db://data/tables/") — the longer one should win
result = mgr._match_template("db://data/tables/users")
assert result is not None
server, template_uri = result
assert server == "long"
assert template_uri == "db://data/tables/{table}"
# URI that only matches the short prefix
result2 = mgr._match_template("db://data/views/active")
assert result2 is not None
assert result2[0] == "short"
def test_template_no_match_raises(self):
"""Completely unrelated URI still raises ValueError."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"db": [
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "db",
"template": True,
},
],
}
mgr._rebuild_resources()
assert mgr._match_template("file:///something") is None
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("file:///something")
def test_read_resource_sync_with_template_uri(self):
"""End-to-end: template discovered, expanded URI dispatched to correct server."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"db": [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "A row",
"mimeType": "application/json",
"server": "db",
"template": True,
},
],
}
mgr._rebuild_resources()
mock_session = MagicMock()
mgr._sessions["db"] = mock_session
mgr._loop = asyncio.new_event_loop()
text_content = MagicMock(spec=["text"])
text_content.text = '{"name": "Alice"}'
mock_result = MagicMock()
mock_result.contents = [text_content]
mock_session.read_resource = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
output = mgr.read_resource_sync("db://tables/users/rows/1", timeout=5)
assert output == '{"name": "Alice"}'
mock_session.read_resource.assert_awaited_once_with("db://tables/users/rows/1")
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
# ---------------------------------------------------------------------------
# MCP Prompts
# ---------------------------------------------------------------------------
class TestMCPPrompts:
def test_prompt_discovery(self):
"""Mock list_prompts(), verify get_prompts() with correct prefixed names."""
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"tmpl": [
_fake_prompt_dict("mcp__tmpl__code_review", "code_review", "tmpl"),
_fake_prompt_dict("mcp__tmpl__summarize", "summarize", "tmpl"),
],
}
mgr._rebuild_prompts()
prompts = mgr.get_prompts()
assert len(prompts) == 2
names = {p["name"] for p in prompts}
assert names == {"mcp__tmpl__code_review", "mcp__tmpl__summarize"}
# Verify map entries
assert mgr._prompt_map["mcp__tmpl__code_review"] == ("tmpl", "code_review")
assert mgr._prompt_map["mcp__tmpl__summarize"] == ("tmpl", "summarize")
def test_rebuild_prompts_copy_on_write(self):
"""Verify mutation safety."""
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"a": [_fake_prompt_dict("mcp__a__p1", "p1", "a")],
}
mgr._rebuild_prompts()
old_prompts = mgr._prompts
old_map = mgr._prompt_map
mgr._per_server_prompts["b"] = [_fake_prompt_dict("mcp__b__p2", "p2", "b")]
mgr._rebuild_prompts()
assert mgr._prompts is not old_prompts
assert mgr._prompt_map is not old_map
def test_get_prompts_returns_copy(self):
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"a": [_fake_prompt_dict("mcp__a__p1", "p1", "a")],
}
mgr._rebuild_prompts()
prompts = mgr.get_prompts()
assert len(prompts) == 1
prompts.clear()
assert len(mgr.get_prompts()) == 1
def test_get_prompt_sync(self):
"""Mock session.get_prompt(), verify message conversion."""
mgr = MCPClientManager({})
mgr._prompt_map = {"mcp__tmpl__review": ("tmpl", "review")}
mock_session = MagicMock()
mgr._sessions["tmpl"] = mock_session
mgr._loop = asyncio.new_event_loop()
# Build mock PromptMessage
msg1 = MagicMock()
msg1.role = "user"
msg1.content = MagicMock()
msg1.content.text = "Review this code"
msg2 = MagicMock()
msg2.role = "assistant"
msg2.content = MagicMock()
msg2.content.text = "Looks good!"
mock_result = MagicMock()
mock_result.messages = [msg1, msg2]
mock_session.get_prompt = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
messages = mgr.get_prompt_sync(
"mcp__tmpl__review", arguments={"language": "python"}, timeout=5
)
assert len(messages) == 2
assert messages[0] == {"role": "user", "content": "Review this code"}
assert messages[1] == {"role": "assistant", "content": "Looks good!"}
mock_session.get_prompt.assert_awaited_once_with(
"review", arguments={"language": "python"}
)
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_get_prompt_sync_unknown(self):
mgr = MCPClientManager({})
with pytest.raises(ValueError, match="Unknown MCP prompt"):
mgr.get_prompt_sync("mcp__no__such")
def test_get_prompt_sync_disconnected(self):
mgr = MCPClientManager({})
mgr._prompt_map = {"mcp__dead__p": ("dead", "p")}
with pytest.raises(RuntimeError, match="not connected"):
mgr.get_prompt_sync("mcp__dead__p")
def test_get_prompt_sync_timeout(self):
"""Verify timeout handling."""
mgr = MCPClientManager({})
mgr._prompt_map = {"mcp__tmpl__slow": ("tmpl", "slow")}
mock_session = MagicMock()
mgr._sessions["tmpl"] = mock_session
mgr._loop = asyncio.new_event_loop()
async def _slow_prompt(_name: str, *, arguments: dict[str, str] | None = None) -> None:
await asyncio.sleep(10)
mock_session.get_prompt = _slow_prompt
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
with pytest.raises(TimeoutError):
mgr.get_prompt_sync("mcp__tmpl__slow", timeout=1)
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_prompt_listener_notification(self):
"""Verify callback fires on rebuild."""
mgr = MCPClientManager({})
calls: list[int] = []
mgr.add_prompt_listener(lambda: calls.append(1))
mgr._per_server_prompts = {"a": [_fake_prompt_dict()]}
mgr._rebuild_prompts()
assert len(calls) == 1
def test_prompt_listener_remove(self):
mgr = MCPClientManager({})
calls: list[int] = []
cb = lambda: calls.append(1) # noqa: E731
mgr.add_prompt_listener(cb)
mgr.remove_prompt_listener(cb)
mgr._rebuild_prompts()
assert calls == []
def test_prompt_listener_error_does_not_propagate(self):
mgr = MCPClientManager({})
mgr.add_prompt_listener(lambda: 1 / 0)
mgr._rebuild_prompts() # should not raise
def test_is_mcp_prompt(self):
"""Verify name lookup."""
mgr = MCPClientManager({})
mgr._prompt_map["mcp__tmpl__review"] = ("tmpl", "review")
assert mgr.is_mcp_prompt("mcp__tmpl__review") is True
assert mgr.is_mcp_prompt("nonexistent") is False
def test_prompt_refresh_on_notification(self):
"""Mock notification, verify re-fetch of prompts."""
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mgr._sessions["tmpl"] = mock_session
mgr._supports_prompts["tmpl"] = True
# Initial state
mgr._per_server_prompts["tmpl"] = [
_fake_prompt_dict("mcp__tmpl__old", "old", "tmpl"),
]
mgr._rebuild_prompts()
assert len(mgr.get_prompts()) == 1
# Mock re-fetch returning a new prompt
new_prompt = _fake_mcp_prompt("new_prompt", "A new prompt")
mock_prompt_result = MagicMock()
mock_prompt_result.prompts = [new_prompt]
mock_session.list_prompts = AsyncMock(return_value=mock_prompt_result)
await mgr._refresh_server_prompts("tmpl")
prompts = mgr.get_prompts()
assert len(prompts) == 1
assert prompts[0]["name"] == "mcp__tmpl__new_prompt"
assert prompts[0]["original_name"] == "new_prompt"
asyncio.run(_run())
def test_rebuild_prompts_empty(self):
mgr = MCPClientManager({})
mgr._per_server_prompts = {}
mgr._rebuild_prompts()
assert mgr._prompts == []
assert mgr._prompt_map == {}
def test_rebuild_prompts_multi_server(self):
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"a": [_fake_prompt_dict("mcp__a__p1", "p1", "a")],
"b": [_fake_prompt_dict("mcp__b__p2", "p2", "b")],
}
mgr._rebuild_prompts()
assert len(mgr._prompts) == 2
assert mgr._prompt_map["mcp__a__p1"] == ("a", "p1")
assert mgr._prompt_map["mcp__b__p2"] == ("b", "p2")
# ---------------------------------------------------------------------------
# Shutdown cleans up new state
# ---------------------------------------------------------------------------
class TestShutdownCleanup:
def test_shutdown_clears_resources_and_prompts(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {"a": [_fake_resource_dict()]}
mgr._rebuild_resources()
mgr._per_server_prompts = {"a": [_fake_prompt_dict()]}
mgr._rebuild_prompts()
assert mgr.get_resources() != []
assert mgr.get_prompts() != []
mgr.shutdown()
assert mgr.get_resources() == []
assert mgr.get_prompts() == []
assert mgr._resource_map == {}
assert mgr._prompt_map == {}
+397
View File
@@ -0,0 +1,397 @@
"""Integration tests for MCPClientManager data flow.
Uses real storage (SQLite) and real MCPClientManager state manipulation,
but mock MCP sessions instead of wire-protocol connections. This validates
the full data pipeline: per-server data -> rebuild -> merged state ->
query methods -> storage sync -> shutdown cleanup.
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_resource(
uri: str, name: str, server: str, description: str = "", mime: str = "text/plain"
) -> dict[str, Any]:
return {
"uri": uri,
"name": name,
"description": description,
"mimeType": mime,
"server": server,
}
def _make_prompt(
prefixed_name: str,
original_name: str,
server: str,
description: str = "",
arguments: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
return {
"name": prefixed_name,
"original_name": original_name,
"server": server,
"description": description,
"arguments": arguments or [],
}
def _make_mock_session(
read_resource_result: Any = None,
get_prompt_result: Any = None,
) -> AsyncMock:
"""Build a mock ClientSession with configurable async return values."""
session = AsyncMock()
if read_resource_result is not None:
session.read_resource.return_value = read_resource_result
else:
# Default: single text content
content_item = MagicMock()
content_item.text = "resource content"
result = MagicMock()
result.contents = [content_item]
session.read_resource.return_value = result
if get_prompt_result is not None:
session.get_prompt.return_value = get_prompt_result
else:
msg = MagicMock()
msg.role = "user"
msg.content = MagicMock()
msg.content.text = "Hello, World!"
result = MagicMock()
result.messages = [msg]
session.get_prompt.return_value = result
return session
# ---------------------------------------------------------------------------
# Integration test class
# ---------------------------------------------------------------------------
class TestFullLifecycleResourcesPrompts:
"""Integration test exercising real code paths with real SQLite storage
but mock MCP sessions.
Validates the complete data flow: per-server data population, rebuild
merging, query methods, resource/prompt dispatch through asyncio, storage
sync, and shutdown cleanup.
"""
@pytest.fixture()
def mgr(self) -> MCPClientManager:
"""Create an MCPClientManager with no server configs (no start())."""
return MCPClientManager({})
@pytest.fixture()
def db(self, tmp_path) -> SQLiteBackend:
"""Create a fresh SQLite backend for each test."""
backend = SQLiteBackend(str(tmp_path / "test.db"))
yield backend
backend.close()
def test_rebuild_resources_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_resources merges per-server resources into a unified list."""
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
_make_resource("file:///b.txt", "b", "alpha"),
]
mgr._per_server_resources["beta"] = [
_make_resource("file:///c.txt", "c", "beta"),
]
mgr._rebuild_resources()
resources = mgr.get_resources()
assert len(resources) == 3
uris = {r["uri"] for r in resources}
assert uris == {"file:///a.txt", "file:///b.txt", "file:///c.txt"}
# resource_map should have entries for all non-template resources
assert "file:///a.txt" in mgr._resource_map
assert "file:///c.txt" in mgr._resource_map
assert mgr.resource_count == 3
def test_rebuild_prompts_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_prompts merges per-server prompts into a unified list."""
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._per_server_prompts["beta"] = [
_make_prompt("mcp__beta__summarize", "summarize", "beta", "Summarize text"),
_make_prompt("mcp__beta__translate", "translate", "beta", "Translate text"),
]
mgr._rebuild_prompts()
prompts = mgr.get_prompts()
assert len(prompts) == 3
names = {p["name"] for p in prompts}
assert names == {"mcp__alpha__greet", "mcp__beta__summarize", "mcp__beta__translate"}
# prompt_map should map prefixed -> (server, original)
assert mgr._prompt_map["mcp__alpha__greet"] == ("alpha", "greet")
assert mgr._prompt_map["mcp__beta__summarize"] == ("beta", "summarize")
assert mgr.prompt_count == 3
assert mgr.is_mcp_prompt("mcp__alpha__greet") is True
assert mgr.is_mcp_prompt("nonexistent") is False
def test_read_resource_sync_dispatches_correctly(self, mgr: MCPClientManager) -> None:
"""read_resource_sync dispatches to the correct session via a real asyncio loop."""
# Set up a real event loop in a thread (simulating start())
loop = asyncio.new_event_loop()
import threading
thread = threading.Thread(target=loop.run_forever, daemon=True)
thread.start()
mgr._loop = loop
try:
# Populate session and resource map
session = _make_mock_session()
mgr._sessions["alpha"] = session
mgr._per_server_resources["alpha"] = [
_make_resource("file:///readme.md", "readme", "alpha"),
]
mgr._rebuild_resources()
result = mgr.read_resource_sync("file:///readme.md", timeout=5)
assert result == "resource content"
session.read_resource.assert_awaited_once_with("file:///readme.md")
finally:
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
loop.close()
def test_read_resource_sync_unknown_uri_raises(self, mgr: MCPClientManager) -> None:
"""read_resource_sync raises ValueError for an unknown URI."""
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("file:///nonexistent")
def test_read_resource_via_template(self, mgr: MCPClientManager) -> None:
"""Expanded template URI dispatched to correct server via real asyncio loop."""
loop = asyncio.new_event_loop()
import threading
thread = threading.Thread(target=loop.run_forever, daemon=True)
thread.start()
mgr._loop = loop
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
# Register a template resource (no concrete resources)
mgr._per_server_resources["alpha"] = [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "Fetch a row",
"mimeType": "application/json",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
# Template should not be in _resource_map
assert "db://tables/{table}/rows/{id}" not in mgr._resource_map
# But expanded URI should resolve via prefix matching
result = mgr.read_resource_sync("db://tables/users/rows/42", timeout=5)
assert result == "resource content"
session.read_resource.assert_awaited_once_with("db://tables/users/rows/42")
finally:
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
loop.close()
def test_get_prompt_sync_dispatches_correctly(self, mgr: MCPClientManager) -> None:
"""get_prompt_sync dispatches to the correct session via a real asyncio loop."""
loop = asyncio.new_event_loop()
import threading
thread = threading.Thread(target=loop.run_forever, daemon=True)
thread.start()
mgr._loop = loop
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._rebuild_prompts()
messages = mgr.get_prompt_sync(
"mcp__alpha__greet", arguments={"name": "World"}, timeout=5
)
assert len(messages) == 1
assert messages[0]["role"] == "user"
assert messages[0]["content"] == "Hello, World!"
session.get_prompt.assert_awaited_once_with("greet", arguments={"name": "World"})
finally:
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
loop.close()
def test_get_prompt_sync_unknown_name_raises(self, mgr: MCPClientManager) -> None:
"""get_prompt_sync raises ValueError for an unknown prompt name."""
with pytest.raises(ValueError, match="Unknown MCP prompt"):
mgr.get_prompt_sync("mcp__nosrv__nope")
def test_sync_prompts_to_storage_creates_templates(
self, mgr: MCPClientManager, db: SQLiteBackend
) -> None:
"""sync_prompts_to_storage creates governance templates in real SQLite."""
mgr.set_storage(db)
mgr._prompts = [
_make_prompt(
"mcp__alpha__greet",
"greet",
"alpha",
"Say hello",
[{"name": "user", "description": "Who to greet", "required": True}],
),
_make_prompt(
"mcp__beta__summarize",
"summarize",
"beta",
"Summarize text",
),
]
# Mark connected so set_storage triggers sync
mgr._connected.set()
# Re-set storage to trigger auto-sync
mgr.set_storage(db)
templates = db.list_prompt_templates()
assert len(templates) == 2
names = {t["name"] for t in templates}
assert names == {"mcp__alpha__greet", "mcp__beta__summarize"}
# Verify details on first template
tpl = db.get_prompt_template_by_name("mcp__alpha__greet")
assert tpl is not None
assert tpl["origin"] == "mcp"
assert tpl["mcp_server"] == "alpha"
assert tpl["readonly"] is True
assert tpl["category"] == "mcp"
assert "user" in tpl["variables"]
def test_sync_prompts_removes_stale_templates(
self, mgr: MCPClientManager, db: SQLiteBackend
) -> None:
"""sync_prompts_to_storage removes templates whose MCP prompts are gone."""
mgr.set_storage(db)
# Create an initial template via sync
mgr._prompts = [
_make_prompt("mcp__alpha__old", "old", "alpha", "Old prompt"),
]
mgr.sync_prompts_to_storage()
assert len(db.list_prompt_templates()) == 1
# Now the prompt is gone
mgr._prompts = []
result = mgr.sync_prompts_to_storage()
assert result["removed"] == ["mcp__alpha__old"]
assert len(db.list_prompt_templates()) == 0
def test_shutdown_clears_all_state(self, mgr: MCPClientManager) -> None:
"""shutdown() clears sessions, tools, resources, prompts, and listeners."""
# Populate state
mgr._sessions["alpha"] = MagicMock()
mgr._per_server_tools["alpha"] = [
{
"type": "function",
"function": {
"name": "mcp__alpha__search",
"description": "Search",
"parameters": {},
},
}
]
mgr._rebuild_tools()
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha"),
]
mgr._rebuild_prompts()
mgr._listeners.append(lambda: None)
mgr._resource_listeners.append(lambda: None)
mgr._prompt_listeners.append(lambda: None)
# Verify populated
assert len(mgr._sessions) == 1
assert len(mgr._tools) == 1
assert len(mgr._resources) == 2 # 1 concrete + 1 template
assert len(mgr._template_prefixes) == 1
assert len(mgr._prompts) == 1
mgr.shutdown()
assert len(mgr._sessions) == 0
assert len(mgr._tools) == 0
assert len(mgr._tool_map) == 0
assert len(mgr._resources) == 0
assert len(mgr._resource_map) == 0
assert len(mgr._template_prefixes) == 0
assert len(mgr._prompts) == 0
assert len(mgr._prompt_map) == 0
assert len(mgr._listeners) == 0
assert len(mgr._resource_listeners) == 0
assert len(mgr._prompt_listeners) == 0
def test_listener_notifications_fire_on_rebuild(self, mgr: MCPClientManager) -> None:
"""Rebuild methods fire the appropriate listener callbacks."""
tool_fired = []
resource_fired = []
prompt_fired = []
mgr.add_listener(lambda: tool_fired.append(1))
mgr.add_resource_listener(lambda: resource_fired.append(1))
mgr.add_prompt_listener(lambda: prompt_fired.append(1))
mgr._per_server_tools["alpha"] = []
mgr._rebuild_tools()
assert len(tool_fired) == 1
mgr._per_server_resources["alpha"] = [
_make_resource("file:///x.txt", "x", "alpha"),
]
mgr._rebuild_resources()
assert len(resource_fired) == 1
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__p1", "p1", "alpha"),
]
mgr._rebuild_prompts()
assert len(prompt_fired) == 1
# Tool and resource listeners should not have been fired again
assert len(tool_fired) == 1
assert len(resource_fired) == 1
+272
View File
@@ -0,0 +1,272 @@
"""Tests for MCP prompt → governance template sync and readonly API guards."""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from turnstone.core.mcp_client import MCPClientManager
@pytest.fixture()
def mgr() -> MCPClientManager:
"""Create an MCPClientManager with no real servers (no start())."""
return MCPClientManager({})
def _make_storage() -> MagicMock:
"""Create a mock storage backend with prompt template methods."""
storage = MagicMock()
storage.get_prompt_template_by_name.return_value = None
storage.list_prompt_templates_by_origin.return_value = []
storage.create_prompt_template.return_value = None
storage.update_prompt_template.return_value = True
storage.delete_prompt_template.return_value = True
return storage
class TestSyncPromptsToStorage:
def test_sync_no_storage(self, mgr: MCPClientManager) -> None:
"""Without storage set, sync returns empty stats."""
result = mgr.sync_prompts_to_storage()
assert result == {"added": [], "removed": [], "skipped": []}
def test_sync_creates_mcp_templates(self, mgr: MCPClientManager) -> None:
"""New MCP prompts are created as templates."""
storage = _make_storage()
mgr.set_storage(storage)
# Populate internal prompts list directly
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Say hello",
"arguments": [
{"name": "name", "description": "Who to greet", "required": True},
],
},
]
result = mgr.sync_prompts_to_storage()
assert result["added"] == ["mcp__test__greeting"]
assert result["removed"] == []
assert result["skipped"] == []
storage.create_prompt_template.assert_called_once()
call_kwargs = storage.create_prompt_template.call_args
assert call_kwargs[1]["name"] == "mcp__test__greeting"
assert call_kwargs[1]["origin"] == "mcp"
assert call_kwargs[1]["mcp_server"] == "test"
assert call_kwargs[1]["readonly"] is True
assert call_kwargs[1]["category"] == "mcp"
assert '"name"' in call_kwargs[1]["variables"]
def test_sync_skips_manual_overrides(self, mgr: MCPClientManager) -> None:
"""A manual template with the same name is not overwritten."""
storage = _make_storage()
storage.get_prompt_template_by_name.return_value = {
"template_id": "existing-id",
"name": "mcp__test__greeting",
"origin": "manual",
"readonly": False,
}
mgr.set_storage(storage)
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Say hello",
"arguments": [],
},
]
result = mgr.sync_prompts_to_storage()
assert result["skipped"] == ["mcp__test__greeting"]
assert result["added"] == []
storage.create_prompt_template.assert_not_called()
storage.update_prompt_template.assert_not_called()
def test_sync_updates_existing_mcp_template(self, mgr: MCPClientManager) -> None:
"""An existing MCP template gets its content/variables updated."""
storage = _make_storage()
storage.get_prompt_template_by_name.return_value = {
"template_id": "existing-id",
"name": "mcp__test__greeting",
"origin": "mcp",
"mcp_server": "test",
"readonly": True,
}
mgr.set_storage(storage)
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Updated description",
"arguments": [
{"name": "user", "description": "The user", "required": False},
],
},
]
result = mgr.sync_prompts_to_storage()
assert result["added"] == []
assert result["skipped"] == []
storage.create_prompt_template.assert_not_called()
storage.update_prompt_template.assert_called_once()
call_args = storage.update_prompt_template.call_args
assert call_args[0][0] == "existing-id"
assert "Updated description" in call_args[1]["content"]
assert "user" in call_args[1]["variables"]
# Security: is_default must be reset to prevent compromised MCP server
# from injecting content into a previously admin-promoted default
assert call_args[1]["is_default"] is False
def test_sync_resets_is_default_on_promoted_template(self, mgr: MCPClientManager) -> None:
"""An MCP template promoted to default by admin gets is_default reset on sync."""
storage = _make_storage()
storage.get_prompt_template_by_name.return_value = {
"template_id": "promoted-id",
"name": "mcp__test__greeting",
"origin": "mcp",
"mcp_server": "test",
"readonly": True,
"is_default": True, # admin toggled this
}
mgr.set_storage(storage)
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Potentially compromised content",
"arguments": [],
},
]
mgr.sync_prompts_to_storage()
call_args = storage.update_prompt_template.call_args
assert call_args[1]["is_default"] is False
def test_sync_removes_deleted_prompts(self, mgr: MCPClientManager) -> None:
"""MCP templates in storage with no matching prompt are deleted."""
storage = _make_storage()
storage.list_prompt_templates_by_origin.return_value = [
{
"template_id": "old-id",
"name": "mcp__test__old_prompt",
"origin": "mcp",
"mcp_server": "test",
},
]
mgr.set_storage(storage)
mgr._prompts = [] # No prompts at all
result = mgr.sync_prompts_to_storage()
assert result["removed"] == ["mcp__test__old_prompt"]
storage.delete_prompt_template.assert_called_once_with("old-id")
class TestSetStorageAutoSync:
"""set_storage() triggers an immediate sync when servers are already connected."""
def test_set_storage_syncs_when_connected(self, mgr) -> None:
storage = _make_storage()
mgr._prompts = [
{
"name": "mcp__srv__p1",
"original_name": "p1",
"server": "srv",
"description": "A prompt",
"arguments": [],
}
]
mgr._connected.set()
mgr.set_storage(storage)
# Should have called create_prompt_template for the discovered prompt
storage.create_prompt_template.assert_called_once()
call_kwargs = storage.create_prompt_template.call_args
assert call_kwargs[1]["name"] == "mcp__srv__p1"
assert call_kwargs[1]["origin"] == "mcp"
def test_set_storage_no_sync_when_not_connected(self, mgr) -> None:
storage = _make_storage()
mgr._prompts = [
{
"name": "mcp__srv__p1",
"original_name": "p1",
"server": "srv",
"description": "A prompt",
"arguments": [],
}
]
# _connected is NOT set
mgr.set_storage(storage)
# Should not have synced
storage.create_prompt_template.assert_not_called()
class TestReadonlyAPIGuards:
"""Test that the console server API guards reject edits to readonly templates."""
@pytest.fixture()
def db(self, tmp_path):
"""Create a fresh SQLite backend for each test."""
from turnstone.core.storage._sqlite import SQLiteBackend
return SQLiteBackend(str(tmp_path / "test.db"))
def test_readonly_guard_update(self, db) -> None:
"""Readonly templates cannot be updated via storage guard logic."""
db.create_prompt_template(
"t1",
"mcp__srv__prompt",
"mcp",
"content",
variables="[]",
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server="srv",
readonly=True,
)
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["readonly"] is True
# Simulate API guard check
assert tpl.get("readonly") is True
def test_readonly_guard_delete(self, db) -> None:
"""Readonly templates are flagged for API-level rejection."""
db.create_prompt_template(
"t1",
"mcp__srv__prompt",
"mcp",
"content",
variables="[]",
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server="srv",
readonly=True,
)
existing = db.get_prompt_template("t1")
assert existing is not None
assert existing.get("readonly") is True
+4 -4
View File
@@ -530,11 +530,11 @@ class TestWorkstreamModelParam:
captured_alias = None
def factory(ui: Any, model_alias: str | None = None) -> Any:
def factory(ui: Any, model_alias: str | None = None, ws_id: str | None = None) -> Any:
nonlocal captured_alias
captured_alias = model_alias
mock_session = MagicMock()
mock_session.session_id = "test123"
mock_session.ws_id = "test123"
return mock_session
mgr = WorkstreamManager(factory)
@@ -544,11 +544,11 @@ class TestWorkstreamModelParam:
def test_create_without_model(self) -> None:
captured_alias = None
def factory(ui: Any, model_alias: str | None = None) -> Any:
def factory(ui: Any, model_alias: str | None = None, ws_id: str | None = None) -> Any:
nonlocal captured_alias
captured_alias = model_alias
mock_session = MagicMock()
mock_session.session_id = "test123"
mock_session.ws_id = "test123"
return mock_session
from turnstone.core.workstream import WorkstreamManager
+338
View File
@@ -0,0 +1,338 @@
"""Tests for the channel gateway HTTP notify endpoint."""
from __future__ import annotations
from unittest.mock import AsyncMock
import pytest
from starlette.testclient import TestClient
from turnstone.channels._http import create_channel_app
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "test.db"))
@pytest.fixture
def mock_adapter():
adapter = AsyncMock()
adapter.channel_type = "discord"
adapter.send = AsyncMock(return_value="msg_001")
return adapter
@pytest.fixture
def no_auth_client(storage, mock_adapter):
"""Client with no auth configured (for fail-closed tests)."""
app = create_channel_app({"discord": mock_adapter}, storage)
return TestClient(app)
@pytest.fixture
def client(storage, mock_adapter):
"""Default client with static auth token configured."""
app = create_channel_app({"discord": mock_adapter}, storage, auth_token="test-secret-token")
return TestClient(app)
@pytest.fixture
def authed_client(storage, mock_adapter):
"""Alias — same as client, for auth-specific test clarity."""
app = create_channel_app({"discord": mock_adapter}, storage, auth_token="test-secret-token")
return TestClient(app)
@pytest.fixture
def jwt_client(storage, mock_adapter):
"""Client with JWT auth configured."""
app = create_channel_app({"discord": mock_adapter}, storage, jwt_secret="a" * 32)
return TestClient(app)
class TestNotifyEndpoint:
def test_health(self, client):
resp = client.get("/health")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
def _headers(self) -> dict[str, str]:
return {"Authorization": "Bearer test-secret-token"}
def test_direct_discord_target(self, client, mock_adapter):
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123456"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert len(results) == 1
assert results[0]["status"] == "sent"
assert results[0]["message_id"] == "msg_001"
mock_adapter.send.assert_called_once_with("123456", "Hello!")
def test_with_title(self, client, mock_adapter):
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123456"},
"message": "Hello!",
"title": "Alert",
},
headers=self._headers(),
)
assert resp.status_code == 200
mock_adapter.send.assert_called_once_with("123456", "**Alert**\nHello!")
def test_username_resolution(self, client, storage, mock_adapter):
# Create a user and link a channel
storage.create_user("u1", "testuser", "Test User", "hash")
storage.create_channel_user("discord", "disc_123", "u1")
resp = client.post(
"/v1/api/notify",
json={
"target": {"username": "testuser"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert len(results) == 1
assert results[0]["status"] == "sent"
mock_adapter.send.assert_called_once_with("disc_123", "Hello!")
def test_unknown_username(self, client):
resp = client.post(
"/v1/api/notify",
json={
"target": {"username": "nobody"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 404
error = resp.json()["error"]
assert "nobody" not in error
assert "not found or has no linked channels" in error
def test_user_no_channels(self, authed_client, storage):
storage.create_user("u1", "testuser", "Test User", "hash")
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"username": "testuser"},
"message": "Hello!",
},
headers={"Authorization": "Bearer test-secret-token"},
)
assert resp.status_code == 404
# Generic message — must not differentiate "not found" vs "no channels"
error = resp.json()["error"]
assert "testuser" not in error
assert "not found or has no linked channels" in error
def test_missing_fields(self, client):
resp = client.post(
"/v1/api/notify",
json={"target": {"username": "x"}},
headers=self._headers(),
)
assert resp.status_code == 400
def test_missing_target(self, client):
resp = client.post(
"/v1/api/notify",
json={"message": "Hello!"},
headers=self._headers(),
)
assert resp.status_code == 400
def test_invalid_target(self, client):
resp = client.post(
"/v1/api/notify",
json={
"target": {"invalid": "field"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 400
def test_no_adapter(self, client, storage):
# App has discord adapter, try email target
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "email", "channel_id": "test@example.com"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert results[0]["status"] == "no_adapter"
def test_adapter_failure(self, client, mock_adapter):
mock_adapter.send.side_effect = RuntimeError("Discord API error")
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123456"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert results[0]["status"] == "failed"
def test_invalid_json(self, client):
resp = client.post(
"/v1/api/notify",
content=b"not json",
headers={
"content-type": "application/json",
"Authorization": "Bearer test-secret-token",
},
)
assert resp.status_code == 400
def test_whitespace_only_message(self, client):
"""Whitespace-only messages should be rejected."""
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": " ",
},
headers=self._headers(),
)
assert resp.status_code == 400
class TestNotifyAuth:
"""Tests for authentication on the /v1/api/notify endpoint."""
def test_reject_when_unconfigured(self, no_auth_client):
"""Requests are rejected (fail closed) when no auth is configured."""
resp = no_auth_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
)
assert resp.status_code == 401
def test_reject_without_token(self, authed_client):
"""Requests without Authorization header are rejected when auth is configured."""
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
)
assert resp.status_code == 401
def test_reject_wrong_token(self, authed_client):
"""Requests with wrong token are rejected."""
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": "Bearer wrong-token"},
)
assert resp.status_code == 401
def test_accept_valid_static_token(self, authed_client, mock_adapter):
"""Requests with correct static token are accepted."""
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": "Bearer test-secret-token"},
)
assert resp.status_code == 200
assert resp.json()["results"][0]["status"] == "sent"
def test_accept_valid_jwt(self, jwt_client, mock_adapter):
"""Requests with a valid JWT for the channel audience are accepted."""
from turnstone.core.auth import JWT_AUD_CHANNEL, create_jwt
token = create_jwt(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret="a" * 32,
audience=JWT_AUD_CHANNEL,
)
resp = jwt_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 200
def test_reject_jwt_wrong_audience(self, jwt_client):
"""JWTs with wrong audience are rejected."""
from turnstone.core.auth import create_jwt
token = create_jwt(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret="a" * 32,
audience="turnstone-server", # wrong audience
)
resp = jwt_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 401
def test_reject_jwt_wrong_secret(self, jwt_client):
"""JWTs signed with wrong secret are rejected."""
from turnstone.core.auth import JWT_AUD_CHANNEL, create_jwt
token = create_jwt(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret="b" * 32, # wrong secret
audience=JWT_AUD_CHANNEL,
)
resp = jwt_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 401
def test_health_bypasses_auth(self, authed_client):
"""Health endpoint is always accessible regardless of auth config."""
resp = authed_client.get("/health")
assert resp.status_code == 200
+618
View File
@@ -0,0 +1,618 @@
"""Tests for the notify tool (prepare + execute) in ChatSession."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
if TYPE_CHECKING:
from turnstone.core.session import ChatSession
def _make_session() -> ChatSession:
"""Create a minimal ChatSession with mocked dependencies."""
from unittest.mock import patch
with (
patch("turnstone.core.memory.register_workstream"),
patch("turnstone.core.session.save_message"),
):
from turnstone.core.session import ChatSession
ui = MagicMock()
session = ChatSession(
client=MagicMock(),
model="test-model",
ui=ui,
instructions=None,
temperature=0.7,
max_tokens=1000,
tool_timeout=30,
)
return session
class TestPrepareNotify:
def test_valid_username_target(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"username": "admin",
},
)
assert "execute" in result
assert result["func_name"] == "notify"
assert result["needs_approval"] is False
assert "@admin" in result["header"]
assert result["username"] == "admin"
assert result["message"] == "Hello!"
def test_valid_direct_target(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"channel_type": "discord",
"channel_id": "123456",
},
)
assert "execute" in result
assert result["channel_type"] == "discord"
assert result["channel_id"] == "123456"
assert "discord:123456" in result["header"]
def test_missing_message(self):
session = _make_session()
result = session._prepare_notify("call_1", {"username": "admin"})
assert "error" in result
assert "message" in result["error"].lower()
def test_empty_message(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "",
"username": "admin",
},
)
assert "error" in result
def test_message_too_long(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "x" * 2001,
"username": "admin",
},
)
assert "error" in result
assert "2000" in result["error"]
def test_both_username_and_direct(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"username": "admin",
"channel_type": "discord",
"channel_id": "123",
},
)
assert "error" in result
assert "both" in result["error"].lower() or "ambiguous" in result["error"].lower()
def test_no_target(self):
session = _make_session()
result = session._prepare_notify("call_1", {"message": "Hello!"})
assert "error" in result
def test_channel_type_without_id(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"channel_type": "discord",
},
)
assert "error" in result
assert "channel_id" in result["error"]
def test_channel_id_without_type(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"channel_id": "123456",
},
)
assert "error" in result
assert "channel_type" in result["error"]
def test_preview_truncated(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "a" * 200,
"username": "admin",
},
)
assert result["preview"].endswith("...")
assert len(result["preview"]) <= 123 # 120 chars + "..."
def test_title_passed_through(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"username": "admin",
"title": "Alert",
},
)
assert result["title"] == "Alert"
class TestExecNotify:
def test_sends_http_to_channel_gateway(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "Alert",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp) as mock_post,
patch.dict("os.environ", {}, clear=False),
):
call_id, msg = session._exec_notify(item)
assert call_id == "call_1"
assert "sent successfully" in msg.lower()
mock_post.assert_called_once()
post_kwargs = mock_post.call_args
assert post_kwargs.kwargs["json"]["target"] == {"username": "admin"}
assert post_kwargs.kwargs["json"]["message"] == "Hello!"
def test_no_services_available(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
# No services registered
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
assert "no channel gateway" in msg.lower()
def test_rate_limit(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp),
patch.dict("os.environ", {}, clear=False),
):
for _i in range(5):
call_id, msg = session._exec_notify(item)
assert "sent successfully" in msg.lower()
# 6th should fail
call_id, msg = session._exec_notify(item)
assert "rate limit" in msg.lower()
def test_rate_limit_not_consumed_on_failure(self, tmp_path):
"""Failed delivery should not consume rate limit slots."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch(
"turnstone.core.session.httpx.post",
side_effect=ConnectionError("refused"),
),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
# All fail — counter should stay at 0
for _i in range(3):
session._exec_notify(item)
assert session._notify_count == 0
def test_counter_on_init(self):
session = _make_session()
assert session._notify_count == 0
def test_http_failure_reported(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "",
"channel_type": "discord",
"channel_id": "999",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch(
"turnstone.core.session.httpx.post",
side_effect=ConnectionError("refused"),
),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
# Error message should be generic (no internal details)
assert "delivery failed" in msg.lower()
assert "refused" not in msg
assert "ch-1" not in msg
def test_first_healthy_only(self, tmp_path):
"""Only the first healthy gateway should receive the request."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
storage.register_service("channel", "ch-2", "http://localhost:8092")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp) as mock_post,
patch.dict("os.environ", {}, clear=False),
):
session._exec_notify(item)
# Should only have been called once (first healthy)
assert mock_post.call_count == 1
def test_ssrf_protection(self, tmp_path):
"""URLs with non-http(s) schemes should be skipped."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
# Register a service with an invalid scheme
storage.register_service("channel", "ch-bad", "ftp://evil.example.com")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post") as mock_post,
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
# httpx.post should never be called for ftp:// URL
mock_post.assert_not_called()
assert "delivery failed" in msg.lower()
def test_retry_on_no_services(self, tmp_path):
"""Retries service lookup when no gateways are initially available."""
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
# First two calls return empty, third returns a service
call_count = 0
def _list_services(stype: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
nonlocal call_count
call_count += 1
if call_count <= 2:
return []
return [
{
"service_type": "channel",
"service_id": "ch-1",
"url": "http://localhost:8091",
"metadata": "{}",
"last_heartbeat": "",
"created": "",
}
]
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
mock_storage = MagicMock()
mock_storage.list_services = _list_services
with (
patch("turnstone.core.session.get_storage", return_value=mock_storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep") as mock_sleep,
):
call_id, msg = session._exec_notify(item)
assert "sent successfully" in msg.lower()
# Should have slept twice (retry delays)
assert mock_sleep.call_count == 2
def test_retry_on_all_gateways_failed(self, tmp_path):
"""Retries when all gateways fail on first attempt but succeed on retry."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
call_count = 0
def _post(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count <= 1:
raise ConnectionError("refused")
resp = MagicMock()
resp.status_code = 200
resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
return resp
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", side_effect=_post),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep") as mock_sleep,
):
call_id, msg = session._exec_notify(item)
assert "sent successfully" in msg.lower()
assert mock_sleep.call_count == 1
def test_no_services_logs_warning(self, tmp_path):
"""Server-side warning is logged when no services are available."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.time.sleep"),
patch("turnstone.core.session.log") as mock_log,
):
session._exec_notify(item)
# Should have logged warnings for retries + final exhaustion
warning_calls = [c for c in mock_log.warning.call_args_list]
assert len(warning_calls) >= 3 # 2 retry warnings + 1 exhaustion
events = [c.args[0] for c in warning_calls]
assert "notify.no_services" in events
assert "notify.no_services_exhausted" in events
def test_all_gateways_failed_logs_warning(self, tmp_path):
"""Server-side warning is logged when all gateways fail."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch(
"turnstone.core.session.httpx.post",
side_effect=ConnectionError("refused"),
),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
patch("turnstone.core.session.log") as mock_log,
):
session._exec_notify(item)
warning_calls = [c for c in mock_log.warning.call_args_list]
events = [c.args[0] for c in warning_calls]
# 2 retry warnings + 1 final failure
assert "notify.all_gateways_failed" in events
assert "notify.delivery_failed" in events
def test_gateway_200_but_no_delivery(self, tmp_path):
"""HTTP 200 with all results failed should not count as success."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "no_adapter"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
assert "delivery failed" in msg.lower()
assert session._notify_count == 0
+1 -1
View File
@@ -27,7 +27,7 @@ class TestServerSpec:
expected = {
"/v1/api/workstreams",
"/v1/api/dashboard",
"/v1/api/sessions",
"/v1/api/workstreams/saved",
"/v1/api/send",
"/v1/api/approve",
"/v1/api/plan",
+393
View File
@@ -0,0 +1,393 @@
"""Tests for prompt template runtime wiring into ChatSession."""
from __future__ import annotations
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession, _render_template
class NullUI:
"""UI adapter that discards all output."""
def on_thinking_start(self):
pass
def on_thinking_stop(self):
pass
def on_reasoning_token(self, text):
pass
def on_content_token(self, text):
pass
def on_stream_end(self):
pass
def approve_tools(self, items):
return True, None
def on_tool_result(self, call_id, name, output):
pass
def on_tool_output_chunk(self, call_id, chunk):
pass
def on_status(self, usage, context_window, effort):
pass
def on_plan_review(self, content):
return ""
def on_info(self, message):
pass
def on_error(self, message):
pass
def on_state_change(self, state):
pass
def on_rename(self, name):
pass
def _make_session(**kwargs):
defaults = dict(
client=MagicMock(),
model="test-model",
ui=NullUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
defaults.update(kwargs)
return ChatSession(**defaults)
def _sys_content(session: ChatSession) -> str:
"""Extract the system message content."""
msgs = [m for m in session.system_messages if m["role"] == "system"]
assert msgs
return msgs[0]["content"]
def _create_template(db, template_id, name, content, is_default=False, **kwargs):
"""Helper to create a prompt template in storage."""
db.create_prompt_template(
template_id=template_id,
name=name,
category=kwargs.get("category", "general"),
content=content,
variables=kwargs.get("variables", "[]"),
is_default=is_default,
org_id=kwargs.get("org_id", ""),
created_by=kwargs.get("created_by", "test"),
origin=kwargs.get("origin", "manual"),
mcp_server=kwargs.get("mcp_server", ""),
readonly=kwargs.get("readonly", False),
)
# ---------------------------------------------------------------------------
# _render_template unit tests
# ---------------------------------------------------------------------------
class TestRenderTemplate:
def test_basic_substitution(self):
result = _render_template("Hello {{name}}", {"name": "world"})
assert result == "Hello world"
def test_multiple_variables(self):
result = _render_template(
"Model: {{model}}, WS: {{ws_id}}", {"model": "gpt-5", "ws_id": "abc123"}
)
assert result == "Model: gpt-5, WS: abc123"
def test_unresolvable_variable_kept(self):
result = _render_template("Hello {{unknown}}", {"model": "gpt-5"})
assert result == "Hello {{unknown}}"
def test_empty_context(self):
result = _render_template("No vars here", {})
assert result == "No vars here"
def test_duplicate_placeholder(self):
result = _render_template("{{x}} and {{x}}", {"x": "val"})
assert result == "val and val"
def test_no_cross_variable_injection(self):
# If model contains {{ws_id}}, it must NOT be expanded
result = _render_template("Model: {{model}}", {"model": "{{ws_id}}", "ws_id": "secret"})
assert result == "Model: {{ws_id}}"
assert "secret" not in result
# ---------------------------------------------------------------------------
# Default templates in system message
# ---------------------------------------------------------------------------
class TestDefaultTemplates:
def test_default_templates_in_system_message(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "alpha", "You are a helpful assistant.", is_default=True)
_create_template(db, "t2", "beta", "Always be concise.", is_default=True)
session = _make_session()
content = _sys_content(session)
assert "You are a helpful assistant." in content
assert "Always be concise." in content
def test_default_templates_ordered_by_name(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t2", "b-template", "SECOND", is_default=True)
_create_template(db, "t1", "a-template", "FIRST", is_default=True)
session = _make_session()
content = _sys_content(session)
first_pos = content.index("FIRST")
second_pos = content.index("SECOND")
assert first_pos < second_pos
def test_no_default_templates(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "alpha", "Not default.", is_default=False)
session = _make_session()
content = _sys_content(session)
assert "Not default." not in content
def test_templates_before_instructions(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "tpl", "TEMPLATE_CONTENT", is_default=True)
session = _make_session(instructions="USER_INSTRUCTIONS")
content = _sys_content(session)
tpl_pos = content.index("TEMPLATE_CONTENT")
instr_pos = content.index("USER_INSTRUCTIONS")
assert tpl_pos < instr_pos
# ---------------------------------------------------------------------------
# Explicit template selection
# ---------------------------------------------------------------------------
class TestExplicitTemplate:
def test_explicit_template_replaces_defaults(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "default-tpl", "DEFAULT_CONTENT", is_default=True)
_create_template(db, "t2", "specific-tpl", "SPECIFIC_CONTENT", is_default=False)
session = _make_session(template="specific-tpl")
content = _sys_content(session)
assert "SPECIFIC_CONTENT" in content
assert "DEFAULT_CONTENT" not in content
def test_explicit_template_not_found(self, tmp_db):
session = _make_session(template="nonexistent")
content = _sys_content(session)
# Graceful degradation — no template content injected
assert "nonexistent" not in content
# ---------------------------------------------------------------------------
# Variable substitution in templates
# ---------------------------------------------------------------------------
class TestTemplateVariables:
def test_model_and_ws_id_substituted(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "vars-tpl", "Model: {{model}}, WS: {{ws_id}}", is_default=True)
session = _make_session()
content = _sys_content(session)
assert "Model: test-model" in content
assert f"WS: {session.ws_id}" in content
def test_node_id_substituted(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "node-tpl", "Node: {{node_id}}", is_default=True)
session = _make_session(node_id="node-42")
content = _sys_content(session)
assert "Node: node-42" in content
def test_unknown_variable_preserved(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "unknown-tpl", "Val: {{unknown_var}}", is_default=True)
session = _make_session()
content = _sys_content(session)
assert "Val: {{unknown_var}}" in content
# ---------------------------------------------------------------------------
# Template persistence and resume
# ---------------------------------------------------------------------------
class TestTemplatePersistence:
def test_template_persisted_in_config(self, tmp_db):
from turnstone.core.memory import load_workstream_config
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "TPL_CONTENT", is_default=False)
session = _make_session(template="my-tpl")
config = load_workstream_config(session.ws_id)
assert config["template"] == "my-tpl"
def test_template_restored_on_resume(self, tmp_db):
from turnstone.core.memory import save_message
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "PERSISTED_TEMPLATE", is_default=False)
# Create session with template, save a message so resume has history
session1 = _make_session(template="my-tpl")
ws_id = session1.ws_id
save_message(ws_id, "user", "hello")
# New session without template, then resume
session2 = _make_session()
assert session2._template_name is None
resumed = session2.resume(ws_id)
assert resumed
assert session2._template_name == "my-tpl"
content = _sys_content(session2)
assert "PERSISTED_TEMPLATE" in content
def test_empty_template_config_means_defaults(self, tmp_db):
from turnstone.core.memory import load_workstream_config
session = _make_session()
config = load_workstream_config(session.ws_id)
assert config["template"] == ""
# ---------------------------------------------------------------------------
# /template slash command
# ---------------------------------------------------------------------------
class TestTemplateSlashCommand:
def test_template_set(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "SLASH_TEMPLATE", is_default=False)
session = _make_session()
content_before = _sys_content(session)
assert "SLASH_TEMPLATE" not in content_before
session.handle_command("/template my-tpl")
assert session._template_name == "my-tpl"
content_after = _sys_content(session)
assert "SLASH_TEMPLATE" in content_after
def test_template_clear(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "EXPLICIT_TEMPLATE", is_default=False)
_create_template(db, "t2", "default-tpl", "DEFAULT_TEMPLATE", is_default=True)
session = _make_session(template="my-tpl")
assert "EXPLICIT_TEMPLATE" in _sys_content(session)
assert "DEFAULT_TEMPLATE" not in _sys_content(session)
session.handle_command("/template clear")
assert session._template_name is None
assert "DEFAULT_TEMPLATE" in _sys_content(session)
assert "EXPLICIT_TEMPLATE" not in _sys_content(session)
def test_template_not_found(self, tmp_db):
ui = NullUI()
ui.on_error = MagicMock()
session = _make_session(ui=ui)
session.handle_command("/template nonexistent")
ui.on_error.assert_called_once()
assert "not found" in ui.on_error.call_args[0][0].lower()
def test_template_show_current(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "content", is_default=False)
ui = NullUI()
ui.on_info = MagicMock()
session = _make_session(ui=ui, template="my-tpl")
session.handle_command("/template")
ui.on_info.assert_called_once()
assert "my-tpl" in ui.on_info.call_args[0][0]
# ---------------------------------------------------------------------------
# MCP-origin templates
# ---------------------------------------------------------------------------
class TestMCPTemplates:
def test_mcp_readonly_template_as_default(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(
db,
"t1",
"mcp__server__prompt",
"MCP_CONTENT",
is_default=True,
origin="mcp",
mcp_server="server",
readonly=True,
)
session = _make_session()
content = _sys_content(session)
assert "MCP_CONTENT" in content
def test_mcp_template_selectable_explicitly(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(
db,
"t1",
"mcp__server__code",
"MCP_EXPLICIT",
is_default=False,
origin="mcp",
mcp_server="server",
readonly=True,
)
session = _make_session(template="mcp__server__code")
content = _sys_content(session)
assert "MCP_EXPLICIT" in content
+16
View File
@@ -8,6 +8,7 @@ from turnstone.mq.protocol import (
AckEvent,
ApprovalRequestEvent,
ApproveMessage,
CancelMessage,
CloseWorkstreamMessage,
CommandMessage,
ContentEvent,
@@ -68,6 +69,7 @@ INBOUND_TYPES = [
(ListWorkstreamsMessage, {}),
(HealthMessage, {}),
(ListNodesMessage, {}),
(CancelMessage, {"ws_id": "abc"}),
]
@@ -207,6 +209,20 @@ def test_create_workstream_target_node():
assert restored.name == "debug-ws"
def test_create_workstream_template_field():
msg = CreateWorkstreamMessage(name="ws", template="code-review")
assert msg.template == "code-review"
raw = msg.to_json()
restored = InboundMessage.from_json(raw)
assert isinstance(restored, CreateWorkstreamMessage)
assert restored.template == "code-review"
def test_create_workstream_template_default_empty():
msg = CreateWorkstreamMessage(name="ws")
assert msg.template == ""
def test_list_nodes_round_trip():
msg = ListNodesMessage()
raw = msg.to_json()
+288
View File
@@ -1103,6 +1103,43 @@ class TestOpenAIParameterGating:
assert "temperature" not in kwargs
assert "reasoning_effort" not in kwargs
def test_gpt5_pro_unsupported_effort_falls_back(self) -> None:
"""GPT-5 pro only supports 'high'; unsupported values fall back to default."""
caps = self.provider.get_capabilities("gpt-5-pro")
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="medium")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "high" # fell back to default
def test_gpt5_pro_supported_effort_passes_through(self) -> None:
"""GPT-5 pro accepts 'high' directly."""
caps = self.provider.get_capabilities("gpt-5-pro")
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="high")
assert kwargs["reasoning_effort"] == "high"
def test_gpt54_1m_context_and_effort(self) -> None:
"""GPT-5.4: 1M context, temperature when effort=none, xhigh supported."""
caps = self.provider.get_capabilities("gpt-5.4")
assert caps.context_window == 1050000
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="none")
assert kwargs["temperature"] == 0.7
assert "reasoning_effort" not in kwargs
kwargs2: dict[str, Any] = {}
self.provider._apply_model_params(kwargs2, caps, temperature=0.7, reasoning_effort="xhigh")
assert "temperature" not in kwargs2
assert kwargs2["reasoning_effort"] == "xhigh"
def test_gpt54_pro_no_temperature_always_reasoning(self) -> None:
"""GPT-5.4 pro: no temperature, medium/high/xhigh only."""
caps = self.provider.get_capabilities("gpt-5.4-pro")
assert caps.context_window == 1050000
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="low")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "medium" # fell back from unsupported "low"
class TestAnthropicReasoningNone:
"""Verify 'none' effort disables thinking for manual-thinking models."""
@@ -1896,3 +1933,254 @@ class TestAnthropicProviderBlocks:
assert blocks[1]["input"] == {"query": "test"} # parsed from accumulated JSON
assert blocks[2]["type"] == "web_search_tool_result"
assert blocks[2]["encrypted_content"] == "enc_data"
# ---------------------------------------------------------------------------
# Tool search tests
# ---------------------------------------------------------------------------
class TestAnthropicToolSearch:
"""Test Anthropic provider tool search injection."""
@pytest.fixture()
def provider(self):
from turnstone.core.providers._anthropic import AnthropicProvider
return AnthropicProvider()
def test_tool_search_capability_flag(self, provider):
caps = provider.get_capabilities("claude-opus-4-6-20260101")
assert caps.supports_tool_search is True
def test_tool_search_not_supported_on_haiku(self, provider):
caps = provider.get_capabilities("claude-haiku-4-5-20251001")
assert caps.supports_tool_search is False
def test_inject_tool_search_marks_deferred(self, provider):
caps = provider.get_capabilities("claude-opus-4-6-20260101")
tools = [
{"name": "bash", "description": "Run commands", "input_schema": {}},
{
"name": "mcp__github__create_issue",
"description": "Create issue",
"input_schema": {},
},
]
deferred = frozenset(["mcp__github__create_issue"])
result = provider._inject_tool_search(tools, caps, deferred)
# bash should not be deferred
assert result[0].get("defer_loading") is None or result[0].get("defer_loading") is False
# MCP tool should be deferred
assert result[1]["defer_loading"] is True
# Search tool should be appended
assert result[-1]["type"] == "tool_search_tool_bm25_20251119"
assert result[-1]["name"] == "tool_search"
def test_inject_tool_search_no_op_without_deferred(self, provider):
caps = provider.get_capabilities("claude-opus-4-6-20260101")
tools = [{"name": "bash", "description": "Run commands", "input_schema": {}}]
result = provider._inject_tool_search(tools, caps, None)
assert result == tools
def test_inject_tool_search_no_op_on_unsupported_model(self, provider):
caps = provider.get_capabilities("claude-haiku-4-5-20251001")
tools = [{"name": "bash", "description": "Run commands", "input_schema": {}}]
deferred = frozenset(["some_tool"])
result = provider._inject_tool_search(tools, caps, deferred)
assert result == tools
class TestOpenAIToolSearch:
"""Test OpenAI provider tool search injection."""
@pytest.fixture()
def provider(self):
return OpenAIProvider()
def test_tool_search_capability_on_gpt54(self, provider):
caps = provider.get_capabilities("gpt-5.4")
assert caps.supports_tool_search is True
def test_tool_search_not_supported_on_gpt5(self, provider):
caps = provider.get_capabilities("gpt-5")
assert caps.supports_tool_search is False
def test_apply_tool_search_marks_deferred(self, provider):
caps = provider.get_capabilities("gpt-5.4")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
{
"type": "function",
"function": {"name": "mcp__slack__send", "description": "Send message"},
},
]
deferred = frozenset(["mcp__slack__send"])
result = provider._apply_tool_search(caps, tools, deferred)
assert result is not None
# bash not deferred
assert result[0].get("defer_loading") is None or result[0].get("defer_loading") is False
# slack tool deferred
assert result[1]["defer_loading"] is True
def test_apply_tool_search_no_op_without_deferred(self, provider):
caps = provider.get_capabilities("gpt-5.4")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
]
result = provider._apply_tool_search(caps, tools, None)
assert result == tools
def test_apply_tool_search_no_op_on_unsupported_model(self, provider):
caps = provider.get_capabilities("gpt-5")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
]
deferred = frozenset(["some_tool"])
result = provider._apply_tool_search(caps, tools, deferred)
assert result == tools
class TestModelCapabilitiesToolSearch:
"""Test supports_tool_search defaults and values."""
def test_default_is_false(self):
from turnstone.core.providers._protocol import ModelCapabilities
caps = ModelCapabilities()
assert caps.supports_tool_search is False
# ---------------------------------------------------------------------------
# Vision support
# ---------------------------------------------------------------------------
class TestVisionCapabilities:
"""Test supports_vision flag across providers."""
def test_default_is_false(self) -> None:
from turnstone.core.providers._protocol import ModelCapabilities
caps = ModelCapabilities()
assert caps.supports_vision is False
def test_openai_commercial_supports_vision(self) -> None:
provider = OpenAIProvider()
for model in ("gpt-5", "gpt-5-mini", "gpt-5.4", "o3", "o4-mini"):
caps = provider.get_capabilities(model)
assert caps.supports_vision is True, f"{model} should support vision"
def test_openai_default_no_vision(self) -> None:
"""Unknown models (local servers) default to no vision."""
provider = OpenAIProvider()
caps = provider.get_capabilities("some-local-model")
assert caps.supports_vision is False
def test_anthropic_supports_vision(self) -> None:
from turnstone.core.providers._anthropic import AnthropicProvider
provider = AnthropicProvider()
for model in ("claude-opus-4-6", "claude-sonnet-4-6", "claude-haiku-4-5"):
caps = provider.get_capabilities(model)
assert caps.supports_vision is True, f"{model} should support vision"
def test_anthropic_default_supports_vision(self) -> None:
"""Anthropic default (unknown Claude model) supports vision."""
from turnstone.core.providers._anthropic import AnthropicProvider
provider = AnthropicProvider()
caps = provider.get_capabilities("claude-unknown-9")
assert caps.supports_vision is True
class TestAnthropicVisionConversion:
"""Test image content conversion in _convert_messages."""
def setup_method(self) -> None:
from turnstone.core.providers._anthropic import AnthropicProvider
self.provider = AnthropicProvider()
def test_tool_result_with_image_content(self) -> None:
"""Tool result with list content converts image_url to Anthropic image."""
messages = [
{"role": "user", "content": "Read this image"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_1",
"function": {"name": "read_file", "arguments": '{"path": "img.png"}'},
}
],
},
{
"role": "tool",
"tool_call_id": "call_1",
"content": [
{"type": "text", "text": "Image file: img.png (1024 bytes)"},
{
"type": "image_url",
"image_url": {"url": "data:image/png;base64,iVBORw0KGgo="},
},
],
},
]
_, converted = self.provider._convert_messages(messages)
# Tool result should be in a user message
tool_user_msg = converted[2]
assert tool_user_msg["role"] == "user"
tool_result = tool_user_msg["content"][0]
assert tool_result["type"] == "tool_result"
assert tool_result["tool_use_id"] == "call_1"
# Content should be a list with converted image block
content = tool_result["content"]
assert isinstance(content, list)
assert content[0] == {"type": "text", "text": "Image file: img.png (1024 bytes)"}
assert content[1]["type"] == "image"
assert content[1]["source"]["type"] == "base64"
assert content[1]["source"]["media_type"] == "image/png"
assert content[1]["source"]["data"] == "iVBORw0KGgo="
def test_tool_result_with_string_content_unchanged(self) -> None:
"""Tool result with plain string content is unchanged."""
messages = [
{"role": "user", "content": "Read file"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "call_2",
"function": {"name": "read_file", "arguments": '{"path": "f.py"}'},
}
],
},
{
"role": "tool",
"tool_call_id": "call_2",
"content": " 1\tprint('hello')",
},
]
_, converted = self.provider._convert_messages(messages)
tool_result = converted[2]["content"][0]
assert tool_result["content"] == " 1\tprint('hello')"
def test_convert_content_parts_static_method(self) -> None:
"""_convert_content_parts handles both image_url and text."""
from turnstone.core.providers._anthropic import AnthropicProvider
parts = [
{"type": "text", "text": "description"},
{
"type": "image_url",
"image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ"},
},
]
result = AnthropicProvider._convert_content_parts(parts)
assert result[0] == {"type": "text", "text": "description"}
assert result[1]["type"] == "image"
assert result[1]["source"]["media_type"] == "image/jpeg"
assert result[1]["source"]["data"] == "/9j/4AAQ"
+97
View File
@@ -0,0 +1,97 @@
"""Tests for the atomic workstream resumption flow.
Covers CreateWorkstreamMessage resume_ws field, WorkstreamResumedEvent,
WorkstreamCreatedEvent resumed fields, and server endpoint handling.
"""
from __future__ import annotations
import json
from turnstone.mq.protocol import (
CreateWorkstreamMessage,
WorkstreamCreatedEvent,
WorkstreamResumedEvent,
)
# ---------------------------------------------------------------------------
# Protocol tests
# ---------------------------------------------------------------------------
class TestCreateWorkstreamMessageResumeField:
def test_resume_ws_defaults_empty(self) -> None:
msg = CreateWorkstreamMessage(name="test")
assert msg.resume_ws == ""
def test_resume_ws_set(self) -> None:
msg = CreateWorkstreamMessage(name="test", resume_ws="ws-abc")
assert msg.resume_ws == "ws-abc"
def test_resume_ws_serializes(self) -> None:
msg = CreateWorkstreamMessage(resume_ws="ws-xyz")
data = json.loads(msg.to_json())
assert data["resume_ws"] == "ws-xyz"
def test_resume_ws_deserializes(self) -> None:
msg = CreateWorkstreamMessage(resume_ws="ws-123")
raw = msg.to_json()
from turnstone.mq.protocol import InboundMessage
restored = InboundMessage.from_json(raw)
assert getattr(restored, "resume_ws", "") == "ws-123"
class TestWorkstreamCreatedEventResumeFields:
def test_default_not_resumed(self) -> None:
event = WorkstreamCreatedEvent(ws_id="ws-1", name="test")
assert event.resumed is False
assert event.message_count == 0
def test_resumed_fields(self) -> None:
event = WorkstreamCreatedEvent(ws_id="ws-1", name="test", resumed=True, message_count=42)
assert event.resumed is True
assert event.message_count == 42
def test_serializes_resumed_fields(self) -> None:
event = WorkstreamCreatedEvent(ws_id="ws-1", resumed=True, message_count=10)
data = json.loads(event.to_json())
assert data["resumed"] is True
assert data["message_count"] == 10
def test_deserializes_resumed_fields(self) -> None:
event = WorkstreamCreatedEvent(ws_id="ws-1", resumed=True, message_count=5)
from turnstone.mq.protocol import OutboundEvent
restored = OutboundEvent.from_json(event.to_json())
assert isinstance(restored, WorkstreamCreatedEvent)
assert restored.resumed is True
assert restored.message_count == 5
class TestWorkstreamResumedEvent:
def test_defaults(self) -> None:
event = WorkstreamResumedEvent(ws_id="ws-1")
assert event.type == "ws_resumed"
assert event.message_count == 0
assert event.name == ""
def test_with_values(self) -> None:
event = WorkstreamResumedEvent(ws_id="ws-1", message_count=25, name="My Chat")
assert event.message_count == 25
assert event.name == "My Chat"
def test_round_trip(self) -> None:
event = WorkstreamResumedEvent(ws_id="ws-1", message_count=10, name="Chat")
from turnstone.mq.protocol import OutboundEvent
restored = OutboundEvent.from_json(event.to_json())
assert isinstance(restored, WorkstreamResumedEvent)
assert restored.message_count == 10
assert restored.name == "Chat"
def test_registered_in_outbound_registry(self) -> None:
from turnstone.mq.protocol import _OUTBOUND_REGISTRY
assert "ws_resumed" in _OUTBOUND_REGISTRY
assert _OUTBOUND_REGISTRY["ws_resumed"] is WorkstreamResumedEvent
+285
View File
@@ -0,0 +1,285 @@
"""Tests for scheduled task admin API endpoints."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import pytest
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
if TYPE_CHECKING:
from starlette.requests import Request
from starlette.responses import Response
from turnstone.console.server import (
admin_create_schedule,
admin_delete_schedule,
admin_get_schedule,
admin_list_schedule_runs,
admin_list_schedules,
admin_update_schedule,
)
from turnstone.core.auth import AuthResult
from turnstone.core.storage._sqlite import SQLiteBackend
class _InjectAuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next: Any) -> Response:
request.state.auth_result = AuthResult(
user_id="test-admin",
scopes=frozenset({"approve"}),
token_source="config",
permissions=frozenset({"admin.schedules"}),
)
return await call_next(request)
@pytest.fixture
def storage(tmp_path):
"""Fresh SQLite backend for each test."""
return SQLiteBackend(str(tmp_path / "test.db"))
@pytest.fixture
def client(storage):
"""TestClient with storage and auth bypassed."""
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/admin/schedules", admin_list_schedules),
Route("/api/admin/schedules", admin_create_schedule, methods=["POST"]),
Route("/api/admin/schedules/{task_id}", admin_get_schedule),
Route(
"/api/admin/schedules/{task_id}",
admin_update_schedule,
methods=["PUT"],
),
Route(
"/api/admin/schedules/{task_id}",
admin_delete_schedule,
methods=["DELETE"],
),
Route(
"/api/admin/schedules/{task_id}/runs",
admin_list_schedule_runs,
),
],
),
],
middleware=[Middleware(_InjectAuthMiddleware)],
)
app.state.auth_storage = storage
return TestClient(app)
def _cron_payload(**overrides):
"""Build default cron schedule creation payload."""
defaults = {
"name": "Daily report",
"description": "Generate the summary",
"schedule_type": "cron",
"cron_expr": "0 9 * * *",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Generate the daily report",
}
defaults.update(overrides)
return defaults
def _at_payload(**overrides):
"""Build default at-time schedule creation payload."""
defaults = {
"name": "One-shot task",
"description": "Run once",
"schedule_type": "at",
"at_time": "2099-01-01T00:00:00+00:00",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Do the thing",
}
defaults.update(overrides)
return defaults
class TestScheduleAPI:
"""Tests for the 6 admin schedule endpoints."""
def test_list_empty(self, client):
resp = client.get("/v1/api/admin/schedules")
assert resp.status_code == 200
data = resp.json()
assert data["schedules"] == []
def test_create_cron(self, client):
resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
assert resp.status_code == 200
task = resp.json()
assert task["name"] == "Daily report"
assert task["schedule_type"] == "cron"
assert task["cron_expr"] == "0 9 * * *"
assert task["enabled"] is True
assert "task_id" in task
assert "created" in task
assert "next_run" in task
assert task["next_run"] != ""
def test_create_at(self, client):
resp = client.post("/v1/api/admin/schedules", json=_at_payload())
assert resp.status_code == 200
task = resp.json()
assert task["schedule_type"] == "at"
assert task["at_time"] == "2099-01-01T00:00:00+00:00"
assert task["next_run"] == "2099-01-01T00:00:00+00:00"
def test_create_missing_name(self, client):
payload = _cron_payload()
del payload["name"]
resp = client.post("/v1/api/admin/schedules", json=payload)
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
def test_create_invalid_cron(self, client):
resp = client.post(
"/v1/api/admin/schedules",
json=_cron_payload(cron_expr="not a cron"),
)
assert resp.status_code == 400
assert "cron" in resp.json()["error"].lower()
def test_create_naive_at_time(self, client):
"""Naive timestamps (no timezone) should be rejected."""
resp = client.post(
"/v1/api/admin/schedules",
json=_at_payload(at_time="2099-01-01T00:00:00"),
)
assert resp.status_code == 400
assert "timezone" in resp.json()["error"].lower()
def test_create_past_at_time(self, client):
resp = client.post(
"/v1/api/admin/schedules",
json=_at_payload(at_time="2000-01-01T00:00:00+00:00"),
)
assert resp.status_code == 400
assert "future" in resp.json()["error"].lower()
def test_get_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.get(f"/v1/api/admin/schedules/{task_id}")
assert resp.status_code == 200
assert resp.json()["task_id"] == task_id
assert resp.json()["name"] == "Daily report"
def test_get_nonexistent(self, client):
resp = client.get("/v1/api/admin/schedules/nonexistent_id")
assert resp.status_code == 404
def test_update_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"name": "Weekly report"},
)
assert resp.status_code == 200
assert resp.json()["name"] == "Weekly report"
# Verify via GET
get_resp = client.get(f"/v1/api/admin/schedules/{task_id}")
assert get_resp.json()["name"] == "Weekly report"
def test_update_nonexistent(self, client):
resp = client.put(
"/v1/api/admin/schedules/nonexistent_id",
json={"name": "Nope"},
)
assert resp.status_code == 404
def test_delete_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.delete(f"/v1/api/admin/schedules/{task_id}")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify gone
get_resp = client.get(f"/v1/api/admin/schedules/{task_id}")
assert get_resp.status_code == 404
def test_delete_nonexistent(self, client):
resp = client.delete("/v1/api/admin/schedules/nonexistent_id")
assert resp.status_code == 404
def test_list_runs_empty(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.get(f"/v1/api/admin/schedules/{task_id}/runs")
assert resp.status_code == 200
assert resp.json()["runs"] == []
def test_list_runs_nonexistent(self, client):
resp = client.get("/v1/api/admin/schedules/nonexistent_id/runs")
assert resp.status_code == 404
def test_create_specific_node_target(self, client):
payload = _cron_payload(target_mode="node-custom-001")
resp = client.post("/v1/api/admin/schedules", json=payload)
assert resp.status_code == 200
data = resp.json()
assert data["target_mode"] == "node-custom-001"
def test_list_runs_with_data(self, client, storage):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
# Record runs directly in storage
storage.record_task_run(
run_id="run_001",
task_id=task_id,
node_id="node-1",
ws_id="ws_abc",
correlation_id="corr_001",
started="2025-06-01T09:00:00",
status="dispatched",
error="",
)
storage.record_task_run(
run_id="run_002",
task_id=task_id,
node_id="node-2",
ws_id="",
correlation_id="corr_002",
started="2025-06-01T09:01:00",
status="failed",
error="No reachable nodes",
)
resp = client.get(f"/v1/api/admin/schedules/{task_id}/runs")
assert resp.status_code == 200
runs = resp.json()["runs"]
assert len(runs) == 2
# Most recent first
assert runs[0]["run_id"] == "run_002"
assert runs[0]["status"] == "failed"
assert runs[1]["run_id"] == "run_001"
def test_list_runs_invalid_limit(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
# Invalid limit should not crash — falls back to 50
resp = client.get(f"/v1/api/admin/schedules/{task_id}/runs?limit=abc")
assert resp.status_code == 200
assert resp.json()["runs"] == []
+259
View File
@@ -0,0 +1,259 @@
"""Tests for scheduled_tasks and scheduled_task_runs storage CRUD."""
from __future__ import annotations
import time
import pytest
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def db(tmp_path):
"""Fresh SQLite backend for each test."""
backend = SQLiteBackend(str(tmp_path / "test.db"))
return backend
def _make_task_kwargs(**overrides):
"""Build default kwargs for create_scheduled_task."""
defaults = {
"task_id": "task_001",
"name": "Daily report",
"description": "Generate the daily summary",
"schedule_type": "cron",
"cron_expr": "0 9 * * *",
"at_time": "",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Generate the daily report",
"auto_approve": False,
"auto_approve_tools": [],
"created_by": "u_admin",
"next_run": "2099-01-01T09:00:00",
}
defaults.update(overrides)
return defaults
class TestScheduledTaskCRUD:
"""Tests for scheduled_tasks table operations."""
def test_create_and_get(self, db):
db.create_scheduled_task(**_make_task_kwargs())
result = db.get_scheduled_task("task_001")
assert result is not None
assert result["task_id"] == "task_001"
assert result["name"] == "Daily report"
assert result["description"] == "Generate the daily summary"
assert result["schedule_type"] == "cron"
assert result["cron_expr"] == "0 9 * * *"
assert result["at_time"] == ""
assert result["target_mode"] == "auto"
assert result["model"] == "gpt-5"
assert result["initial_message"] == "Generate the daily report"
assert result["auto_approve"] == 0
assert result["auto_approve_tools"] == ""
assert result["enabled"] == 1
assert result["created_by"] == "u_admin"
assert result["next_run"] == "2099-01-01T09:00:00"
assert "created" in result
assert "updated" in result
def test_get_nonexistent(self, db):
assert db.get_scheduled_task("no_such_task") is None
def test_create_duplicate_noop(self, db):
db.create_scheduled_task(**_make_task_kwargs(name="First"))
db.create_scheduled_task(**_make_task_kwargs(name="Second"))
result = db.get_scheduled_task("task_001")
assert result is not None
assert result["name"] == "First" # first write wins
def test_list_tasks(self, db):
db.create_scheduled_task(**_make_task_kwargs(task_id="task_a", name="Alpha"))
# Ensure different created timestamps (resolution is 1 second)
time.sleep(1.1)
db.create_scheduled_task(**_make_task_kwargs(task_id="task_b", name="Beta"))
tasks = db.list_scheduled_tasks()
assert len(tasks) == 2
# Ordered by created DESC — most recent first
assert tasks[0]["task_id"] == "task_b"
assert tasks[1]["task_id"] == "task_a"
def test_update_task(self, db):
db.create_scheduled_task(**_make_task_kwargs())
original = db.get_scheduled_task("task_001")
assert original is not None
original_updated = original["updated"]
time.sleep(0.05)
result = db.update_scheduled_task("task_001", name="Weekly report")
assert result is True
updated = db.get_scheduled_task("task_001")
assert updated is not None
assert updated["name"] == "Weekly report"
assert updated["updated"] >= original_updated
def test_update_enable_disable(self, db):
db.create_scheduled_task(**_make_task_kwargs())
task = db.get_scheduled_task("task_001")
assert task is not None
assert task["enabled"] == 1
db.update_scheduled_task("task_001", enabled=False)
task = db.get_scheduled_task("task_001")
assert task is not None
assert task["enabled"] == 0
db.update_scheduled_task("task_001", enabled=True)
task = db.get_scheduled_task("task_001")
assert task is not None
assert task["enabled"] == 1
def test_delete_task(self, db):
db.create_scheduled_task(**_make_task_kwargs())
assert db.delete_scheduled_task("task_001") is True
assert db.get_scheduled_task("task_001") is None
# Deleting again returns False
assert db.delete_scheduled_task("task_001") is False
def test_delete_cascades_runs(self, db):
db.create_scheduled_task(**_make_task_kwargs())
db.record_task_run(
run_id="run_001",
task_id="task_001",
node_id="node_1",
ws_id="ws_abc",
correlation_id="corr_001",
started="2025-01-01T09:00:00",
status="dispatched",
error="",
)
assert len(db.list_task_runs("task_001")) == 1
db.delete_scheduled_task("task_001")
assert db.list_task_runs("task_001") == []
def test_list_due_tasks(self, db):
db.create_scheduled_task(
**_make_task_kwargs(task_id="past", next_run="2020-01-01T00:00:00")
)
db.create_scheduled_task(
**_make_task_kwargs(task_id="future", next_run="2099-12-31T23:59:59")
)
now = "2025-06-01T12:00:00"
due = db.list_due_tasks(now)
assert len(due) == 1
assert due[0]["task_id"] == "past"
def test_list_due_tasks_skips_disabled(self, db):
db.create_scheduled_task(
**_make_task_kwargs(task_id="disabled_task", next_run="2020-01-01T00:00:00")
)
db.update_scheduled_task("disabled_task", enabled=False)
due = db.list_due_tasks("2025-06-01T12:00:00")
assert len(due) == 0
def test_list_due_tasks_empty_next_run(self, db):
db.create_scheduled_task(**_make_task_kwargs(task_id="empty_next", next_run=""))
due = db.list_due_tasks("2099-12-31T23:59:59")
assert len(due) == 0
def test_at_task_fields(self, db):
db.create_scheduled_task(
**_make_task_kwargs(
task_id="at_task",
schedule_type="at",
cron_expr="",
at_time="2099-06-15T14:00:00",
next_run="2099-06-15T14:00:00",
)
)
result = db.get_scheduled_task("at_task")
assert result is not None
assert result["schedule_type"] == "at"
assert result["at_time"] == "2099-06-15T14:00:00"
class TestScheduledTaskRuns:
"""Tests for scheduled_task_runs table operations."""
def test_record_and_list(self, db):
db.create_scheduled_task(**_make_task_kwargs())
db.record_task_run(
run_id="run_a",
task_id="task_001",
node_id="node_1",
ws_id="ws_1",
correlation_id="corr_a",
started="2025-01-01T09:00:00",
status="dispatched",
error="",
)
db.record_task_run(
run_id="run_b",
task_id="task_001",
node_id="node_2",
ws_id="ws_2",
correlation_id="corr_b",
started="2025-01-02T09:00:00",
status="dispatched",
error="",
)
runs = db.list_task_runs("task_001")
assert len(runs) == 2
# Ordered by started DESC — most recent first
assert runs[0]["run_id"] == "run_b"
assert runs[1]["run_id"] == "run_a"
def test_list_runs_respects_limit(self, db):
db.create_scheduled_task(**_make_task_kwargs())
for i in range(3):
db.record_task_run(
run_id=f"run_{i}",
task_id="task_001",
node_id="node_1",
ws_id="",
correlation_id=f"corr_{i}",
started=f"2025-01-0{i + 1}T09:00:00",
status="dispatched",
error="",
)
runs = db.list_task_runs("task_001", limit=2)
assert len(runs) == 2
def test_list_runs_empty(self, db):
assert db.list_task_runs("no_such_task") == []
def test_prune_task_runs(self, db):
db.create_scheduled_task(**_make_task_kwargs())
# Old run (should be pruned)
db.record_task_run(
run_id="old_run",
task_id="task_001",
node_id="node_1",
ws_id="",
correlation_id="c_old",
started="2020-01-01T00:00:00",
status="dispatched",
error="",
)
# Recent run (should survive)
db.record_task_run(
run_id="new_run",
task_id="task_001",
node_id="node_1",
ws_id="",
correlation_id="c_new",
started="2099-01-01T00:00:00",
status="dispatched",
error="",
)
pruned = db.prune_task_runs(retention_days=90)
assert pruned == 1
runs = db.list_task_runs("task_001")
assert len(runs) == 1
assert runs[0]["run_id"] == "new_run"
+272
View File
@@ -0,0 +1,272 @@
"""Tests for turnstone.console.scheduler — TaskScheduler tick and dispatch."""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from turnstone.console.scheduler import TaskScheduler
@pytest.fixture
def mocks():
"""Broker, collector, and storage mocks for scheduler tests."""
broker = MagicMock()
broker._redis = MagicMock()
collector = MagicMock()
storage = MagicMock()
return broker, collector, storage
def _make_task(**overrides):
"""Build a minimal task dict matching storage row format."""
defaults = {
"task_id": "task_001",
"name": "Test task",
"description": "",
"schedule_type": "cron",
"cron_expr": "0 9 * * *",
"at_time": "",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Run the tests",
"auto_approve": 0,
"auto_approve_tools": "",
"enabled": 1,
"created_by": "u_admin",
"next_run": "2020-01-01T09:00:00",
"last_run": "",
"created": "2020-01-01T00:00:00",
"updated": "2020-01-01T00:00:00",
}
defaults.update(overrides)
return defaults
def _make_node(node_id="node-001", reachable=True, ws_total=2, max_ws=10):
"""Build a minimal node dict matching collector output."""
return {
"node_id": node_id,
"reachable": reachable,
"ws_total": ws_total,
"max_ws": max_ws,
}
class TestSchedulerTick:
"""Tests for _tick() lock acquisition and dispatch logic."""
def test_tick_acquires_lock(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
storage.list_due_tasks.return_value = []
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker._redis.set.assert_called_once()
storage.list_due_tasks.assert_called_once()
# Lock released via Lua eval (conditional delete)
broker._redis.eval.assert_called_once()
def test_tick_skips_when_locked(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = None # lock held by another console
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
storage.list_due_tasks.assert_not_called()
def test_dispatch_auto_mode(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="auto")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
_, kwargs = broker.push_inbound.call_args
assert (
kwargs.get("node_id") == "node-001"
or broker.push_inbound.call_args[1].get("node_id") == "node-001"
)
storage.record_task_run.assert_called_once()
run_kwargs = storage.record_task_run.call_args[1]
assert run_kwargs["node_id"] == "node-001"
assert run_kwargs["status"] == "dispatched"
def test_dispatch_pool_mode(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="pool")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
# Pool dispatch calls push_inbound without node_id kwarg
args, kwargs = broker.push_inbound.call_args
assert kwargs.get("node_id") is None or "node_id" not in kwargs
storage.record_task_run.assert_called_once()
run_kwargs = storage.record_task_run.call_args[1]
assert run_kwargs["node_id"] == "pool"
def test_dispatch_all_mode(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="all")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = (
[_make_node("node-001"), _make_node("node-002")],
2,
)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
assert broker.push_inbound.call_count == 2
assert storage.record_task_run.call_count == 2
def test_dispatch_specific_node(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="node-001")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
_, kwargs = broker.push_inbound.call_args
assert kwargs["node_id"] == "node-001"
def test_at_task_disables_after_dispatch(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(schedule_type="at", cron_expr="", at_time="2099-01-01T00:00:00")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
# At-task should be disabled after dispatch
update_calls = storage.update_scheduled_task.call_args_list
assert len(update_calls) == 1
args, kwargs = update_calls[0]
assert args[0] == "task_001"
assert kwargs["enabled"] is False
assert kwargs["next_run"] == ""
def test_cron_task_updates_next_run(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(schedule_type="cron", cron_expr="0 9 * * *")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
update_calls = storage.update_scheduled_task.call_args_list
assert len(update_calls) == 1
_, kwargs = update_calls[0]
assert kwargs["next_run"] != ""
assert "enabled" not in kwargs # cron tasks stay enabled
def test_no_reachable_nodes_records_failure(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="auto")
storage.list_due_tasks.return_value = [task]
# No reachable nodes
collector.get_nodes.return_value = (
[_make_node("node-001", reachable=False)],
1,
)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_not_called()
storage.record_task_run.assert_called_once()
run_kwargs = storage.record_task_run.call_args[1]
assert run_kwargs["status"] == "failed"
assert run_kwargs["error"] != ""
def test_failure_does_not_advance_schedule(self, mocks):
"""When dispatch fails, last_run/next_run should not be updated."""
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="auto")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([], 0) # no nodes at all
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
# update_scheduled_task should NOT be called (no last_run/next_run advance)
storage.update_scheduled_task.assert_not_called()
def test_fan_out_capped(self, mocks):
"""Fan-out 'all' mode should respect max_fan_out limit."""
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="all")
storage.list_due_tasks.return_value = [task]
# 10 reachable nodes but max_fan_out=3
nodes = [_make_node(f"node-{i:03d}") for i in range(10)]
collector.get_nodes.return_value = (nodes, 10)
scheduler = TaskScheduler(broker, collector, storage, max_fan_out=3)
scheduler._tick()
assert broker.push_inbound.call_count == 3
assert storage.record_task_run.call_count == 3
def test_specific_node_target(self, mocks):
"""Non-enum target_mode is treated as a specific node_id."""
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="node-custom-123")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
call_kwargs = broker.push_inbound.call_args
assert call_kwargs[1]["node_id"] == "node-custom-123"
def test_user_id_in_dispatched_message(self, mocks):
"""Dispatched message should include created_by as user_id."""
import json
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="pool", created_by="u_scheduler_admin")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
msg_json = broker.push_inbound.call_args[0][0]
msg_data = json.loads(msg_json)
assert msg_data["user_id"] == "u_scheduler_admin"
+139
View File
@@ -2,6 +2,8 @@
from __future__ import annotations
import json
import httpx
import pytest
@@ -240,3 +242,140 @@ async def test_query_params_passed():
assert "state=running" in captured_url[0]
assert "page=2" in captured_url[0]
assert "per_page=25" in captured_url[0]
# ---------------------------------------------------------------------------
# Schedules
# ---------------------------------------------------------------------------
_SCHEDULE_FIXTURE = {
"task_id": "t1",
"name": "nightly",
"description": "",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run nightly checks",
"auto_approve": False,
"auto_approve_tools": [],
"enabled": True,
"created_by": "u1",
"last_run": None,
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-05T12:00:00Z",
"updated": "2026-03-05T12:00:00Z",
}
@pytest.mark.anyio
async def test_list_schedules():
transport = _mock_transport(
{"GET /v1/api/admin/schedules": _json_response({"schedules": [_SCHEDULE_FIXTURE]})}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.list_schedules()
assert len(resp.schedules) == 1
assert resp.schedules[0].task_id == "t1"
assert resp.schedules[0].name == "nightly"
@pytest.mark.anyio
async def test_create_schedule():
captured_body: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
captured_body.append(json.loads(request.content))
return _json_response(_SCHEDULE_FIXTURE)
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.create_schedule(
name="nightly",
schedule_type="cron",
initial_message="Run nightly checks",
cron_expr="0 2 * * *",
)
assert resp.task_id == "t1"
body = captured_body[0]
assert body["name"] == "nightly"
assert body["schedule_type"] == "cron"
assert body["cron_expr"] == "0 2 * * *"
assert body["initial_message"] == "Run nightly checks"
# Optional fields with defaults should not appear when not set
assert "description" not in body
assert "model" not in body
@pytest.mark.anyio
async def test_get_schedule():
transport = _mock_transport(
{"GET /v1/api/admin/schedules/t1": _json_response(_SCHEDULE_FIXTURE)}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.get_schedule("t1")
assert resp.task_id == "t1"
assert resp.schedule_type == "cron"
@pytest.mark.anyio
async def test_update_schedule_partial():
"""Only explicitly-passed fields should appear in the request body."""
captured_body: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
captured_body.append(json.loads(request.content))
return _json_response({**_SCHEDULE_FIXTURE, "enabled": False})
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.update_schedule("t1", enabled=False)
assert resp.enabled is False
body = captured_body[0]
assert body == {"enabled": False}
@pytest.mark.anyio
async def test_delete_schedule():
transport = _mock_transport(
{"DELETE /v1/api/admin/schedules/t1": _json_response({"status": "ok"})}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.delete_schedule("t1")
assert resp.status == "ok"
@pytest.mark.anyio
async def test_list_schedule_runs():
transport = _mock_transport(
{
"GET /v1/api/admin/schedules/t1/runs": _json_response(
{
"runs": [
{
"run_id": "r1",
"task_id": "t1",
"node_id": "n1",
"ws_id": "ws1",
"correlation_id": "c1",
"started": "2026-03-05T02:00:00Z",
"status": "dispatched",
"error": "",
}
]
}
)
}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.list_schedule_runs("t1", limit=10)
assert len(resp.runs) == 1
assert resp.runs[0].run_id == "r1"
assert resp.runs[0].status == "dispatched"

Some files were not shown because too many files have changed in this diff Show More