Compare commits

...

15 Commits

Author SHA1 Message Date
Patrick Buckley f1f448277f chore: bump version to 0.5.6
Prompt template runtime wiring, security hardening, scheduler/channel/MQ
template support, migration 010.
2026-03-12 16:58:20 -07:00
Patrick Buckley 2f7f70825b feat: wire prompt templates into session startup with full creation-p… (#47)
* feat: wire prompt templates into session startup with full creation-path support

Prompt templates (prompt_templates table) now have runtime effect:

- is_default=true templates auto-apply as system message content,
  concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
  field on POST /v1/api/workstreams/new, console creation modal dropdown,
  scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
  regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession

Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
  server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion

Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.

Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.

Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.

Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.

* fix: address PR #47 review feedback

- Defer template validation until after resume_ws — a bad template name
  no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
  openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
  reject unknown template names with 400 instead of allowing schedules
  that would silently fail at dispatch time
2026-03-12 16:57:31 -07:00
Patrick Buckley 4866c9873c feat: ddgCluster compose profile with DuckDuckGo Search MCP sidecar (#48)
Add ddgCluster profile extending the 10-node cluster with a DuckDuckGo
Search MCP sidecar. All cluster nodes connect via streamable-http and
gain duckduckgo_web_search + duckduckgo_fetch_content tools. No API
key required.

Key implementation details learned during testing:
- MCP SDK DNS rebinding protection must be disabled for Docker
  internal networking (Host header uses container names)
- FastMCP server binds to 127.0.0.1 by default; must set
  mcp.settings.host='0.0.0.0' for cross-container access
- DDG CLI lacks --host/--port flags; settings configured via Python
  entry point that patches FastMCP.settings directly
- Safe search disabled by default

Also adds MCP_CONFIG env var support to all server commands (shell
conditional, no-op when empty) and moves default server/bridge to
production profile for cleaner profile separation.
2026-03-12 16:51:53 -07:00
Patrick Buckley 8b2e2130fc fix: MCP resource template URI expansion via prefix matching (#46)
* fix: MCP resource template URI expansion via prefix matching

Resource templates (RFC 6570 URI patterns like `db://tables/{table}/rows/{id}`)
were discovered from MCP servers but non-functional — `read_resource_sync()`
only accepted exact URIs from `_resource_map`, which excludes templates.

Add prefix-based fallback: extract the static prefix from each template
(everything before the first `{`), store a prefix→server mapping, and
fall back to longest-prefix matching when exact URI lookup fails. MCP
servers handle URI routing internally so we just need to route the
expanded URI to the correct server.

Also surface templates in the system message catalog and `/mcp` command
so the model knows they exist and can construct expanded URIs.

* fix: address PR #46 review feedback

- Template prefix collision now keeps more specific (longer) template
  URI instead of blindly overriding
- Fix _match_template docstring to accurately describe startswith
  matching on static prefixes (not full template matching)
- Add missing loop.close() in integration test finally block
- Rewrite test_template_longest_prefix_wins with genuinely different
  prefix lengths to avoid brittle collision-order dependency
2026-03-12 15:46:00 -07:00
Patrick Buckley f81c06761d chore: remove dead code, add MCP integration + collector tests (#45)
* chore: remove dead code, add MCP integration + collector tests

Remove unused delete_prompt_templates_by_server from protocol and
both storage backends (sync uses per-template deletion).

Add 10 MCP integration tests exercising full lifecycle: rebuild
resources/prompts, read_resource_sync/get_prompt_sync with real
asyncio loop, governance sync to real SQLite, shutdown cleanup,
listener notification isolation.

Add 3 console collector MCP aggregation tests: multi-node sums,
absent when zero, mixed nodes with/without MCP.

* fix: close event loops and SQLite backend in MCP integration tests
2026-03-12 15:16:51 -07:00
Patrick Buckley be165c1971 feat: MCP resource and prompt discovery with read_resource tool (#44)
* feat: MCP resource and prompt discovery with read_resource tool

Extends MCPClientManager with resource and prompt discovery alongside
existing tool support. Resources and prompts are discovered on connect,
cached per-server with copy-on-write rebuilds, and refreshed via push
notifications, periodic polling, or manual /mcp refresh.

New read_resource built-in tool reads MCP resources by URI. Requires
user approval (same as MCP tool calls) since resources are served by
external MCP servers. Resource catalog injected into system message
with XML delimiters. Error messages sanitized to prevent leaking
server internals to the model.

Prompt discovery stores prefixed names (mcp__server__prompt) and
exposes get_prompt_sync() for future use_prompt tool (Chunk D).

/mcp command now shows tools, resources, and prompts. Docs and
diagrams updated.

* feat: MCP prompt governance sync with origin tracking and readonly guards

Migration 009 adds origin, mcp_server, and readonly columns to
prompt_templates. MCP prompts discovered by MCPClientManager are
automatically synced into the governance table as read-only templates
with origin="mcp".

Sync engine handles: create on connect, update on prompt refresh,
delete when prompts are removed from server. Manual templates take
precedence on name collision (MCP prompt skipped with warning).

Admin API returns 403 on update/delete of readonly templates. Console
UI shows MCP origin badge and disables edit/delete buttons. Storage
backends gain get_prompt_template_by_name, list_prompt_templates_by_origin,
and delete_prompt_templates_by_server methods.

Also addresses PR #44 review feedback: concurrent.futures.TimeoutError
handling in sync dispatch, XML-escape resource catalog descriptions,
resource template entries excluded from _resource_map, URI collision
warnings, needs_periodic capability-aware computation, malformed JSON
primary key fallback for read_resource.

* feat: use_prompt tool, prompt catalog, and PR review hardening

New use_prompt built-in tool invokes MCP prompt templates by name,
expanding them into messages. Requires user approval (external MCP
servers). Prompt catalog injected into system message with XML
delimiters (up to 30 prompts, HTML-escaped).

Prompt listener registered in session for catalog rebuild on changes.

Addresses PR #44 review feedback:
- _init_system_messages() now uses copy-on-write (build locally,
  assign atomically) so background thread callbacks never see
  partial system messages
- sync_prompts_to_storage() serialized behind _sync_lock to prevent
  races between set_storage() (main thread) and MCP background thread
- shutdown() clears listener lists to release callback references

Docs and diagrams updated for 18 built-in tools.

* feat: granular tool policies for MCP resources, prompts, and tools

Policy evaluation now uses approval_label (falling back to func_name)
for fnmatch pattern matching, enabling fine-grained per-URI and
per-server policies:
- read_resource: mcp_resource__{normalized_uri}
- use_prompt: mcp__{server}__{prompt} (prefixed name)
- MCP tools: mcp__{server}__{tool} (was static "mcp_tool")

URI normalization resolves .. path segments to prevent traversal
bypasses in policy matching. Resource templates filtered from system
message catalog (not directly readable). use_prompt arguments
validated as dict with string coercion.

TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly
fields. Governance docs updated with MCP policy patterns.

* feat: MCP visibility in server and console UIs

Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts
counts. Server UI status bar shows magenta MCP indicator with tooltip.
Console cluster status bar shows MCP metrics with magenta LED dot.
Console node detail view shows per-node MCP summary. Console collector
aggregates MCP counts across nodes in overview.

Uses var(--magenta) design token with new --magenta-glow for theme
adaptation. ARIA roles on MCP status elements. Tooltips on console
MCP metric labels. Node MCP summary hidden on mobile (< 700px).

New diagram: 20-mcp-architecture.puml covering full MCP lifecycle
(connection, discovery, refresh, governance sync, policy, UI).

* fix: McpStatus in health schema, count properties, catalog name fidelity

Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so
typed clients see the mcp field from /health.

Addresses Copilot review feedback:
- resource_count/prompt_count properties avoid list allocation on
  /health and /metrics polls
- get_tools/resources/prompts return shallow-copied dicts to prevent
  callers from mutating internal cache
- Prompt names and arg names in system message catalog are NOT
  HTML-escaped (model must use exact strings in use_prompt calls);
  only descriptions are escaped

* fix: OpenAPI spec McpStatus + diagram approval column accuracy

Adds McpStatus schema and optional mcp field to HealthResponse in
openapi-server.json, matching the Python schema and TypeScript types.

Fixes tool pipeline diagram: math, web_fetch, web_search correctly
shown as auto-approve (not "Yes" for approval).
2026-03-12 14:49:58 -07:00
Patrick Buckley 3264fdefca fix: channel bidirectional routing — emit TurnCompleteEvent on all id… (#43)
* fix: channel bidirectional routing — emit TurnCompleteEvent on all idle transitions

Bridge previously only emitted TurnCompleteEvent for MQ-initiated turns
(those with a correlation_id in _active_sends). Server-UI-initiated turns
went idle without emitting TurnCompleteEvent, so the Discord bot's
StreamingMessage never finalized — content accumulated in the buffer and
collided with the next Discord-triggered response.

Now TurnCompleteEvent is emitted unconditionally on every idle transition.
correlation_id is empty for non-MQ turns; SDK client filters by
correlation_id so existing consumers are unaffected.

* fix: remove unused variable flagged by ruff
2026-03-12 11:57:15 -07:00
Patrick Buckley 28cb3a5c51 fix: approval timeout UI state and content flush before tool calls (#42)
* fix: approval timeout UI state and content flush before tool calls

Two bug fixes:

1. Approval timeout now shows denied state in UI — resolve_approval()
   emits an approval_resolved SSE event so the browser transitions
   from pending to denied (red border + badge). Also fixes the cancel-
   during-approval path. Frontend resolveInlineApproval() gains a
   skipPost parameter to avoid redundant POST when server-initiated.
   ApprovalResolvedEvent added to Python and TypeScript SDKs.

2. Content streaming flushes pending buffer before tool call deltas —
   _stream_response() held up to 13 trailing chars in the pending
   buffer (for <think> tag detection) when transitioning to tool calls.
   Now flushed eagerly when tool_call_deltas arrive, before clearing
   in_think so reasoning text is correctly categorized.

* fix: address Copilot review feedback on PR #42

Patch _execute_tools in stream flush test to prevent real bash execution,
simplify confusing nested comprehension, and update resolve_approval()
docstring to reflect cancel/timeout call paths.
2026-03-12 11:43:06 -07:00
Patrick Buckley 8b11e0a6f9 fix: bridge retries node_id fetch indefinitely with capped backoff 2026-03-12 11:12:03 -07:00
Patrick Buckley 648ba477e1 refactor: list_user_roles uses _row_to_dict instead of positional row mapping 2026-03-11 21:14:58 -07:00
Patrick Buckley 7960784786 fix: usage events now record per-request tool_calls delta, not cumulative total 2026-03-11 21:12:03 -07:00
Patrick Buckley e06554d1ec feat: add channel admin endpoints to console OpenAPI spec 2026-03-11 21:08:28 -07:00
Patrick Buckley 8eb8722346 Bump version to 0.5.5 2026-03-11 20:24:12 -07:00
Patrick Buckley a2e2ffacd8 feat: robust plan quality gate, iterative refinement, and amend UX (#41)
* feat: robust plan quality gate, iterative refinement, and amend UX

Plan agent output from weak models often produced garbage (11-char plans
that echo the prompt). Two fixes:

1. Quality validation (_validate_plan) checks length, section structure,
   echo detection, and refusal patterns. Fails trigger one automatic
   retry with a coaching message injected into the agent's existing
   conversation, preserving all prior exploration context.

2. Iterative feedback loop — user feedback at plan review re-runs the
   plan agent via _refine_plan() instead of appending text to the tool
   result. Up to 5 refinement rounds. The plan file path is always
   included in the tool result so the outer model knows where it lives.

UI improvements:
- Web: Reject button dynamically becomes "Amend" (amber) when feedback
  is typed. Key hint badges (Esc/Enter) on plan buttons. Main input
  disabled during review. Light-theme contrast fix via --on-color var.
- CLI: Prompt shows all three actions (approve/amend/reject).
- Bridge: Race condition fix — clear pending entry before HTTP POST so
  sequential plan reviews from the refinement loop aren't skipped.

15 new tests covering validation, retry, and refinement.

* fix: address PR 41 review feedback

- Escape key in plan dialog now mirrors the Amend button: if feedback is
  typed, Esc sends the feedback (amend); if empty, Esc rejects. Previously
  Esc always hard-coded "reject", discarding typed feedback.

- Coaching message for plan retry now says "should include at least two of"
  instead of "MUST include these", matching the actual validation rule
  (_MIN_PLAN_SECTIONS = 2).

* feat: render plan inline in chat after approval

After the plan review dialog closes, the plan content is now rendered
as a collapsible inline block in the chat stream — styled with a
status header (approved/rejected/amending), markdown-rendered body,
and feedback note when amending. Uses the same makeCollapsible pattern
as tool output blocks.

* fix: prevent plan approval hang when inline render fails

The authFetch call that unblocks the server must fire before the
cosmetic inline plan rendering. Previously _addInlinePlan ran first
and any JS error (e.g. from renderMarkdown) prevented the API call,
leaving the session thread blocked forever.

- Move authFetch before _addInlinePlan
- Wrap _addInlinePlan in try-catch
- Guard against empty content
- Only auto-collapse plans longer than 12 lines

* fix: address PR 41 review feedback (round 2)

- Max refinement rounds no longer implicitly approve: the loop now
  shows the final plan for explicit approve/reject before proceeding.
  Previously exhausting 5 rounds silently accepted the last revision.

- Plan inline block: correct aria-label from "Tool output" to
  "Plan content" when makeCollapsible is applied.

- XSS concern (not applicable): renderMarkdown is used for all
  assistant messages — plan content follows the same trust model.

- Test loop concern (acknowledged): refinement tests verify component
  logic; full _execute_tools integration would require extensive
  mocking for marginal coverage gain.

* feat: thinking spinner + inline plan hardening

* fix lint
2026-03-11 20:22:41 -07:00
Patrick Buckley c6ba8d59b0 feat: bootstrap wizard — LLM-guided interactive setup for deployments
Add `turnstone-bootstrap`, a new entry point that uses any LLM (OpenAI,
Anthropic, or local/vLLM) to conversationally walk users through
configuring a Turnstone deployment. Generates .env files, setup.sh
scripts, and optional docker-compose overrides.

- Fully interactive startup (zero CLI args) with provider/model selection
- Auto-detects available models on local OpenAI-compatible endpoints
- 7 tools: read_file, write_file, generate_secret, check_port,
  validate_api_key, check_docker, finish
- Path traversal protection on file read/write
- Duplicate write detection (skips identical content)
- Bounded retry loop (3 attempts) on LLM errors
- Anthropic message conversion with consecutive-role merging
2026-03-11 01:58:39 -07:00
81 changed files with 6976 additions and 333 deletions
+92
View File
@@ -0,0 +1,92 @@
# Bootstrap Wizard
Interactive, AI-guided setup for Turnstone deployments. Instead of manually
editing `.env` files and reading deployment docs, the wizard walks you through
every decision conversationally and generates all the config files for you.
## Quick Start
```bash
turnstone-bootstrap
```
That's it — no flags, no arguments. The wizard prompts for everything.
## How It Works
1. **Pick a model** — Choose OpenAI, Anthropic, or a local/vLLM endpoint to
power the wizard. Local endpoints auto-detect available models.
2. **Answer questions** — The AI walks you through deployment mode, LLM
provider, database, authentication, ports, and optional features.
3. **Review generated files** — Each file is previewed before writing. You
confirm or reject every write.
4. **Start the stack** — The wizard prints the exact `docker compose` command
and a `setup.sh` script to create your first admin user, roles, and policies.
## What Gets Generated
| File | Purpose |
|------|---------|
| `.env` | All environment variables for `compose.yaml` |
| `setup.sh` | Post-start script: creates admin user, roles, tool policies, prompt templates via the API |
| `docker-compose.override.yaml` | Only if customizations beyond env vars are needed |
## Requirements
- **Python 3.11+** with turnstone installed (`pip install turnstone`)
- **An LLM API key** — for the wizard itself (OpenAI, Anthropic, or a local
model). This can differ from the LLM your deployment will use.
- **Docker & Docker Compose** — needed to run the stack. The wizard detects
whether Docker is installed and gives platform-specific install instructions
if it's missing. You can still generate config files without Docker.
## Deployment Modes
The wizard supports two deployment modes:
- **Single-node production** (`docker compose --profile production up`) —
1 server + bridge + console + PostgreSQL + Redis. Good for most use cases.
- **Multi-node cluster** (`docker compose --profile cluster up`) —
10-node server/bridge fleet + PostgreSQL + Redis. For high-throughput or
HA deployments.
## Example Session
```
$ turnstone-bootstrap
Turnstone Bootstrap Wizard v0.5.4
────────────────────────────────────────────────
Which provider for this wizard?
[1] OpenAI
[2] Anthropic
[3] OpenAI-compatible (local/vLLM)
> 3
Base URL [http://localhost:8000/v1]:
API key (press Enter for 'none'):
Querying http://localhost:8000/v1 for available models...
Found model: Qwen/Qwen3-32B
Connected to Qwen/Qwen3-32B. Handing off to AI assistant...
> (AI walks you through the rest interactively)
```
## Tips
- **Re-run safely** — running the wizard again detects your existing `.env`
and offers to update it rather than overwriting.
- **Duplicate writes are skipped** — if the LLM tries to write the same file
twice with identical content, it's silently ignored.
- **Type `quit` to exit** at any time during the conversation.
- **Ctrl+C** is handled gracefully — press once to interrupt, twice to exit.
## See Also
- [Docker Deployment](docker.md) — manual compose setup and profiles
- [Security](security.md) — auth architecture and token types
- [Governance](governance.md) — roles, policies, and templates
+56 -5
View File
@@ -2,10 +2,11 @@
# Turnstone Docker Compose Stack
#
# Usage:
# Default (SQLite): docker compose up
# Infra only: docker compose up
# Single node: docker compose --profile production up
# Production (PG): DB_BACKEND=postgresql docker compose --profile production up
# (or set DB_BACKEND=postgresql in .env)
# 10-node cluster: docker compose --profile cluster up
# Cluster + DDG: docker compose --profile ddgCluster up
# With simulator: docker compose --profile sim up
# =============================================================================
@@ -29,6 +30,7 @@ services:
profiles:
- production
- cluster
- ddgCluster
environment:
POSTGRES_DB: turnstone
POSTGRES_USER: ${POSTGRES_USER:-turnstone}
@@ -88,6 +90,8 @@ services:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
command:
- sh
- -c
@@ -99,10 +103,12 @@ services:
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
ports:
- "${SERVER_PORT:-8080}:8080"
volumes:
- turnstone-data:/data
- ./docker/mcp-ddg.json:/etc/turnstone/mcp-ddg.json:ro
environment:
- LLM_BASE_URL=${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
- OPENAI_API_KEY=${OPENAI_API_KEY:-dummy}
@@ -112,6 +118,7 @@ services:
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- MODEL=${MODEL:-}
- MCP_CONFIG=${MCP_CONFIG:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
- TURNSTONE_NODE_ID=${TURNSTONE_NODE_ID:-}
@@ -125,6 +132,9 @@ services:
postgres:
condition: service_healthy
required: false
ddg-search:
condition: service_healthy
required: false
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
@@ -141,6 +151,8 @@ services:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
command:
- turnstone-bridge
- --server-url=http://server:8080
@@ -208,6 +220,7 @@ services:
profiles:
- production
- cluster
- ddgCluster
command:
- sh
- -c
@@ -235,6 +248,39 @@ services:
required: false
restart: unless-stopped
# -------------------------------------------------------------------
# ddg-search — DuckDuckGo Search MCP server (HTTP transport)
# Provides web search + content fetch tools to turnstone via MCP.
# No API key required.
#
# Start with: MCP_CONFIG=/etc/turnstone/mcp-ddg.json \
# docker compose --profile ddgCluster up
# -------------------------------------------------------------------
ddg-search:
image: python:3.13-slim
profiles:
- ddgCluster
command:
- sh
- -c
- >-
pip install --no-cache-dir duckduckgo-mcp-server &&
python -c "from mcp.server.transport_security import TransportSecuritySettings; import duckduckgo_mcp_server.server as s; s.safe_search=s.SafeSearchMode.OFF; s.mcp.settings.host='0.0.0.0'; s.mcp.settings.port=3000; s.mcp.settings.transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False); s.mcp.run(transport='streamable-http')"
networks:
- turnstone-net
healthcheck:
test: ["CMD-SHELL", "python -c \"import socket; s=socket.create_connection(('0.0.0.0',3000),2); s.close()\""]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
deploy:
resources:
limits:
memory: 256M
cpus: '0.25'
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-sim — Multi-node cluster simulator (no LLM needed)
# Start with: docker compose --profile sim up
@@ -288,7 +334,7 @@ services:
server-1: &cluster-server
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster]
profiles: [cluster, ddgCluster]
command: &cluster-server-cmd
- sh
- -c
@@ -300,7 +346,10 @@ services:
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
volumes: [turnstone-data:/data]
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
volumes:
- turnstone-data:/data
- ./docker/mcp-ddg.json:/etc/turnstone/mcp-ddg.json:ro
environment: &cluster-server-env
LLM_BASE_URL: ${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
OPENAI_API_KEY: ${OPENAI_API_KEY:-dummy}
@@ -310,6 +359,7 @@ services:
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
MODEL: ${MODEL:-}
MCP_CONFIG: ${MCP_CONFIG:-}
TURNSTONE_DB_BACKEND: ${DB_BACKEND:-postgresql}
TURNSTONE_DB_URL: ${DATABASE_URL:-postgresql://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:?}@postgres:5432/turnstone}
TURNSTONE_NODE_ID: node-1
@@ -318,6 +368,7 @@ services:
depends_on:
redis: { condition: service_healthy }
postgres: { condition: service_healthy }
ddg-search: { condition: service_healthy, required: false }
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
@@ -361,7 +412,7 @@ services:
bridge-1: &cluster-bridge
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster]
profiles: [cluster, ddgCluster]
command:
- turnstone-bridge
- --server-url=http://server-1:8080
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"ddg": {
"url": "http://ddg-search:3000/mcp"
}
}
}
+1
View File
@@ -764,6 +764,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `model` | string | default | Model alias from the registry (`[models.*]`) |
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `resume_ws` | string | "" | Workstream ID to resume atomically during creation (empty = fresh)|
| `template` | string | "" | Prompt template name (replaces default templates; 400 if not found)|
**Response (success):**
+1 -1
View File
@@ -3,7 +3,7 @@
Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent
memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or
Anthropic's native Messages API via pluggable provider adapters, and gives the
model 14 built-in tools plus external tools via MCP (Model Context Protocol) for
model 18 built-in tools plus external tools via MCP (Model Context Protocol) for
reading, writing, searching, planning, and executing code.
The core design principle is a **UI-agnostic engine with pluggable frontends**.
+1 -1
View File
@@ -96,7 +96,7 @@ package "turnstone/sdk/" <<Rectangle>> {
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\n15 tool schemas] as schemas <<artifact>>
component [*.json\n18 tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
+10
View File
@@ -211,15 +211,23 @@ enum "WorkstreamState" as WsState {
class "MCPClientManager" as MCPMgr {
- _sessions: dict[str, ClientSession]
- _per_server_tools: dict[str, list[dict]]
- _per_server_resources: dict[str, list[dict]]
- _per_server_prompts: dict[str, list[dict]]
- _tools: list[dict]
- _tool_map: dict[str, tuple]
- _resource_map: dict[str, tuple]
- _prompt_map: dict[str, tuple]
- _supports_list_changed: dict[str, bool]
- _listeners: list[Callable]
--
+ start()
+ get_tools() → list[dict]
+ get_resources() → list[dict]
+ get_prompts() → list[dict]
+ is_mcp_tool(name) → bool
+ call_tool_sync(name, args) → str
+ read_resource_sync(uri) → str
+ get_prompt_sync(name, args?) → list[dict]
+ refresh_sync(server?) → dict
+ add_listener(callback)
+ remove_listener(callback)
@@ -230,6 +238,8 @@ class "MCPClientManager" as MCPMgr {
bridges async MCP SDK to
sync ChatSession dispatch.
Push + periodic + manual refresh.
Resources + prompts discovered
alongside tools at startup.
--
core/mcp_client.py
}
+27 -23
View File
@@ -24,29 +24,31 @@ partition "Phase 1: Prepare" #E8F5E9 {
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (16 tools):**
┌──────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├──────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✓ Yes
│ man │ ✗ Auto-approve │
│ web_fetch │ ✓ Yes
│ web_search │ ✓ Yes
│ tool_search │ ✗ Auto-approve │
│ task │ ✓ Yes │
│ plan │ ✓ Yes │
│ remember │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ forget │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
├──────────────┼──────────────────┤
mcp__* │ ✓ Yes (external)
────────────────────────────────
**Dispatch table (18 tools):**
┌──────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├──────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math ✗ Auto-approve
│ man │ ✗ Auto-approve │
│ web_fetch ✗ Auto-approve
│ web_search ✗ Auto-approve
│ tool_search │ ✗ Auto-approve │
│ task │ ✓ Yes │
│ plan │ ✓ Yes │
│ remember │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ forget │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
│ read_resource │ ✓ Yes │
use_prompt │ ✓ Yes
├─────────────────────────────────
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
@@ -117,6 +119,8 @@ partition "Phase 3: Execute" #E3F2FD {
├─ _exec_remember: SQLite INSERT OR REPLACE
├─ _exec_recall: SQLite FTS5/LIKE search
├─ _exec_forget: SQLite DELETE
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
end note
@@ -35,6 +35,13 @@ package "Runtime Enforcement" {
[record_audit()] as audit
}
package "Template Runtime" {
[_load_templates()] as tload
[_render_template()\n{{model}}, {{ws_id}}, {{node_id}}] as trender
[_init_system_messages()] as tsys
[set_template() / /template] as tset
}
package "Console UI" {
[Admin Panel\n10 tabs] as ui
[governance.js] as govjs
@@ -64,6 +71,11 @@ govjs --> pt_db : /v1/api/admin/templates
govjs --> ue_db : /v1/api/admin/usage
govjs --> ae_db : /v1/api/admin/audit
tload --> pt_db : list_default_templates()\nor get_by_name()
tload --> trender : template content
trender --> tsys : rendered content
tset --> tload : name or None
auth -[hidden]-> mw
mw -[hidden]-> approve
@enduml
+158
View File
@@ -0,0 +1,158 @@
@startuml
!theme plain
title Turnstone — MCP Architecture (Resources, Prompts, Tools)
skinparam participant {
BackgroundColor<<mcp>> #E1BEE7
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<ui>> #E8EAF6
}
participant "MCP Server\n(external)" as MCPSrv <<mcp>>
participant "MCPClientManager\n(mcp_client.py)" as MCPMgr <<mcp>>
participant "ChatSession\n(session.py)" as Session <<session>>
participant "StorageBackend\n(governance)" as Storage <<storage>>
participant "Server / Console\n(health + UI)" as UI <<server>>
== Startup: Connection & Discovery ==
MCPMgr -> MCPSrv : initialize (stdio or HTTP)
MCPSrv --> MCPMgr : capabilities\n(tools, resources, prompts)
MCPMgr -> MCPSrv : tools/list
MCPSrv --> MCPMgr : Tool[]
opt resources capability
MCPMgr -> MCPSrv : resources/list
MCPSrv --> MCPMgr : Resource[]
MCPMgr -> MCPSrv : resources/templates/list
MCPSrv --> MCPMgr : ResourceTemplate[]
end
opt prompts capability
MCPMgr -> MCPSrv : prompts/list
MCPSrv --> MCPMgr : Prompt[]
end
note over MCPMgr
Per-server storage:
_per_server_tools, _per_server_resources, _per_server_prompts
Copy-on-write rebuild into _tools, _resources, _prompts
Prefix: mcp__{server}__{name}
end note
MCPMgr -> Session : notify tool listeners
MCPMgr -> Session : notify resource listeners
== Governance Sync (on connect & refresh) ==
MCPMgr -> Storage : sync_prompts_to_storage()
note right
For each MCP prompt:
- Manual template exists? → skip
- MCP template exists? → update
(reset is_default=False)
- New? → create (origin="mcp",
readonly=True, is_default=False)
Removed prompts → delete
Protected by _sync_lock
end note
== set_storage() from entry point ==
UI -> MCPMgr : set_storage(backend)
note right
If servers already connected,
triggers immediate sync
end note
== Runtime: Tool Execution ==
Session -> Session : _prepare_mcp_tool(func_name, args)
note right
approval_label = func_name
(e.g. mcp__github__search)
needs_approval = True
end note
Session -> MCPMgr : call_tool_sync(name, args)
MCPMgr -> MCPSrv : tools/call
MCPSrv --> MCPMgr : ToolResult
MCPMgr --> Session : output (text)
== Runtime: Resource Read ==
Session -> Session : _prepare_read_resource(uri)
note right
approval_label = mcp_resource__{normalized_uri}
URI normalized (.. resolved)
needs_approval = True
end note
Session -> MCPMgr : read_resource_sync(uri)
MCPMgr -> MCPSrv : resources/read
MCPSrv --> MCPMgr : ReadResourceResult
MCPMgr --> Session : content (text/blob)
== Runtime: Prompt Invocation ==
Session -> Session : _prepare_use_prompt(name, arguments)
note right
approval_label = mcp__srv__prompt
Validated via is_mcp_prompt()
needs_approval = True
end note
Session -> MCPMgr : get_prompt_sync(name, args)
MCPMgr -> MCPSrv : prompts/get
MCPSrv --> MCPMgr : GetPromptResult
MCPMgr --> Session : messages [{role, content}]
== Three-Tier Refresh ==
group Push Notifications
MCPSrv -> MCPMgr : ToolListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_tools()
MCPSrv -> MCPMgr : ResourceListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_resources()
MCPSrv -> MCPMgr : PromptListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_prompts()
MCPMgr -> Storage : sync_prompts_to_storage()
end
group Periodic Polling (default 4h)
MCPMgr -> MCPMgr : _periodic_refresh()
note right
Only polls capabilities
without push support.
Staggered per-server.
end note
end
group Manual Refresh
Session -> MCPMgr : refresh_sync()
note right: /mcp refresh [server]
end
== Policy Evaluation ==
note over Session
Tool policies use fnmatch on approval_label:
- mcp__github__* → allow (all GitHub tools/prompts)
- mcp_resource__file:///docs/* → allow
- mcp_resource__* → deny (block all resource reads)
- mcp__untrusted__* → ask
end note
== UI Visibility ==
UI -> MCPMgr : server_count, get_resources(), get_prompts()
note over UI
/health → mcp.servers, mcp.resources, mcp.prompts
Server UI: magenta status badge
Console: cluster status bar + node detail
System message: <mcp-resources> + <mcp-prompts> catalogs
end note
@enduml
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0ee0a9391bd19d92e9271bf6bd531e9c2e18baf8c5a11ead49b3c10db4d8939b
size 329625
oid sha256:c9daca81971ba7a8ed6736d23d5373c69435158fa6240b9880d14fc4759ab580
size 329673
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:760c37e67736588dadee21d500419a48e9fc50f8bdc5667e686c580022bd40e2
size 554869
oid sha256:01fbb3338df6426cefc2811541a865f268673b4febf32f524c264d120bc068fa
size 589546
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a3ffd93ccb634f76560f1dd65242b89cd34443b37e355f25f29a1c63a22001be
size 265259
oid sha256:6dd3c923d1e1c49b5f91d8d342fb4b0d49a46d432460379ad146a9e3b075a05a
size 277234
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b68663599922f72d7ca21be820523a5b472c268897d194e5237bba2441c004ec
size 124497
oid sha256:a889d4bb84c4afa3c822c3acb7021395a9463462f7aeae5382e583b783412814
size 144960
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8e6dc5142c7908314ce01229b3c4f13bf9450adcbb62a178838bd4cf81d9f4da
size 250417
+31 -5
View File
@@ -43,15 +43,41 @@ Admin-defined rules that control tool execution:
- **Priority**: Higher priority evaluated first, first match wins
- **Enforcement**: `evaluate_tool_policies_batch()` called in `WebUI.approve_tools()`
before the `auto_approve` check
- **MCP granular policies**: MCP resources and prompts are evaluated using their
`approval_label` for fine-grained control:
- Resource reads: `mcp_resource__{uri}` (e.g., `mcp_resource__file:///docs/*` to allow,
`mcp_resource__*` to deny all)
- Prompt invocations: `mcp__{server}__{prompt}` (e.g., `mcp__trusted__*` to allow,
`mcp__*` to require approval for all)
- Built-in tools continue to use `func_name` for backward compatibility
### Prompt Templates
Reusable system message templates with variable substitution:
Admin-curated system message templates injected at workstream startup:
- **Variables**: `{{variable_name}}` placeholders in content
- **Categories**: general, engineering, support, custom
- **Default flag**: `is_default=true` templates intended for new workstreams
- **Storage**: `prompt_templates` table with JSON `variables` array
- **Runtime behavior**: Templates are loaded once at session creation and injected
into the system message *before* user `instructions`. Templates set the baseline;
instructions customize per-workstream behavior.
- **Default templates**: All `is_default=true` templates auto-apply to new
workstreams, concatenated in alphabetical order by name. Use name prefixes
(e.g. `01-safety`, `02-style`) to control ordering.
- **Explicit selection**: `--template <name>` CLI flag, `template` field on
`POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task
config, and channel adapter config. An explicit template *replaces* defaults.
- **Variables**: Three built-in placeholders resolved at load time:
`{{model}}` (active model name), `{{ws_id}}` (workstream ID),
`{{node_id}}` (server node ID). Unrecognized placeholders are kept as-is.
- **Runtime switching**: `/template <name>` to switch, `/template clear` to revert
to defaults, `/template` to show current. Persisted across resume.
- **Categories**: general, engineering, support, custom, mcp
- **Content limit**: 32 KB per template (enforced on create/update)
- **Storage**: `prompt_templates` table with JSON `variables` array. Migration 010
adds `template` column to `scheduled_tasks`.
- **MCP sync**: MCP server prompts auto-sync into prompt_templates with
`origin="mcp"`, `mcp_server` set, and `readonly=True`. Manual templates take
precedence on name collision. MCP-synced content updates reset `is_default` to
prevent compromised servers from injecting defaults. Admin UI shows origin badge
and disables edit/delete for MCP-sourced templates.
### Usage Tracking
+149 -6
View File
@@ -1,6 +1,6 @@
# Tools Reference
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
turnstone exposes 18 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -46,12 +46,12 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 16 tool definitions (sent to the model). |
| `TOOLS` | All 18 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 16 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 18 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -69,7 +69,7 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 15
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 18
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
@@ -168,6 +168,8 @@ Every tool defines a `primary_key`. The mapping is:
| `recall` | `query` |
| `forget` | `key` |
| `notify` | `message` |
| `read_resource` | `uri` |
| `use_prompt` | `name` |
---
@@ -517,6 +519,8 @@ data.get("mergedAt") is not None
| `forget` | Memory | Yes | No | No | `key` |
| `notify` | Notify | Yes | Yes | Yes | `message` |
| `watch` | Monitor | No (create) | No | No | `command` |
| `read_resource`| MCP | No | Yes | Yes | `uri` |
| `use_prompt` | MCP | No | Yes | Yes | `name` |
| `tool_search`| Search | Yes | No | No | `query` |
---
@@ -569,7 +573,7 @@ CLI flags override the config file:
search stays off and all tools are sent to the model directly.
2. **Partitioning**: When active, tools are split into two sets:
- **Always-on** -- the 15 built-in tools (members of `BUILTIN_TOOL_NAMES`).
- **Always-on** -- the 18 built-in tools (members of `BUILTIN_TOOL_NAMES`).
These are always visible to the model.
- **Deferred** -- all MCP tools. These are not sent in the tool list unless
the model searches for them.
@@ -593,6 +597,8 @@ where the model can interactively search for tools it needs.
## MCP Tools (External)
> See also: [MCP Architecture diagram](diagrams/png/20-mcp-architecture.png)
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/)
(MCP) for connecting external tool servers — GitHub, databases, filesystems, or any
MCP-compatible service.
@@ -610,7 +616,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 15 built-in tools via
4. **Merging**: MCP tools are appended after the 18 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
@@ -729,3 +735,140 @@ MCP refresh complete:
MCP refresh complete:
github: no changes
```
---
## MCP Resources
MCP servers can expose **resources** -- named data items (files, database rows,
API responses) addressable by URI. turnstone discovers resources at startup and
makes them available to the model via the `read_resource` built-in tool.
### Discovery
During the MCP `initialize` handshake, `MCPClientManager` checks each server's
capabilities for the `resources` capability. For servers that declare it:
1. `list_resources` fetches static resources (fixed URIs).
2. `list_resource_templates` fetches URI templates (parameterized patterns like
`db://tables/{table}/rows/{id}`).
Both are stored as `{uri, name, description, mimeType, server}` dicts and
merged into a unified catalog.
### Resource catalog in system message
The first 50 resources are injected into the system message as an XML-delimited
block so the model knows what URIs are available:
```xml
<mcp-resources>
file:///project/README.md Project readme
db://users/schema User table schema
</mcp-resources>
Use read_resource(uri='...') to access the resources listed above.
```
### read_resource tool
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `uri` | string | yes | The resource URI to read. |
- **What it does**: Reads the resource from its MCP server via `MCPClientManager.read_resource_sync()`. Returns text content for text resources or base64-encoded data for binary resources. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (reads external data).
- **Agent availability**: `agent` and `task_agent`.
### Capability guards
The `read_resource` tool schema is always loaded (it is a built-in JSON schema),
but resource discovery only runs for servers that declare the `resources`
capability. Servers without the capability contribute zero resources to the
catalog.
### Refresh
Resource lists stay current through the same three-tier mechanism as tool lists:
1. **Push** -- Servers declaring `resources.listChanged: true` send
`notifications/resources/list_changed`, triggering an immediate refresh.
2. **Periodic** -- Servers without push are polled on the configured refresh
interval (default 4 hours, same timer as tools).
3. **Manual** -- `/mcp refresh` re-fetches resources alongside tools.
---
## MCP Prompts
MCP servers can also expose **prompts** -- reusable message templates with
optional arguments. turnstone discovers prompts at startup for servers that
declare the `prompts` capability.
### Discovery
Prompt discovery mirrors resource discovery: `list_prompts` is called during
the `initialize` handshake. Each prompt is stored with its prefixed name
(`mcp__{server}__{prompt}`), description, and argument schema.
### use_prompt tool
| Parameter | Type | Required | Description |
|-------------|--------|----------|-------------|
| `name` | string | yes | The prompt name (e.g. `mcp__server__prompt_name`). |
| `arguments` | object | no | Key-value argument pairs for the prompt. Values must be strings. |
- **What it does**: Invokes an MCP prompt template by name via `MCPClientManager.get_prompt_sync()`, expanding it into messages. Returns the expanded prompt content formatted as `[role]: content` blocks joined with blank lines. The prompt catalog is listed in the system message so the model knows which prompts are available. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (invokes external prompt servers).
- **Agent availability**: `agent` and `task_agent`.
### Invocation
`MCPClientManager.get_prompt_sync()` calls the server's `get_prompt` method
with the provided arguments and returns the expanded messages. The `use_prompt`
built-in tool exposes this to the model as a function call.
### Governance Sync
Discovered MCP prompts are automatically synced into the `prompt_templates`
governance table as first-class governed templates:
- **Origin tracking**: MCP-sourced templates have `origin="mcp"` and
`mcp_server` set to the server name. Manual templates have
`origin="manual"`.
- **Read-only**: MCP-sourced templates are `readonly=True`. The admin API
returns 403 on update/delete attempts. The admin UI disables edit/delete
buttons and shows an origin badge.
- **Precedence**: If a manual template and MCP prompt share the same name,
the manual template wins and the MCP prompt is skipped (with a log
warning).
- **Lifecycle**: Templates are created on connect, updated on prompt list
refresh, and removed when the MCP server no longer exposes the prompt.
The sync runs automatically on connect, on `PromptListChangedNotification`,
and on manual `/mcp refresh`.
- **Schema**: Migration 009 adds `origin`, `mcp_server`, and `readonly`
columns to the `prompt_templates` table.
The `use_prompt` tool allows the model to invoke any discovered MCP prompt at
runtime. A catalog of up to 30 prompts is injected into the system message
inside `<mcp-prompts>` XML tags so the model can discover available prompts.
---
## MCP UI Visibility
MCP server, resource, and prompt counts are surfaced across the UI:
- **Server `/health` endpoint**: Returns `mcp.servers`, `mcp.resources`,
`mcp.prompts` when MCP is configured
- **Server UI**: Magenta status badge in the header showing server count,
with resource/prompt counts in tooltip
- **Console cluster status bar**: MCP metrics (servers/resources/prompts)
with magenta LED dot indicator, shown after a divider from workstream
metrics
- **Console node detail**: Per-node MCP summary showing server, resource,
and prompt counts
- **Console collector**: Aggregates MCP counts across all nodes in the
cluster overview
MCP indicators use the `--magenta` design token for consistent theming
across light and dark modes.
+2 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "0.5.4"
version = "0.5.6"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -62,6 +62,7 @@ turnstone-console = "turnstone.console.server:main"
turnstone-sim = "turnstone.sim.cli:main"
turnstone-admin = "turnstone.admin:main"
turnstone-channel = "turnstone.channels.cli:main"
turnstone-bootstrap = "turnstone.bootstrap:main"
[tool.hatch.build.targets.wheel]
include = [
+29 -78
View File
@@ -10,9 +10,7 @@
"get": {
"summary": "Cluster state summary",
"operationId": "v1_api_cluster_overview_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"responses": {
"200": {
"description": "Success",
@@ -31,9 +29,7 @@
"get": {
"summary": "Paginated node list",
"operationId": "v1_api_cluster_nodes_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"parameters": [
{
"name": "sort",
@@ -42,11 +38,7 @@
"schema": {
"type": "string",
"default": "activity",
"enum": [
"activity",
"tokens",
"name"
]
"enum": ["activity", "tokens", "name"]
},
"description": "Sort field"
},
@@ -89,9 +81,7 @@
"get": {
"summary": "Filtered workstream list",
"operationId": "v1_api_cluster_workstreams_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"parameters": [
{
"name": "state",
@@ -99,13 +89,7 @@
"required": false,
"schema": {
"type": "string",
"enum": [
"running",
"thinking",
"attention",
"idle",
"error"
]
"enum": ["running", "thinking", "attention", "idle", "error"]
},
"description": "Filter by state"
},
@@ -134,11 +118,7 @@
"schema": {
"type": "string",
"default": "state",
"enum": [
"state",
"tokens",
"name"
]
"enum": ["state", "tokens", "name"]
},
"description": "Sort field"
},
@@ -181,9 +161,7 @@
"get": {
"summary": "Single node detail",
"operationId": "v1_api_cluster_node_{node_id}_get",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"parameters": [
{
"name": "node_id",
@@ -222,9 +200,7 @@
"post": {
"summary": "Create workstream via MQ dispatch",
"operationId": "v1_api_cluster_workstreams_new_post",
"tags": [
"Cluster"
],
"tags": ["Cluster"],
"requestBody": {
"required": true,
"content": {
@@ -283,9 +259,7 @@
"get": {
"summary": "Cluster SSE event stream",
"operationId": "v1_api_cluster_events_get",
"tags": [
"Streaming"
],
"tags": ["Streaming"],
"description": "Server-Sent Events stream for real-time cluster updates. Returns text/event-stream with node_joined, node_lost, cluster_state, ws_created, ws_closed, ws_rename events.",
"responses": {
"200": {
@@ -298,9 +272,7 @@
"post": {
"summary": "Authenticate with a token",
"operationId": "v1_api_auth_login_post",
"tags": [
"Auth"
],
"tags": ["Auth"],
"requestBody": {
"required": true,
"content": {
@@ -339,9 +311,7 @@
"post": {
"summary": "Clear auth cookie",
"operationId": "v1_api_auth_logout_post",
"tags": [
"Auth"
],
"tags": ["Auth"],
"responses": {
"200": {
"description": "Success",
@@ -360,9 +330,7 @@
"get": {
"summary": "Console health check",
"operationId": "health_get",
"tags": [
"Observability"
],
"tags": ["Observability"],
"responses": {
"200": {
"description": "Success",
@@ -389,9 +357,7 @@
"type": "string"
}
},
"required": [
"error"
],
"required": ["error"],
"title": "ErrorResponse",
"type": "object"
},
@@ -400,9 +366,7 @@
"properties": {
"status": {
"default": "ok",
"examples": [
"ok"
],
"examples": ["ok"],
"title": "Status",
"type": "string"
}
@@ -419,9 +383,7 @@
"type": "string"
}
},
"required": [
"token"
],
"required": ["token"],
"title": "AuthLoginRequest",
"type": "object"
},
@@ -435,17 +397,12 @@
},
"role": {
"description": "Assigned role",
"examples": [
"full",
"read"
],
"examples": ["full", "read"],
"title": "Role",
"type": "string"
}
},
"required": [
"role"
],
"required": ["role"],
"title": "AuthLoginResponse",
"type": "object"
},
@@ -557,9 +514,7 @@
"type": "integer"
}
},
"required": [
"nodes"
],
"required": ["nodes"],
"title": "ClusterNodesResponse",
"type": "object"
},
@@ -632,9 +587,7 @@
"type": "string"
}
},
"required": [
"node_id"
],
"required": ["node_id"],
"title": "ClusterNodeInfo",
"type": "object"
},
@@ -668,9 +621,7 @@
"type": "integer"
}
},
"required": [
"workstreams"
],
"required": ["workstreams"],
"title": "ClusterWorkstreamsResponse",
"type": "object"
},
@@ -726,9 +677,7 @@
"type": "integer"
}
},
"required": [
"id"
],
"required": ["id"],
"title": "ClusterWorkstreamInfo",
"type": "object"
},
@@ -771,9 +720,7 @@
"type": "boolean"
}
},
"required": [
"node_id"
],
"required": ["node_id"],
"title": "NodeDetailResponse",
"type": "object"
},
@@ -802,6 +749,12 @@
"description": "Optional first message sent after creation",
"title": "Initial Message",
"type": "string"
},
"template": {
"default": "",
"description": "Prompt template name (replaces default templates)",
"title": "Template",
"type": "string"
}
},
"title": "ConsoleCreateWsRequest",
@@ -832,9 +785,7 @@
"properties": {
"status": {
"default": "ok",
"examples": [
"ok"
],
"examples": ["ok"],
"title": "Status",
"type": "string"
},
+38
View File
@@ -877,6 +877,12 @@
"description": "Workstream ID to resume atomically during creation (empty = fresh start)",
"title": "Resume Ws",
"type": "string"
},
"template": {
"default": "",
"description": "Prompt template name (replaces default templates)",
"title": "Template",
"type": "string"
}
},
"title": "CreateWorkstreamRequest",
@@ -1177,12 +1183,44 @@
}
],
"default": null
},
"mcp": {
"anyOf": [
{
"$ref": "#/components/schemas/McpStatus"
},
{
"type": "null"
}
],
"default": null
}
},
"required": ["status"],
"title": "HealthResponse",
"type": "object"
},
"McpStatus": {
"properties": {
"servers": {
"default": 0,
"title": "Servers",
"type": "integer"
},
"resources": {
"default": 0,
"title": "Resources",
"type": "integer"
},
"prompts": {
"default": 0,
"title": "Prompts",
"type": "integer"
}
},
"title": "McpStatus",
"type": "object"
},
"BackendStatus": {
"properties": {
"status": {
+13
View File
@@ -48,6 +48,12 @@ export interface ApproveRequestEvent {
items: Array<Record<string, unknown>>;
}
export interface ApprovalResolvedEvent {
type: "approval_resolved";
approved: boolean;
feedback: string;
}
export interface ToolResultEvent {
type: "tool_result";
call_id: string;
@@ -141,6 +147,7 @@ export type ServerEvent =
| StreamEndEvent
| ToolInfoEvent
| ApproveRequestEvent
| ApprovalResolvedEvent
| ToolResultEvent
| ToolOutputChunkEvent
| StatusEvent
@@ -249,6 +256,12 @@ export function isApproveRequestEvent(
return e.type === "approve_request";
}
export function isApprovalResolvedEvent(
e: ServerEvent,
): e is ApprovalResolvedEvent {
return e.type === "approval_resolved";
}
export function isPlanReviewEvent(e: ServerEvent): e is PlanReviewEvent {
return e.type === "plan_review";
}
+3
View File
@@ -37,6 +37,7 @@ export type {
StreamEndEvent,
ToolInfoEvent,
ApproveRequestEvent,
ApprovalResolvedEvent,
ToolResultEvent,
ToolOutputChunkEvent,
StatusEvent,
@@ -67,6 +68,7 @@ export {
isToolResultEvent,
isWsStateEvent,
isApproveRequestEvent,
isApprovalResolvedEvent,
isPlanReviewEvent,
isCancelledEvent,
} from "./events.js";
@@ -89,6 +91,7 @@ export type {
SavedWorkstreamInfo,
ListSavedWorkstreamsResponse,
BackendStatus,
McpStatus,
WorkstreamCounts,
HealthResponse,
AuthLoginRequest,
+12
View File
@@ -72,6 +72,7 @@ export interface CreateWorkstreamRequest {
model?: string;
auto_approve?: boolean;
resume_ws?: string;
template?: string;
}
export interface CreateWorkstreamResponse {
@@ -159,6 +160,12 @@ export interface WorkstreamCounts {
error?: number;
}
export interface McpStatus {
servers: number;
resources: number;
prompts: number;
}
export interface HealthResponse {
status: string;
version?: string;
@@ -166,6 +173,7 @@ export interface HealthResponse {
model?: string;
workstreams?: WorkstreamCounts;
backend?: BackendStatus | null;
mcp?: McpStatus | null;
}
// ---------------------------------------------------------------------------
@@ -266,6 +274,7 @@ export interface ConsoleCreateWsRequest {
name?: string;
model?: string;
initial_message?: string;
template?: string;
}
export interface ConsoleCreateWsResponse {
@@ -452,6 +461,9 @@ export interface PromptTemplateInfo {
created_by: string;
created: string;
updated: string;
origin: string;
mcp_server: string;
readonly: boolean;
}
export interface CreateTemplateOptions {
+10
View File
@@ -6,6 +6,7 @@ import {
isToolResultEvent,
isWsStateEvent,
isApproveRequestEvent,
isApprovalResolvedEvent,
isPlanReviewEvent,
isReasoningEvent,
} from "../src/events.js";
@@ -62,6 +63,15 @@ describe("event type guards", () => {
expect(isApproveRequestEvent(e)).toBe(true);
});
it("isApprovalResolvedEvent", () => {
const e: ServerEvent = {
type: "approval_resolved",
approved: false,
feedback: "Approval timed out",
};
expect(isApprovalResolvedEvent(e)).toBe(true);
});
it("isPlanReviewEvent", () => {
const e: ServerEvent = { type: "plan_review", content: "## Plan" };
expect(isPlanReviewEvent(e)).toBe(true);
+630
View File
@@ -0,0 +1,630 @@
"""Tests for the bootstrap wizard module."""
from __future__ import annotations
import os
import socket
from pathlib import Path
from unittest.mock import MagicMock, patch
from turnstone.bootstrap import (
SYSTEM_PROMPT,
TOOLS,
_BootstrapLLM,
_FinishError,
_mask_secrets,
_tool_check_docker,
_tool_check_port,
_tool_finish,
_tool_generate_secret,
_tool_read_file,
_tool_validate_api_key,
_tool_write_file,
execute_tool,
)
# ---------------------------------------------------------------------------
# Tool function tests
# ---------------------------------------------------------------------------
class TestReadFile:
def test_existing_file(self, tmp_path: Path) -> None:
f = tmp_path / "test.txt"
f.write_text("hello world")
result = _tool_read_file(tmp_path, {"path": "test.txt"})
assert result == "hello world"
def test_missing_file(self, tmp_path: Path) -> None:
result = _tool_read_file(tmp_path, {"path": "nope.txt"})
assert "Error: file not found" in result
def test_nested_path(self, tmp_path: Path) -> None:
sub = tmp_path / "sub"
sub.mkdir()
f = sub / "nested.txt"
f.write_text("nested content")
result = _tool_read_file(tmp_path, {"path": "sub/nested.txt"})
assert result == "nested content"
def test_path_traversal_blocked(self, tmp_path: Path) -> None:
result = _tool_read_file(tmp_path, {"path": "../../etc/passwd"})
assert "escapes project directory" in result
def test_absolute_path_blocked(self, tmp_path: Path) -> None:
result = _tool_read_file(tmp_path, {"path": "/etc/passwd"})
assert "escapes project directory" in result
class TestWriteFile:
def test_write_confirmed(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="y"):
result = _tool_write_file(tmp_path, {"path": "out.txt", "content": "data\n"})
assert "written successfully" in result
assert (tmp_path / "out.txt").read_text() == "data\n"
def test_write_declined(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="n"):
result = _tool_write_file(tmp_path, {"path": "out.txt", "content": "data\n"})
assert "declined" in result
assert not (tmp_path / "out.txt").exists()
def test_write_creates_parent_dirs(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="y"):
result = _tool_write_file(tmp_path, {"path": "a/b/c.txt", "content": "deep\n"})
assert "written successfully" in result
assert (tmp_path / "a" / "b" / "c.txt").read_text() == "deep\n"
def test_sh_files_are_executable(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value="y"):
_tool_write_file(tmp_path, {"path": "setup.sh", "content": "#!/bin/bash\n"})
mode = (tmp_path / "setup.sh").stat().st_mode
assert mode & 0o110 # user + group executable, not world
def test_path_traversal_blocked(self, tmp_path: Path) -> None:
result = _tool_write_file(tmp_path, {"path": "../../escape.txt", "content": "bad\n"})
assert "escapes project directory" in result
def test_default_enter_confirms(self, tmp_path: Path) -> None:
with patch("builtins.input", return_value=""):
result = _tool_write_file(tmp_path, {"path": "ok.txt", "content": "ok\n"})
assert "written successfully" in result
def test_duplicate_write_skipped(self, tmp_path: Path) -> None:
(tmp_path / "dup.txt").write_text("same\n")
result = _tool_write_file(tmp_path, {"path": "dup.txt", "content": "same\n"})
assert "already exists" in result
def test_different_content_still_prompts(self, tmp_path: Path) -> None:
(tmp_path / "changed.txt").write_text("old\n")
with patch("builtins.input", return_value="y"):
result = _tool_write_file(tmp_path, {"path": "changed.txt", "content": "new\n"})
assert "written successfully" in result
assert (tmp_path / "changed.txt").read_text() == "new\n"
class TestGenerateSecret:
def test_default_length(self) -> None:
secret = _tool_generate_secret({})
assert len(secret) == 64 # 32 bytes -> 64 hex chars
def test_custom_length(self) -> None:
secret = _tool_generate_secret({"length": 16})
assert len(secret) == 32
def test_uniqueness(self) -> None:
s1 = _tool_generate_secret({})
s2 = _tool_generate_secret({})
assert s1 != s2
def test_invalid_length_fallback(self) -> None:
secret = _tool_generate_secret({"length": -1})
assert len(secret) == 64 # falls back to 32 bytes
def test_excessive_length_capped(self) -> None:
secret = _tool_generate_secret({"length": 99999})
assert len(secret) == 64 # falls back to 32 bytes
class TestCheckPort:
def test_available_port(self) -> None:
# Pick a random high port that's likely free
result = _tool_check_port({"port": 59123})
assert "AVAILABLE" in result or "IN USE" in result
def test_in_use_port(self) -> None:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("127.0.0.1", 0))
port = sock.getsockname()[1]
sock.listen(1)
result = _tool_check_port({"port": port})
assert "IN USE" in result
def test_invalid_port(self) -> None:
result = _tool_check_port({"port": -1})
assert "Error" in result
def test_port_zero(self) -> None:
result = _tool_check_port({"port": 0})
assert "Error" in result
class TestCheckDocker:
def test_docker_installed(self) -> None:
mock_docker = MagicMock()
mock_docker.returncode = 0
mock_docker.stdout = "24.0.7"
mock_compose = MagicMock()
mock_compose.returncode = 0
mock_compose.stdout = "2.24.5"
with patch("subprocess.run", side_effect=[mock_docker, mock_compose]):
result = _tool_check_docker({})
assert "Docker: installed" in result
assert "Docker Compose: installed" in result
def test_docker_not_installed(self) -> None:
with patch("subprocess.run", side_effect=FileNotFoundError):
result = _tool_check_docker({})
assert "NOT installed" in result or "NOT available" in result
def test_docker_daemon_not_running(self) -> None:
mock_docker = MagicMock()
mock_docker.returncode = 1
mock_docker.stderr = "Cannot connect to the Docker daemon"
mock_compose = MagicMock()
mock_compose.returncode = 1
with patch("subprocess.run", side_effect=[mock_docker, mock_compose]):
result = _tool_check_docker({})
assert "NOT running" in result
class TestValidateApiKey:
def test_openai_success(self) -> None:
mock_client = MagicMock()
mock_client.models.list.return_value = []
with patch("openai.OpenAI", return_value=mock_client):
result = _tool_validate_api_key({"provider": "openai", "api_key": "sk-test"})
assert "Success" in result
def test_openai_failure(self) -> None:
with patch("openai.OpenAI") as mock_cls:
mock_cls.return_value.models.list.side_effect = Exception("Invalid key")
result = _tool_validate_api_key({"provider": "openai", "api_key": "bad"})
assert "Failed" in result
def test_unknown_provider(self) -> None:
result = _tool_validate_api_key({"provider": "unknown", "api_key": "x"})
assert "unknown" in result
class TestExecuteTool:
def test_unknown_tool(self, tmp_path: Path) -> None:
result = execute_tool("nonexistent", {}, tmp_path)
assert "unknown tool" in result
def test_dispatches_correctly(self, tmp_path: Path) -> None:
f = tmp_path / "hello.txt"
f.write_text("hi")
result = execute_tool("read_file", {"path": "hello.txt"}, tmp_path)
assert result == "hi"
def test_finish_raises(self, tmp_path: Path) -> None:
import pytest
with pytest.raises(_FinishError, match="All done"):
execute_tool("finish", {"summary": "All done"}, tmp_path)
class TestFinishTool:
def test_raises_with_summary(self) -> None:
import pytest
with pytest.raises(_FinishError) as exc_info:
_tool_finish({"summary": "Configured production deployment."})
assert exc_info.value.summary == "Configured production deployment."
def test_default_summary(self) -> None:
import pytest
with pytest.raises(_FinishError) as exc_info:
_tool_finish({})
assert exc_info.value.summary == "Setup complete."
# ---------------------------------------------------------------------------
# Secret masking tests
# ---------------------------------------------------------------------------
class TestMaskSecrets:
def test_masks_api_key(self) -> None:
text = "OPENAI_API_KEY=sk-1234567890abcdef"
result = _mask_secrets(text)
assert "sk-1" in result
assert "cdef" in result
assert "1234567890abcde" not in result
def test_preserves_comments(self) -> None:
text = "# OPENAI_API_KEY=sk-1234567890abcdef"
result = _mask_secrets(text)
assert result == text
def test_preserves_short_values(self) -> None:
text = "TOKEN=short"
result = _mask_secrets(text)
assert result == text
def test_preserves_non_sensitive(self) -> None:
text = "MODEL=gpt-5.4"
result = _mask_secrets(text)
assert result == text
# ---------------------------------------------------------------------------
# Message conversion tests (Anthropic)
# ---------------------------------------------------------------------------
class TestAnthropicConversion:
"""Test the Anthropic message/tool conversion inside _BootstrapLLM."""
def _make_llm(self) -> _BootstrapLLM:
return _BootstrapLLM("anthropic", MagicMock(), "test-model")
def test_tool_format_conversion(self) -> None:
"""OpenAI tool format should convert to Anthropic format."""
llm = self._make_llm()
# The conversion happens inside _complete_anthropic; we test indirectly
# by checking the tools passed to the mock client
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="hello")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
llm.complete(
[{"role": "system", "content": "sys"}, {"role": "user", "content": "hi"}],
TOOLS[:1], # Just read_file
)
call_kwargs = llm.client.messages.create.call_args[1]
api_tools = call_kwargs["tools"]
assert len(api_tools) == 1
assert api_tools[0]["name"] == "read_file"
assert "input_schema" in api_tools[0]
assert "description" in api_tools[0]
def test_system_message_extraction(self) -> None:
"""System message should be extracted to system parameter."""
llm = self._make_llm()
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="ok")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
llm.complete(
[{"role": "system", "content": "test system"}, {"role": "user", "content": "hi"}],
[],
)
call_kwargs = llm.client.messages.create.call_args[1]
assert call_kwargs["system"] == "test system"
# System should NOT appear in messages
for msg in call_kwargs["messages"]:
assert msg["role"] != "system"
def test_tool_result_conversion(self) -> None:
"""OpenAI tool result messages should convert to Anthropic format."""
llm = self._make_llm()
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="got it")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
messages = [
{"role": "system", "content": "sys"},
{"role": "user", "content": "hi"},
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": "tc_1",
"type": "function",
"function": {"name": "check_docker", "arguments": "{}"},
}
],
},
{
"role": "tool",
"tool_call_id": "tc_1",
"content": "Docker: installed",
},
]
llm.complete(messages, TOOLS)
call_kwargs = llm.client.messages.create.call_args[1]
api_messages = call_kwargs["messages"]
# Find the tool_result message
tool_result_found = False
for msg in api_messages:
if msg["role"] == "user" and isinstance(msg.get("content"), list):
for block in msg["content"]:
if isinstance(block, dict) and block.get("type") == "tool_result":
assert block["tool_use_id"] == "tc_1"
assert block["content"] == "Docker: installed"
tool_result_found = True
assert tool_result_found
def test_tool_use_blocks_in_assistant(self) -> None:
"""Assistant messages with tool_calls should convert to content blocks."""
llm = self._make_llm()
mock_response = MagicMock()
mock_response.content = [MagicMock(type="text", text="ok")]
mock_response.stop_reason = "end_turn"
llm.client.messages.create.return_value = mock_response
messages = [
{"role": "system", "content": "sys"},
{"role": "user", "content": "hi"},
{
"role": "assistant",
"content": "Let me check",
"tool_calls": [
{
"id": "tc_1",
"type": "function",
"function": {"name": "check_docker", "arguments": "{}"},
}
],
},
{"role": "tool", "tool_call_id": "tc_1", "content": "ok"},
]
llm.complete(messages, TOOLS)
call_kwargs = llm.client.messages.create.call_args[1]
api_messages = call_kwargs["messages"]
# First message should be user "hi"
assert api_messages[0]["role"] == "user"
# Second should be assistant with content blocks
assistant_msg = api_messages[1]
assert assistant_msg["role"] == "assistant"
assert isinstance(assistant_msg["content"], list)
# Should have text block + tool_use block
types = [b["type"] for b in assistant_msg["content"]]
assert "text" in types
assert "tool_use" in types
class TestOpenAICompletion:
"""Test the OpenAI path of _BootstrapLLM."""
def test_text_response(self) -> None:
llm = _BootstrapLLM("openai", MagicMock(), "gpt-5.4")
mock_choice = MagicMock()
mock_choice.message.content = "Hello!"
mock_choice.message.tool_calls = None
mock_choice.finish_reason = "stop"
llm.client.chat.completions.create.return_value = MagicMock(choices=[mock_choice])
content, tool_calls, reason = llm.complete([{"role": "user", "content": "hi"}], TOOLS)
assert content == "Hello!"
assert tool_calls is None
assert reason == "stop"
def test_tool_call_response(self) -> None:
llm = _BootstrapLLM("openai", MagicMock(), "gpt-5.4")
mock_tc = MagicMock()
mock_tc.id = "call_123"
mock_tc.function.name = "check_docker"
mock_tc.function.arguments = "{}"
mock_choice = MagicMock()
mock_choice.message.content = ""
mock_choice.message.tool_calls = [mock_tc]
mock_choice.finish_reason = "tool_calls"
llm.client.chat.completions.create.return_value = MagicMock(choices=[mock_choice])
content, tool_calls, reason = llm.complete(
[{"role": "user", "content": "check docker"}], TOOLS
)
assert tool_calls is not None
assert len(tool_calls) == 1
assert tool_calls[0]["function"]["name"] == "check_docker"
assert tool_calls[0]["id"] == "call_123"
def test_no_content(self) -> None:
llm = _BootstrapLLM("openai", MagicMock(), "gpt-5.4")
mock_choice = MagicMock()
mock_choice.message.content = None
mock_choice.message.tool_calls = None
mock_choice.finish_reason = "stop"
llm.client.chat.completions.create.return_value = MagicMock(choices=[mock_choice])
content, tool_calls, reason = llm.complete([{"role": "user", "content": "hi"}], [])
assert content == ""
assert tool_calls is None
# ---------------------------------------------------------------------------
# Conversation loop tests
# ---------------------------------------------------------------------------
class TestConversationLoop:
def test_quit_exits(self) -> None:
"""User typing 'quit' should exit the loop."""
llm = MagicMock(spec=_BootstrapLLM)
llm.complete.return_value = ("What would you like?", None, "stop")
with patch("builtins.input", return_value="quit"):
from turnstone.bootstrap import _run_conversation
_run_conversation(llm, Path("/tmp"))
def test_tool_calls_executed(self, tmp_path: Path) -> None:
"""Tool calls should be executed and results fed back."""
llm = MagicMock(spec=_BootstrapLLM)
# First call: LLM returns a tool call
llm.complete.side_effect = [
(
"",
[
{
"id": "tc_1",
"type": "function",
"function": {"name": "generate_secret", "arguments": "{}"},
}
],
"tool_calls",
),
# Second call: LLM responds with text after seeing tool result
("Here's your secret!", None, "stop"),
]
with patch("builtins.input", return_value="quit"):
from turnstone.bootstrap import _run_conversation
_run_conversation(llm, tmp_path)
# Verify two calls were made
assert llm.complete.call_count == 2
# Verify tool result was fed back in second call's messages
second_call_messages = llm.complete.call_args_list[1][0][0]
tool_results = [m for m in second_call_messages if m.get("role") == "tool"]
assert len(tool_results) == 1
assert tool_results[0]["tool_call_id"] == "tc_1"
# Result should be a 64-char hex string
assert len(tool_results[0]["content"]) == 64
def test_empty_input_skipped(self) -> None:
"""Empty user input should be skipped."""
llm = MagicMock(spec=_BootstrapLLM)
llm.complete.return_value = ("Ask me something.", None, "stop")
call_count = 0
def mock_input(prompt: str = "") -> str:
nonlocal call_count
call_count += 1
if call_count <= 2:
return "" # Empty inputs
return "quit"
with patch("builtins.input", side_effect=mock_input):
from turnstone.bootstrap import _run_conversation
_run_conversation(llm, Path("/tmp"))
def test_finish_tool_exits_loop(self, tmp_path: Path) -> None:
"""LLM calling finish tool should exit the conversation cleanly."""
llm = MagicMock(spec=_BootstrapLLM)
llm.complete.return_value = (
"",
[
{
"id": "tc_fin",
"type": "function",
"function": {
"name": "finish",
"arguments": '{"summary": "All configured."}',
},
}
],
"tool_calls",
)
from turnstone.bootstrap import _run_conversation
# Should return without needing user input
_run_conversation(llm, tmp_path)
assert llm.complete.call_count == 1
# ---------------------------------------------------------------------------
# Interactive startup tests
# ---------------------------------------------------------------------------
class TestProviderDefaults:
def test_openai_default_model(self) -> None:
from turnstone.bootstrap import _DEFAULT_MODELS
assert _DEFAULT_MODELS["openai"] == "gpt-5.4"
def test_anthropic_default_model(self) -> None:
from turnstone.bootstrap import _DEFAULT_MODELS
assert _DEFAULT_MODELS["anthropic"] == "claude-sonnet-4-6"
class TestSelectProvider:
def test_openai_selection(self) -> None:
"""Selecting '1' should set up OpenAI."""
mock_client = MagicMock()
with (
patch("builtins.input", side_effect=["1", ""]),
patch("getpass.getpass", return_value="sk-test"),
patch("openai.OpenAI", return_value=mock_client),
):
from turnstone.bootstrap import _select_provider
provider, client, model = _select_provider()
assert provider == "openai"
assert model == "gpt-5.4"
def test_local_selection(self) -> None:
"""Selecting '3' should set up local/vLLM."""
mock_client = MagicMock()
# Ensure OPENAI_API_KEY is not in env so we hit the getpass path
env = {k: v for k, v in os.environ.items() if k != "OPENAI_API_KEY"}
with (
patch.dict("os.environ", env, clear=True),
patch("builtins.input", side_effect=["3", "http://localhost:8000/v1", "my-model"]),
patch("getpass.getpass", return_value="none"),
patch("openai.OpenAI", return_value=mock_client),
):
from turnstone.bootstrap import _select_provider
provider, client, model = _select_provider()
assert provider == "openai"
assert model == "my-model"
# ---------------------------------------------------------------------------
# System prompt and tools sanity checks
# ---------------------------------------------------------------------------
class TestConstants:
def test_system_prompt_not_empty(self) -> None:
assert len(SYSTEM_PROMPT) > 500
def test_system_prompt_mentions_turnstone(self) -> None:
assert "Turnstone" in SYSTEM_PROMPT
def test_all_tools_have_required_fields(self) -> None:
for tool in TOOLS:
assert tool["type"] == "function"
func = tool["function"]
assert "name" in func
assert "description" in func
assert "parameters" in func
assert func["parameters"]["type"] == "object"
def test_tool_count(self) -> None:
assert len(TOOLS) == 7
def test_all_tools_have_implementations(self) -> None:
from turnstone.bootstrap import TOOL_FUNCTIONS
for tool in TOOLS:
name = tool["function"]["name"]
assert name in TOOL_FUNCTIONS, f"Missing implementation for tool: {name}"
+69
View File
@@ -0,0 +1,69 @@
"""Tests for bridge event publishing — TurnCompleteEvent on idle transitions."""
from unittest.mock import MagicMock, patch
from turnstone.mq.bridge import Bridge
from turnstone.mq.protocol import StateChangeEvent, TurnCompleteEvent
def _make_bridge():
"""Create a Bridge with a mock broker (no Redis or HTTP needed)."""
broker = MagicMock()
bridge = Bridge(server_url="http://localhost:8080", broker=broker, node_id="test-node")
return bridge
class TestIdleTurnComplete:
"""TurnCompleteEvent should be emitted on every idle transition."""
def test_idle_emits_turn_complete_with_correlation_id(self):
"""Bridge-initiated turn: TurnCompleteEvent has the correlation_id."""
bridge = _make_bridge()
bridge._active_sends["ws-1"] = "cid-abc"
published = []
with patch.object(
bridge, "_publish_ws", side_effect=lambda ws, ev: published.append((ws, ev))
):
bridge._handle_global_event({"type": "ws_state", "ws_id": "ws-1", "state": "idle"})
turn_completes = [(ws, ev) for ws, ev in published if isinstance(ev, TurnCompleteEvent)]
assert len(turn_completes) == 1
ws, ev = turn_completes[0]
assert ws == "ws-1"
assert ev.correlation_id == "cid-abc"
# correlation_id should be removed from _active_sends
assert "ws-1" not in bridge._active_sends
def test_idle_emits_turn_complete_without_correlation_id(self):
"""Server-UI-initiated turn: TurnCompleteEvent has empty correlation_id."""
bridge = _make_bridge()
# No entry in _active_sends for this workstream
published = []
with patch.object(
bridge, "_publish_ws", side_effect=lambda ws, ev: published.append((ws, ev))
):
bridge._handle_global_event({"type": "ws_state", "ws_id": "ws-2", "state": "idle"})
turn_completes = [(ws, ev) for ws, ev in published if isinstance(ev, TurnCompleteEvent)]
assert len(turn_completes) == 1
ws, ev = turn_completes[0]
assert ws == "ws-2"
assert ev.correlation_id == ""
def test_non_idle_state_does_not_emit_turn_complete(self):
"""Non-idle state transitions should emit StateChangeEvent but not TurnCompleteEvent."""
bridge = _make_bridge()
published = []
with patch.object(
bridge, "_publish_ws", side_effect=lambda ws, ev: published.append((ws, ev))
):
bridge._handle_global_event({"type": "ws_state", "ws_id": "ws-3", "state": "thinking"})
state_changes = [ev for _, ev in published if isinstance(ev, StateChangeEvent)]
turn_completes = [ev for _, ev in published if isinstance(ev, TurnCompleteEvent)]
assert len(state_changes) == 1
assert state_changes[0].state == "thinking"
assert len(turn_completes) == 0
+69
View File
@@ -335,3 +335,72 @@ class TestGenerationCancelledException:
raise GenerationCancelled()
except Exception:
pytest.fail("GenerationCancelled was caught by except Exception")
class TestStreamFlushBeforeToolCalls:
"""Content pending buffer must be flushed before tool call processing."""
def test_pending_content_flushed_before_tool_calls(self, tmp_db):
"""All content tokens arrive via on_content_token before tool calls."""
events: list[tuple[str, ...]] = []
class TrackingUI(NullUI):
def on_content_token(self, text):
events.append(("content", text))
def on_stream_end(self):
events.append(("stream_end",))
super().on_stream_end()
ui = TrackingUI()
session = _make_session(ui=ui)
@dataclass
class FakeChunk:
content_delta: str = ""
reasoning_delta: str = ""
tool_call_deltas: list = field(default_factory=list)
usage: None = None
finish_reason: str = ""
info_delta: str = ""
provider_blocks: list = field(default_factory=list)
@dataclass
class FakeToolDelta:
index: int = 0
id: str = ""
name: str = ""
arguments_delta: str = ""
def stream_content_then_tool():
# Content long enough to leave chars in pending buffer
# (_MAX_TAG_LEN = 13, so _drain_pending retains last 13 chars)
yield FakeChunk(content_delta="Hello world, this is a test message")
yield FakeChunk(
tool_call_deltas=[FakeToolDelta(index=0, id="tc_1", name="bash")],
)
yield FakeChunk(
tool_call_deltas=[FakeToolDelta(index=0, arguments_delta='{"command":"echo hi"}')],
finish_reason="tool_calls",
)
with (
patch.object(
session,
"_create_stream_with_retry",
return_value=stream_content_then_tool(),
),
patch.object(session, "_full_messages", return_value=[]),
# Prevent real tool execution (e.g., bash) during this test.
patch.object(session, "_execute_tools", return_value=([], None)),
):
session.send("test")
# All content should have been emitted
total = "".join(e[1] for e in events if e[0] == "content")
assert total == "Hello world, this is a test message"
# No content events after stream_end
stream_end_idx = next(i for i, e in enumerate(events) if e[0] == "stream_end")
late_content = [e for e in events[stream_end_idx + 1 :] if e[0] == "content"]
assert late_content == [], f"Content after stream_end: {late_content}"
+53
View File
@@ -312,6 +312,59 @@ class TestParseFooter:
# ---------------------------------------------------------------------------
class TestWsEventFinalization:
"""TurnCompleteEvent should finalize streaming messages in the Discord bot."""
def test_turn_complete_finalizes_streaming(self):
"""ContentEvent + TurnCompleteEvent(correlation_id='') finalizes the message."""
from turnstone.channels.discord.bot import TurnstoneBot
from turnstone.mq.protocol import ContentEvent, TurnCompleteEvent
bot = MagicMock(spec=TurnstoneBot)
bot.config = MagicMock()
bot.config.max_message_length = 2000
bot.config.streaming_edit_interval = 1.5
bot.config.auto_approve = False
bot.config.auto_approve_tools = []
bot._streaming = {}
# Use the real _on_ws_event method
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
thread = AsyncMock()
# Feed content event
content_raw = ContentEvent(ws_id="ws-1", text="Hello world").to_json()
_run(bot._on_ws_event("ws-1", thread, content_raw))
# StreamingMessage should exist
assert "ws-1" in bot._streaming
# Feed turn complete with empty correlation_id (server-UI-initiated)
complete_raw = TurnCompleteEvent(ws_id="ws-1", correlation_id="").to_json()
_run(bot._on_ws_event("ws-1", thread, complete_raw))
# StreamingMessage should be removed and finalized
assert "ws-1" not in bot._streaming
def test_turn_complete_no_streaming_is_noop(self):
"""TurnCompleteEvent without prior content should not error."""
from turnstone.channels.discord.bot import TurnstoneBot
from turnstone.mq.protocol import TurnCompleteEvent
bot = MagicMock(spec=TurnstoneBot)
bot._streaming = {}
bot._on_ws_event = TurnstoneBot._on_ws_event.__get__(bot, TurnstoneBot)
thread = AsyncMock()
complete_raw = TurnCompleteEvent(ws_id="ws-1", correlation_id="").to_json()
_run(bot._on_ws_event("ws-1", thread, complete_raw))
# No error, no streaming message
assert "ws-1" not in bot._streaming
class TestChannelCLI:
"""Tests for the channel CLI entry point."""
+66
View File
@@ -1609,3 +1609,69 @@ class TestSSEProxy:
assert b"chunk3" not in body
asyncio.run(_run())
# ---------------------------------------------------------------------------
# Collector — MCP aggregation in get_overview()
# ---------------------------------------------------------------------------
class TestCollectorMCPAggregation:
"""Verify MCP server/resource/prompt aggregation in overview and snapshot."""
def test_overview_mcp_aggregation(self):
"""Two nodes with MCP data produce correct sums in the overview."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
health={"mcp": {"servers": 2, "resources": 5, "prompts": 3}},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
health={"mcp": {"servers": 1, "resources": 4, "prompts": 2}},
)
overview = c.get_overview()
assert overview["mcp_servers"] == 3
assert overview["mcp_resources"] == 9
assert overview["mcp_prompts"] == 5
def test_overview_mcp_absent_when_zero(self):
"""Nodes without MCP data produce no mcp_servers key in the overview."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
health={"status": "ok"},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
health={},
)
overview = c.get_overview()
assert "mcp_servers" not in overview
assert "mcp_resources" not in overview
assert "mcp_prompts" not in overview
def test_overview_mcp_mixed_nodes(self):
"""One node with MCP, one without — only the MCP node contributes."""
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
health={"mcp": {"servers": 3, "resources": 10, "prompts": 7}},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
health={"status": "ok"},
)
overview = c.get_overview()
assert overview["mcp_servers"] == 3
assert overview["mcp_resources"] == 10
assert overview["mcp_prompts"] == 7
+62
View File
@@ -375,6 +375,68 @@ class TestPromptTemplateCRUD:
assert t2["is_default"] is False
assert isinstance(t2["is_default"], bool)
def test_create_with_mcp_origin(self, db):
db.create_prompt_template(
"t1",
"mcp__srv__prompt",
"mcp",
"content",
variables="[]",
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server="srv",
readonly=True,
)
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["origin"] == "mcp"
assert tpl["mcp_server"] == "srv"
assert tpl["readonly"] is True
assert isinstance(tpl["readonly"], bool)
def test_default_origin_values(self, db):
db.create_prompt_template("t1", "basic", "general", "Hello")
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["origin"] == "manual"
assert tpl["mcp_server"] == ""
assert tpl["readonly"] is False
def test_get_prompt_template_by_name(self, db):
db.create_prompt_template("t1", "greeting", "general", "Hello!")
tpl = db.get_prompt_template_by_name("greeting")
assert tpl is not None
assert tpl["template_id"] == "t1"
assert tpl["name"] == "greeting"
def test_get_prompt_template_by_name_nonexistent(self, db):
assert db.get_prompt_template_by_name("nope") is None
def test_list_default_templates(self, db):
db.create_prompt_template("t1", "alpha", "general", "A", is_default=True)
db.create_prompt_template("t2", "beta", "general", "B", is_default=False)
db.create_prompt_template("t3", "gamma", "general", "C", is_default=True)
result = db.list_default_templates()
assert len(result) == 2
assert result[0]["name"] == "alpha"
assert result[1]["name"] == "gamma"
def test_list_default_templates_empty(self, db):
db.create_prompt_template("t1", "alpha", "general", "A", is_default=False)
assert db.list_default_templates() == []
def test_list_prompt_templates_by_origin(self, db):
db.create_prompt_template("t1", "manual_one", "general", "A", origin="manual")
db.create_prompt_template("t2", "mcp_one", "mcp", "B", origin="mcp", mcp_server="srv1")
db.create_prompt_template("t3", "mcp_two", "mcp", "C", origin="mcp", mcp_server="srv2")
result = db.list_prompt_templates_by_origin("mcp")
assert len(result) == 2
names = [r["name"] for r in result]
assert "mcp_one" in names
assert "mcp_two" in names
# ---------------------------------------------------------------------------
# Usage Events
+660 -1
View File
@@ -51,6 +51,83 @@ def _fake_openai_tool(name: str = "mcp__test__search") -> dict[str, Any]:
}
def _fake_mcp_resource(
uri: str = "file:///README.md",
name: str = "readme",
description: str = "Project readme",
mime_type: str = "text/plain",
) -> MagicMock:
"""Create a mock MCP Resource object matching the SDK's Resource type."""
res = MagicMock()
res.uri = uri
res.name = name
res.description = description
res.mimeType = mime_type
return res
def _fake_resource_dict(
uri: str = "file:///README.md",
name: str = "readme",
description: str = "Project readme",
mime_type: str = "text/plain",
server: str = "test",
) -> dict[str, Any]:
"""Create a fake resource dict as stored in per-server state."""
return {
"uri": uri,
"name": name,
"description": description,
"mimeType": mime_type,
"server": server,
}
def _fake_mcp_prompt(
name: str = "code_review",
description: str = "Generate a code review",
arguments: list[dict[str, Any]] | None = None,
) -> MagicMock:
"""Create a mock MCP Prompt object matching the SDK's Prompt type."""
prompt = MagicMock()
prompt.name = name
prompt.description = description
if arguments is None:
arg = MagicMock()
arg.name = "language"
arg.description = "Programming language"
arg.required = True
prompt.arguments = [arg]
else:
mock_args = []
for a in arguments:
arg = MagicMock()
arg.name = a["name"]
arg.description = a.get("description", "")
arg.required = a.get("required", False)
mock_args.append(arg)
prompt.arguments = mock_args
return prompt
def _fake_prompt_dict(
name: str = "mcp__test__code_review",
original_name: str = "code_review",
server: str = "test",
description: str = "Generate a code review",
) -> dict[str, Any]:
"""Create a fake prompt dict as stored in per-server state."""
return {
"name": name,
"original_name": original_name,
"server": server,
"description": description,
"arguments": [
{"name": "language", "description": "Programming language", "required": True}
],
}
# ---------------------------------------------------------------------------
# Schema conversion
# ---------------------------------------------------------------------------
@@ -453,6 +530,23 @@ class TestRebuildTools:
class TestRefreshServer:
@staticmethod
def _add_empty_resource_prompt_mocks(
mgr: MCPClientManager, server_name: str, mock_session: MagicMock
) -> None:
"""Add empty list_resources/list_prompts mocks so _refresh_server works."""
mgr._supports_resources[server_name] = True
mgr._supports_prompts[server_name] = True
empty_res = MagicMock()
empty_res.resources = []
mock_session.list_resources = AsyncMock(return_value=empty_res)
empty_tmpl = MagicMock()
empty_tmpl.resourceTemplates = []
mock_session.list_resource_templates = AsyncMock(return_value=empty_tmpl)
empty_prompts = MagicMock()
empty_prompts.prompts = []
mock_session.list_prompts = AsyncMock(return_value=empty_prompts)
def test_refresh_detects_added_tools(self):
async def _run() -> None:
mgr = MCPClientManager({})
@@ -463,6 +557,7 @@ class TestRefreshServer:
_fake_mcp_tool("create"), # new tool
]
mock_session.list_tools = AsyncMock(return_value=mock_result)
self._add_empty_resource_prompt_mocks(mgr, "github", mock_session)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
@@ -481,6 +576,7 @@ class TestRefreshServer:
mock_result = MagicMock()
mock_result.tools = [] # all tools removed
mock_session.list_tools = AsyncMock(return_value=mock_result)
self._add_empty_resource_prompt_mocks(mgr, "github", mock_session)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
@@ -499,6 +595,7 @@ class TestRefreshServer:
mock_result = MagicMock()
mock_result.tools = [_fake_mcp_tool("search")]
mock_session.list_tools = AsyncMock(return_value=mock_result)
self._add_empty_resource_prompt_mocks(mgr, "github", mock_session)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
@@ -513,7 +610,7 @@ class TestRefreshServer:
async def _run() -> None:
mgr = MCPClientManager({})
with pytest.raises(RuntimeError, match="not connected"):
await mgr._refresh_server("ghost")
await mgr._refresh_server_tools("ghost")
asyncio.run(_run())
@@ -709,3 +806,565 @@ class TestSessionRefresh:
session.handle_command("/mcp refresh")
session.ui.on_error.assert_called_once()
assert "MCP refresh failed" in session.ui.on_error.call_args[0][0]
# ---------------------------------------------------------------------------
# MCP Resources
# ---------------------------------------------------------------------------
class TestMCPResources:
def test_resource_discovery(self):
"""Mock list_resources() returning 2 resources, verify get_resources()."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"fs": [
_fake_resource_dict("file:///a.txt", "a", "File A", "text/plain", "fs"),
_fake_resource_dict("file:///b.txt", "b", "File B", "text/plain", "fs"),
],
}
mgr._rebuild_resources()
resources = mgr.get_resources()
assert len(resources) == 2
uris = {r["uri"] for r in resources}
assert uris == {"file:///a.txt", "file:///b.txt"}
assert all(r["server"] == "fs" for r in resources)
def test_rebuild_resources_copy_on_write(self):
"""Verify mutation safety — get_resources() returns independent copy."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"a": [_fake_resource_dict("file:///x", "x", "", "", "a")],
}
mgr._rebuild_resources()
old_resources = mgr._resources
old_map = mgr._resource_map
mgr._per_server_resources["b"] = [_fake_resource_dict("file:///y", "y", "", "", "b")]
mgr._rebuild_resources()
assert mgr._resources is not old_resources
assert mgr._resource_map is not old_map
def test_get_resources_returns_copy(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {
"a": [_fake_resource_dict("file:///x", "x", "", "", "a")],
}
mgr._rebuild_resources()
resources = mgr.get_resources()
assert len(resources) == 1
resources.clear()
assert len(mgr.get_resources()) == 1
def test_read_resource_sync(self):
"""Mock session.read_resource(), verify text extraction."""
mgr = MCPClientManager({})
mgr._resource_map = {"file:///readme": ("fs", "file:///readme")}
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._loop = asyncio.new_event_loop()
# Mock the read_resource result
text_content = MagicMock(spec=["text"])
text_content.text = "Hello, world!"
mock_result = MagicMock()
mock_result.contents = [text_content]
mock_session.read_resource = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
output = mgr.read_resource_sync("file:///readme", timeout=5)
assert output == "Hello, world!"
mock_session.read_resource.assert_awaited_once_with("file:///readme")
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_read_resource_sync_blob(self):
"""Verify base64 blob extraction."""
mgr = MCPClientManager({})
mgr._resource_map = {"file:///img.png": ("fs", "file:///img.png")}
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._loop = asyncio.new_event_loop()
blob_content = MagicMock(spec=["blob"])
blob_content.blob = "aGVsbG8="
mock_result = MagicMock()
mock_result.contents = [blob_content]
mock_session.read_resource = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
output = mgr.read_resource_sync("file:///img.png", timeout=5)
assert output == "aGVsbG8="
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_read_resource_sync_unknown_uri(self):
mgr = MCPClientManager({})
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("file:///nonexistent")
def test_read_resource_sync_disconnected(self):
mgr = MCPClientManager({})
mgr._resource_map = {"file:///x": ("dead", "file:///x")}
with pytest.raises(RuntimeError, match="not connected"):
mgr.read_resource_sync("file:///x")
def test_read_resource_sync_timeout(self):
"""Verify timeout handling."""
mgr = MCPClientManager({})
mgr._resource_map = {"file:///x": ("fs", "file:///x")}
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._loop = asyncio.new_event_loop()
async def _slow_read(_uri: str) -> None:
await asyncio.sleep(10)
mock_session.read_resource = _slow_read
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
with pytest.raises(TimeoutError):
mgr.read_resource_sync("file:///x", timeout=1)
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_resource_listener_notification(self):
"""Verify callback fires on rebuild."""
mgr = MCPClientManager({})
calls: list[int] = []
mgr.add_resource_listener(lambda: calls.append(1))
mgr._per_server_resources = {"a": [_fake_resource_dict()]}
mgr._rebuild_resources()
assert len(calls) == 1
def test_resource_listener_remove(self):
mgr = MCPClientManager({})
calls: list[int] = []
cb = lambda: calls.append(1) # noqa: E731
mgr.add_resource_listener(cb)
mgr.remove_resource_listener(cb)
mgr._rebuild_resources()
assert calls == []
def test_resource_listener_error_does_not_propagate(self):
mgr = MCPClientManager({})
mgr.add_resource_listener(lambda: 1 / 0)
mgr._rebuild_resources() # should not raise
def test_resource_refresh_on_notification(self):
"""Mock notification, verify re-fetch of resources."""
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mgr._sessions["fs"] = mock_session
mgr._supports_resources["fs"] = True
# Initial state
mgr._per_server_resources["fs"] = [
_fake_resource_dict("file:///old", server="fs"),
]
mgr._rebuild_resources()
assert len(mgr.get_resources()) == 1
# Mock the re-fetch returning a new resource
new_res = _fake_mcp_resource("file:///new", "new")
mock_res_result = MagicMock()
mock_res_result.resources = [new_res]
mock_session.list_resources = AsyncMock(return_value=mock_res_result)
mock_tmpl_result = MagicMock()
mock_tmpl_result.resourceTemplates = []
mock_session.list_resource_templates = AsyncMock(return_value=mock_tmpl_result)
await mgr._refresh_server_resources("fs")
resources = mgr.get_resources()
assert len(resources) == 1
assert resources[0]["uri"] == "file:///new"
asyncio.run(_run())
def test_rebuild_resources_empty(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {}
mgr._rebuild_resources()
assert mgr._resources == []
assert mgr._resource_map == {}
def test_rebuild_resources_multi_server(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {
"fs": [_fake_resource_dict("file:///a", server="fs")],
"db": [_fake_resource_dict("db://table", name="table", server="db")],
}
mgr._rebuild_resources()
assert len(mgr._resources) == 2
assert mgr._resource_map["file:///a"] == ("fs", "file:///a")
assert mgr._resource_map["db://table"] == ("db", "db://table")
def test_template_prefix_matching(self):
"""Expanded URI matches template by prefix."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"db": [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "A row",
"mimeType": "application/json",
"server": "db",
"template": True,
},
],
}
mgr._rebuild_resources()
# Template should not be in resource_map
assert "db://tables/{table}/rows/{id}" not in mgr._resource_map
# But prefix matching should find it
result = mgr._match_template("db://tables/users/rows/1")
assert result is not None
server, template_uri = result
assert server == "db"
assert template_uri == "db://tables/{table}/rows/{id}"
def test_template_longest_prefix_wins(self):
"""When two templates have overlapping prefixes, the longer one wins."""
mgr = MCPClientManager({})
# Use templates with genuinely different prefix lengths:
# "db://data/" (6 chars after scheme) vs "db://data/tables/" (13 chars after scheme)
mgr._per_server_resources = {
"short": [
{
"uri": "db://data/{collection}",
"name": "collection",
"description": "",
"mimeType": "",
"server": "short",
"template": True,
},
],
"long": [
{
"uri": "db://data/tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "long",
"template": True,
},
],
}
mgr._rebuild_resources()
# "db://data/tables/users" matches both prefixes ("db://data/" and
# "db://data/tables/") — the longer one should win
result = mgr._match_template("db://data/tables/users")
assert result is not None
server, template_uri = result
assert server == "long"
assert template_uri == "db://data/tables/{table}"
# URI that only matches the short prefix
result2 = mgr._match_template("db://data/views/active")
assert result2 is not None
assert result2[0] == "short"
def test_template_no_match_raises(self):
"""Completely unrelated URI still raises ValueError."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"db": [
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "db",
"template": True,
},
],
}
mgr._rebuild_resources()
assert mgr._match_template("file:///something") is None
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("file:///something")
def test_read_resource_sync_with_template_uri(self):
"""End-to-end: template discovered, expanded URI dispatched to correct server."""
mgr = MCPClientManager({})
mgr._per_server_resources = {
"db": [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "A row",
"mimeType": "application/json",
"server": "db",
"template": True,
},
],
}
mgr._rebuild_resources()
mock_session = MagicMock()
mgr._sessions["db"] = mock_session
mgr._loop = asyncio.new_event_loop()
text_content = MagicMock(spec=["text"])
text_content.text = '{"name": "Alice"}'
mock_result = MagicMock()
mock_result.contents = [text_content]
mock_session.read_resource = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
output = mgr.read_resource_sync("db://tables/users/rows/1", timeout=5)
assert output == '{"name": "Alice"}'
mock_session.read_resource.assert_awaited_once_with("db://tables/users/rows/1")
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
# ---------------------------------------------------------------------------
# MCP Prompts
# ---------------------------------------------------------------------------
class TestMCPPrompts:
def test_prompt_discovery(self):
"""Mock list_prompts(), verify get_prompts() with correct prefixed names."""
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"tmpl": [
_fake_prompt_dict("mcp__tmpl__code_review", "code_review", "tmpl"),
_fake_prompt_dict("mcp__tmpl__summarize", "summarize", "tmpl"),
],
}
mgr._rebuild_prompts()
prompts = mgr.get_prompts()
assert len(prompts) == 2
names = {p["name"] for p in prompts}
assert names == {"mcp__tmpl__code_review", "mcp__tmpl__summarize"}
# Verify map entries
assert mgr._prompt_map["mcp__tmpl__code_review"] == ("tmpl", "code_review")
assert mgr._prompt_map["mcp__tmpl__summarize"] == ("tmpl", "summarize")
def test_rebuild_prompts_copy_on_write(self):
"""Verify mutation safety."""
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"a": [_fake_prompt_dict("mcp__a__p1", "p1", "a")],
}
mgr._rebuild_prompts()
old_prompts = mgr._prompts
old_map = mgr._prompt_map
mgr._per_server_prompts["b"] = [_fake_prompt_dict("mcp__b__p2", "p2", "b")]
mgr._rebuild_prompts()
assert mgr._prompts is not old_prompts
assert mgr._prompt_map is not old_map
def test_get_prompts_returns_copy(self):
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"a": [_fake_prompt_dict("mcp__a__p1", "p1", "a")],
}
mgr._rebuild_prompts()
prompts = mgr.get_prompts()
assert len(prompts) == 1
prompts.clear()
assert len(mgr.get_prompts()) == 1
def test_get_prompt_sync(self):
"""Mock session.get_prompt(), verify message conversion."""
mgr = MCPClientManager({})
mgr._prompt_map = {"mcp__tmpl__review": ("tmpl", "review")}
mock_session = MagicMock()
mgr._sessions["tmpl"] = mock_session
mgr._loop = asyncio.new_event_loop()
# Build mock PromptMessage
msg1 = MagicMock()
msg1.role = "user"
msg1.content = MagicMock()
msg1.content.text = "Review this code"
msg2 = MagicMock()
msg2.role = "assistant"
msg2.content = MagicMock()
msg2.content.text = "Looks good!"
mock_result = MagicMock()
mock_result.messages = [msg1, msg2]
mock_session.get_prompt = AsyncMock(return_value=mock_result)
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
messages = mgr.get_prompt_sync(
"mcp__tmpl__review", arguments={"language": "python"}, timeout=5
)
assert len(messages) == 2
assert messages[0] == {"role": "user", "content": "Review this code"}
assert messages[1] == {"role": "assistant", "content": "Looks good!"}
mock_session.get_prompt.assert_awaited_once_with(
"review", arguments={"language": "python"}
)
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_get_prompt_sync_unknown(self):
mgr = MCPClientManager({})
with pytest.raises(ValueError, match="Unknown MCP prompt"):
mgr.get_prompt_sync("mcp__no__such")
def test_get_prompt_sync_disconnected(self):
mgr = MCPClientManager({})
mgr._prompt_map = {"mcp__dead__p": ("dead", "p")}
with pytest.raises(RuntimeError, match="not connected"):
mgr.get_prompt_sync("mcp__dead__p")
def test_get_prompt_sync_timeout(self):
"""Verify timeout handling."""
mgr = MCPClientManager({})
mgr._prompt_map = {"mcp__tmpl__slow": ("tmpl", "slow")}
mock_session = MagicMock()
mgr._sessions["tmpl"] = mock_session
mgr._loop = asyncio.new_event_loop()
async def _slow_prompt(_name: str, *, arguments: dict[str, str] | None = None) -> None:
await asyncio.sleep(10)
mock_session.get_prompt = _slow_prompt
thread = None
try:
thread = __import__("threading").Thread(target=mgr._loop.run_forever, daemon=True)
thread.start()
with pytest.raises(TimeoutError):
mgr.get_prompt_sync("mcp__tmpl__slow", timeout=1)
finally:
mgr._loop.call_soon_threadsafe(mgr._loop.stop)
if thread:
thread.join(timeout=5)
mgr._loop.close()
def test_prompt_listener_notification(self):
"""Verify callback fires on rebuild."""
mgr = MCPClientManager({})
calls: list[int] = []
mgr.add_prompt_listener(lambda: calls.append(1))
mgr._per_server_prompts = {"a": [_fake_prompt_dict()]}
mgr._rebuild_prompts()
assert len(calls) == 1
def test_prompt_listener_remove(self):
mgr = MCPClientManager({})
calls: list[int] = []
cb = lambda: calls.append(1) # noqa: E731
mgr.add_prompt_listener(cb)
mgr.remove_prompt_listener(cb)
mgr._rebuild_prompts()
assert calls == []
def test_prompt_listener_error_does_not_propagate(self):
mgr = MCPClientManager({})
mgr.add_prompt_listener(lambda: 1 / 0)
mgr._rebuild_prompts() # should not raise
def test_is_mcp_prompt(self):
"""Verify name lookup."""
mgr = MCPClientManager({})
mgr._prompt_map["mcp__tmpl__review"] = ("tmpl", "review")
assert mgr.is_mcp_prompt("mcp__tmpl__review") is True
assert mgr.is_mcp_prompt("nonexistent") is False
def test_prompt_refresh_on_notification(self):
"""Mock notification, verify re-fetch of prompts."""
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mgr._sessions["tmpl"] = mock_session
mgr._supports_prompts["tmpl"] = True
# Initial state
mgr._per_server_prompts["tmpl"] = [
_fake_prompt_dict("mcp__tmpl__old", "old", "tmpl"),
]
mgr._rebuild_prompts()
assert len(mgr.get_prompts()) == 1
# Mock re-fetch returning a new prompt
new_prompt = _fake_mcp_prompt("new_prompt", "A new prompt")
mock_prompt_result = MagicMock()
mock_prompt_result.prompts = [new_prompt]
mock_session.list_prompts = AsyncMock(return_value=mock_prompt_result)
await mgr._refresh_server_prompts("tmpl")
prompts = mgr.get_prompts()
assert len(prompts) == 1
assert prompts[0]["name"] == "mcp__tmpl__new_prompt"
assert prompts[0]["original_name"] == "new_prompt"
asyncio.run(_run())
def test_rebuild_prompts_empty(self):
mgr = MCPClientManager({})
mgr._per_server_prompts = {}
mgr._rebuild_prompts()
assert mgr._prompts == []
assert mgr._prompt_map == {}
def test_rebuild_prompts_multi_server(self):
mgr = MCPClientManager({})
mgr._per_server_prompts = {
"a": [_fake_prompt_dict("mcp__a__p1", "p1", "a")],
"b": [_fake_prompt_dict("mcp__b__p2", "p2", "b")],
}
mgr._rebuild_prompts()
assert len(mgr._prompts) == 2
assert mgr._prompt_map["mcp__a__p1"] == ("a", "p1")
assert mgr._prompt_map["mcp__b__p2"] == ("b", "p2")
# ---------------------------------------------------------------------------
# Shutdown cleans up new state
# ---------------------------------------------------------------------------
class TestShutdownCleanup:
def test_shutdown_clears_resources_and_prompts(self):
mgr = MCPClientManager({})
mgr._per_server_resources = {"a": [_fake_resource_dict()]}
mgr._rebuild_resources()
mgr._per_server_prompts = {"a": [_fake_prompt_dict()]}
mgr._rebuild_prompts()
assert mgr.get_resources() != []
assert mgr.get_prompts() != []
mgr.shutdown()
assert mgr.get_resources() == []
assert mgr.get_prompts() == []
assert mgr._resource_map == {}
assert mgr._prompt_map == {}
+397
View File
@@ -0,0 +1,397 @@
"""Integration tests for MCPClientManager data flow.
Uses real storage (SQLite) and real MCPClientManager state manipulation,
but mock MCP sessions instead of wire-protocol connections. This validates
the full data pipeline: per-server data -> rebuild -> merged state ->
query methods -> storage sync -> shutdown cleanup.
"""
from __future__ import annotations
import asyncio
from typing import Any
from unittest.mock import AsyncMock, MagicMock
import pytest
from turnstone.core.mcp_client import MCPClientManager
from turnstone.core.storage._sqlite import SQLiteBackend
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_resource(
uri: str, name: str, server: str, description: str = "", mime: str = "text/plain"
) -> dict[str, Any]:
return {
"uri": uri,
"name": name,
"description": description,
"mimeType": mime,
"server": server,
}
def _make_prompt(
prefixed_name: str,
original_name: str,
server: str,
description: str = "",
arguments: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
return {
"name": prefixed_name,
"original_name": original_name,
"server": server,
"description": description,
"arguments": arguments or [],
}
def _make_mock_session(
read_resource_result: Any = None,
get_prompt_result: Any = None,
) -> AsyncMock:
"""Build a mock ClientSession with configurable async return values."""
session = AsyncMock()
if read_resource_result is not None:
session.read_resource.return_value = read_resource_result
else:
# Default: single text content
content_item = MagicMock()
content_item.text = "resource content"
result = MagicMock()
result.contents = [content_item]
session.read_resource.return_value = result
if get_prompt_result is not None:
session.get_prompt.return_value = get_prompt_result
else:
msg = MagicMock()
msg.role = "user"
msg.content = MagicMock()
msg.content.text = "Hello, World!"
result = MagicMock()
result.messages = [msg]
session.get_prompt.return_value = result
return session
# ---------------------------------------------------------------------------
# Integration test class
# ---------------------------------------------------------------------------
class TestFullLifecycleResourcesPrompts:
"""Integration test exercising real code paths with real SQLite storage
but mock MCP sessions.
Validates the complete data flow: per-server data population, rebuild
merging, query methods, resource/prompt dispatch through asyncio, storage
sync, and shutdown cleanup.
"""
@pytest.fixture()
def mgr(self) -> MCPClientManager:
"""Create an MCPClientManager with no server configs (no start())."""
return MCPClientManager({})
@pytest.fixture()
def db(self, tmp_path) -> SQLiteBackend:
"""Create a fresh SQLite backend for each test."""
backend = SQLiteBackend(str(tmp_path / "test.db"))
yield backend
backend.close()
def test_rebuild_resources_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_resources merges per-server resources into a unified list."""
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
_make_resource("file:///b.txt", "b", "alpha"),
]
mgr._per_server_resources["beta"] = [
_make_resource("file:///c.txt", "c", "beta"),
]
mgr._rebuild_resources()
resources = mgr.get_resources()
assert len(resources) == 3
uris = {r["uri"] for r in resources}
assert uris == {"file:///a.txt", "file:///b.txt", "file:///c.txt"}
# resource_map should have entries for all non-template resources
assert "file:///a.txt" in mgr._resource_map
assert "file:///c.txt" in mgr._resource_map
assert mgr.resource_count == 3
def test_rebuild_prompts_produces_merged_state(self, mgr: MCPClientManager) -> None:
"""_rebuild_prompts merges per-server prompts into a unified list."""
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._per_server_prompts["beta"] = [
_make_prompt("mcp__beta__summarize", "summarize", "beta", "Summarize text"),
_make_prompt("mcp__beta__translate", "translate", "beta", "Translate text"),
]
mgr._rebuild_prompts()
prompts = mgr.get_prompts()
assert len(prompts) == 3
names = {p["name"] for p in prompts}
assert names == {"mcp__alpha__greet", "mcp__beta__summarize", "mcp__beta__translate"}
# prompt_map should map prefixed -> (server, original)
assert mgr._prompt_map["mcp__alpha__greet"] == ("alpha", "greet")
assert mgr._prompt_map["mcp__beta__summarize"] == ("beta", "summarize")
assert mgr.prompt_count == 3
assert mgr.is_mcp_prompt("mcp__alpha__greet") is True
assert mgr.is_mcp_prompt("nonexistent") is False
def test_read_resource_sync_dispatches_correctly(self, mgr: MCPClientManager) -> None:
"""read_resource_sync dispatches to the correct session via a real asyncio loop."""
# Set up a real event loop in a thread (simulating start())
loop = asyncio.new_event_loop()
import threading
thread = threading.Thread(target=loop.run_forever, daemon=True)
thread.start()
mgr._loop = loop
try:
# Populate session and resource map
session = _make_mock_session()
mgr._sessions["alpha"] = session
mgr._per_server_resources["alpha"] = [
_make_resource("file:///readme.md", "readme", "alpha"),
]
mgr._rebuild_resources()
result = mgr.read_resource_sync("file:///readme.md", timeout=5)
assert result == "resource content"
session.read_resource.assert_awaited_once_with("file:///readme.md")
finally:
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
loop.close()
def test_read_resource_sync_unknown_uri_raises(self, mgr: MCPClientManager) -> None:
"""read_resource_sync raises ValueError for an unknown URI."""
with pytest.raises(ValueError, match="Unknown MCP resource"):
mgr.read_resource_sync("file:///nonexistent")
def test_read_resource_via_template(self, mgr: MCPClientManager) -> None:
"""Expanded template URI dispatched to correct server via real asyncio loop."""
loop = asyncio.new_event_loop()
import threading
thread = threading.Thread(target=loop.run_forever, daemon=True)
thread.start()
mgr._loop = loop
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
# Register a template resource (no concrete resources)
mgr._per_server_resources["alpha"] = [
{
"uri": "db://tables/{table}/rows/{id}",
"name": "row",
"description": "Fetch a row",
"mimeType": "application/json",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
# Template should not be in _resource_map
assert "db://tables/{table}/rows/{id}" not in mgr._resource_map
# But expanded URI should resolve via prefix matching
result = mgr.read_resource_sync("db://tables/users/rows/42", timeout=5)
assert result == "resource content"
session.read_resource.assert_awaited_once_with("db://tables/users/rows/42")
finally:
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
loop.close()
def test_get_prompt_sync_dispatches_correctly(self, mgr: MCPClientManager) -> None:
"""get_prompt_sync dispatches to the correct session via a real asyncio loop."""
loop = asyncio.new_event_loop()
import threading
thread = threading.Thread(target=loop.run_forever, daemon=True)
thread.start()
mgr._loop = loop
try:
session = _make_mock_session()
mgr._sessions["alpha"] = session
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha", "Say hello"),
]
mgr._rebuild_prompts()
messages = mgr.get_prompt_sync(
"mcp__alpha__greet", arguments={"name": "World"}, timeout=5
)
assert len(messages) == 1
assert messages[0]["role"] == "user"
assert messages[0]["content"] == "Hello, World!"
session.get_prompt.assert_awaited_once_with("greet", arguments={"name": "World"})
finally:
loop.call_soon_threadsafe(loop.stop)
thread.join(timeout=5)
loop.close()
def test_get_prompt_sync_unknown_name_raises(self, mgr: MCPClientManager) -> None:
"""get_prompt_sync raises ValueError for an unknown prompt name."""
with pytest.raises(ValueError, match="Unknown MCP prompt"):
mgr.get_prompt_sync("mcp__nosrv__nope")
def test_sync_prompts_to_storage_creates_templates(
self, mgr: MCPClientManager, db: SQLiteBackend
) -> None:
"""sync_prompts_to_storage creates governance templates in real SQLite."""
mgr.set_storage(db)
mgr._prompts = [
_make_prompt(
"mcp__alpha__greet",
"greet",
"alpha",
"Say hello",
[{"name": "user", "description": "Who to greet", "required": True}],
),
_make_prompt(
"mcp__beta__summarize",
"summarize",
"beta",
"Summarize text",
),
]
# Mark connected so set_storage triggers sync
mgr._connected.set()
# Re-set storage to trigger auto-sync
mgr.set_storage(db)
templates = db.list_prompt_templates()
assert len(templates) == 2
names = {t["name"] for t in templates}
assert names == {"mcp__alpha__greet", "mcp__beta__summarize"}
# Verify details on first template
tpl = db.get_prompt_template_by_name("mcp__alpha__greet")
assert tpl is not None
assert tpl["origin"] == "mcp"
assert tpl["mcp_server"] == "alpha"
assert tpl["readonly"] is True
assert tpl["category"] == "mcp"
assert "user" in tpl["variables"]
def test_sync_prompts_removes_stale_templates(
self, mgr: MCPClientManager, db: SQLiteBackend
) -> None:
"""sync_prompts_to_storage removes templates whose MCP prompts are gone."""
mgr.set_storage(db)
# Create an initial template via sync
mgr._prompts = [
_make_prompt("mcp__alpha__old", "old", "alpha", "Old prompt"),
]
mgr.sync_prompts_to_storage()
assert len(db.list_prompt_templates()) == 1
# Now the prompt is gone
mgr._prompts = []
result = mgr.sync_prompts_to_storage()
assert result["removed"] == ["mcp__alpha__old"]
assert len(db.list_prompt_templates()) == 0
def test_shutdown_clears_all_state(self, mgr: MCPClientManager) -> None:
"""shutdown() clears sessions, tools, resources, prompts, and listeners."""
# Populate state
mgr._sessions["alpha"] = MagicMock()
mgr._per_server_tools["alpha"] = [
{
"type": "function",
"function": {
"name": "mcp__alpha__search",
"description": "Search",
"parameters": {},
},
}
]
mgr._rebuild_tools()
mgr._per_server_resources["alpha"] = [
_make_resource("file:///a.txt", "a", "alpha"),
{
"uri": "db://tables/{table}",
"name": "table",
"description": "",
"mimeType": "",
"server": "alpha",
"template": True,
},
]
mgr._rebuild_resources()
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__greet", "greet", "alpha"),
]
mgr._rebuild_prompts()
mgr._listeners.append(lambda: None)
mgr._resource_listeners.append(lambda: None)
mgr._prompt_listeners.append(lambda: None)
# Verify populated
assert len(mgr._sessions) == 1
assert len(mgr._tools) == 1
assert len(mgr._resources) == 2 # 1 concrete + 1 template
assert len(mgr._template_prefixes) == 1
assert len(mgr._prompts) == 1
mgr.shutdown()
assert len(mgr._sessions) == 0
assert len(mgr._tools) == 0
assert len(mgr._tool_map) == 0
assert len(mgr._resources) == 0
assert len(mgr._resource_map) == 0
assert len(mgr._template_prefixes) == 0
assert len(mgr._prompts) == 0
assert len(mgr._prompt_map) == 0
assert len(mgr._listeners) == 0
assert len(mgr._resource_listeners) == 0
assert len(mgr._prompt_listeners) == 0
def test_listener_notifications_fire_on_rebuild(self, mgr: MCPClientManager) -> None:
"""Rebuild methods fire the appropriate listener callbacks."""
tool_fired = []
resource_fired = []
prompt_fired = []
mgr.add_listener(lambda: tool_fired.append(1))
mgr.add_resource_listener(lambda: resource_fired.append(1))
mgr.add_prompt_listener(lambda: prompt_fired.append(1))
mgr._per_server_tools["alpha"] = []
mgr._rebuild_tools()
assert len(tool_fired) == 1
mgr._per_server_resources["alpha"] = [
_make_resource("file:///x.txt", "x", "alpha"),
]
mgr._rebuild_resources()
assert len(resource_fired) == 1
mgr._per_server_prompts["alpha"] = [
_make_prompt("mcp__alpha__p1", "p1", "alpha"),
]
mgr._rebuild_prompts()
assert len(prompt_fired) == 1
# Tool and resource listeners should not have been fired again
assert len(tool_fired) == 1
assert len(resource_fired) == 1
+272
View File
@@ -0,0 +1,272 @@
"""Tests for MCP prompt → governance template sync and readonly API guards."""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from turnstone.core.mcp_client import MCPClientManager
@pytest.fixture()
def mgr() -> MCPClientManager:
"""Create an MCPClientManager with no real servers (no start())."""
return MCPClientManager({})
def _make_storage() -> MagicMock:
"""Create a mock storage backend with prompt template methods."""
storage = MagicMock()
storage.get_prompt_template_by_name.return_value = None
storage.list_prompt_templates_by_origin.return_value = []
storage.create_prompt_template.return_value = None
storage.update_prompt_template.return_value = True
storage.delete_prompt_template.return_value = True
return storage
class TestSyncPromptsToStorage:
def test_sync_no_storage(self, mgr: MCPClientManager) -> None:
"""Without storage set, sync returns empty stats."""
result = mgr.sync_prompts_to_storage()
assert result == {"added": [], "removed": [], "skipped": []}
def test_sync_creates_mcp_templates(self, mgr: MCPClientManager) -> None:
"""New MCP prompts are created as templates."""
storage = _make_storage()
mgr.set_storage(storage)
# Populate internal prompts list directly
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Say hello",
"arguments": [
{"name": "name", "description": "Who to greet", "required": True},
],
},
]
result = mgr.sync_prompts_to_storage()
assert result["added"] == ["mcp__test__greeting"]
assert result["removed"] == []
assert result["skipped"] == []
storage.create_prompt_template.assert_called_once()
call_kwargs = storage.create_prompt_template.call_args
assert call_kwargs[1]["name"] == "mcp__test__greeting"
assert call_kwargs[1]["origin"] == "mcp"
assert call_kwargs[1]["mcp_server"] == "test"
assert call_kwargs[1]["readonly"] is True
assert call_kwargs[1]["category"] == "mcp"
assert '"name"' in call_kwargs[1]["variables"]
def test_sync_skips_manual_overrides(self, mgr: MCPClientManager) -> None:
"""A manual template with the same name is not overwritten."""
storage = _make_storage()
storage.get_prompt_template_by_name.return_value = {
"template_id": "existing-id",
"name": "mcp__test__greeting",
"origin": "manual",
"readonly": False,
}
mgr.set_storage(storage)
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Say hello",
"arguments": [],
},
]
result = mgr.sync_prompts_to_storage()
assert result["skipped"] == ["mcp__test__greeting"]
assert result["added"] == []
storage.create_prompt_template.assert_not_called()
storage.update_prompt_template.assert_not_called()
def test_sync_updates_existing_mcp_template(self, mgr: MCPClientManager) -> None:
"""An existing MCP template gets its content/variables updated."""
storage = _make_storage()
storage.get_prompt_template_by_name.return_value = {
"template_id": "existing-id",
"name": "mcp__test__greeting",
"origin": "mcp",
"mcp_server": "test",
"readonly": True,
}
mgr.set_storage(storage)
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Updated description",
"arguments": [
{"name": "user", "description": "The user", "required": False},
],
},
]
result = mgr.sync_prompts_to_storage()
assert result["added"] == []
assert result["skipped"] == []
storage.create_prompt_template.assert_not_called()
storage.update_prompt_template.assert_called_once()
call_args = storage.update_prompt_template.call_args
assert call_args[0][0] == "existing-id"
assert "Updated description" in call_args[1]["content"]
assert "user" in call_args[1]["variables"]
# Security: is_default must be reset to prevent compromised MCP server
# from injecting content into a previously admin-promoted default
assert call_args[1]["is_default"] is False
def test_sync_resets_is_default_on_promoted_template(self, mgr: MCPClientManager) -> None:
"""An MCP template promoted to default by admin gets is_default reset on sync."""
storage = _make_storage()
storage.get_prompt_template_by_name.return_value = {
"template_id": "promoted-id",
"name": "mcp__test__greeting",
"origin": "mcp",
"mcp_server": "test",
"readonly": True,
"is_default": True, # admin toggled this
}
mgr.set_storage(storage)
mgr._prompts = [
{
"name": "mcp__test__greeting",
"original_name": "greeting",
"server": "test",
"description": "Potentially compromised content",
"arguments": [],
},
]
mgr.sync_prompts_to_storage()
call_args = storage.update_prompt_template.call_args
assert call_args[1]["is_default"] is False
def test_sync_removes_deleted_prompts(self, mgr: MCPClientManager) -> None:
"""MCP templates in storage with no matching prompt are deleted."""
storage = _make_storage()
storage.list_prompt_templates_by_origin.return_value = [
{
"template_id": "old-id",
"name": "mcp__test__old_prompt",
"origin": "mcp",
"mcp_server": "test",
},
]
mgr.set_storage(storage)
mgr._prompts = [] # No prompts at all
result = mgr.sync_prompts_to_storage()
assert result["removed"] == ["mcp__test__old_prompt"]
storage.delete_prompt_template.assert_called_once_with("old-id")
class TestSetStorageAutoSync:
"""set_storage() triggers an immediate sync when servers are already connected."""
def test_set_storage_syncs_when_connected(self, mgr) -> None:
storage = _make_storage()
mgr._prompts = [
{
"name": "mcp__srv__p1",
"original_name": "p1",
"server": "srv",
"description": "A prompt",
"arguments": [],
}
]
mgr._connected.set()
mgr.set_storage(storage)
# Should have called create_prompt_template for the discovered prompt
storage.create_prompt_template.assert_called_once()
call_kwargs = storage.create_prompt_template.call_args
assert call_kwargs[1]["name"] == "mcp__srv__p1"
assert call_kwargs[1]["origin"] == "mcp"
def test_set_storage_no_sync_when_not_connected(self, mgr) -> None:
storage = _make_storage()
mgr._prompts = [
{
"name": "mcp__srv__p1",
"original_name": "p1",
"server": "srv",
"description": "A prompt",
"arguments": [],
}
]
# _connected is NOT set
mgr.set_storage(storage)
# Should not have synced
storage.create_prompt_template.assert_not_called()
class TestReadonlyAPIGuards:
"""Test that the console server API guards reject edits to readonly templates."""
@pytest.fixture()
def db(self, tmp_path):
"""Create a fresh SQLite backend for each test."""
from turnstone.core.storage._sqlite import SQLiteBackend
return SQLiteBackend(str(tmp_path / "test.db"))
def test_readonly_guard_update(self, db) -> None:
"""Readonly templates cannot be updated via storage guard logic."""
db.create_prompt_template(
"t1",
"mcp__srv__prompt",
"mcp",
"content",
variables="[]",
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server="srv",
readonly=True,
)
tpl = db.get_prompt_template("t1")
assert tpl is not None
assert tpl["readonly"] is True
# Simulate API guard check
assert tpl.get("readonly") is True
def test_readonly_guard_delete(self, db) -> None:
"""Readonly templates are flagged for API-level rejection."""
db.create_prompt_template(
"t1",
"mcp__srv__prompt",
"mcp",
"content",
variables="[]",
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server="srv",
readonly=True,
)
existing = db.get_prompt_template("t1")
assert existing is not None
assert existing.get("readonly") is True
+393
View File
@@ -0,0 +1,393 @@
"""Tests for prompt template runtime wiring into ChatSession."""
from __future__ import annotations
from unittest.mock import MagicMock
from turnstone.core.session import ChatSession, _render_template
class NullUI:
"""UI adapter that discards all output."""
def on_thinking_start(self):
pass
def on_thinking_stop(self):
pass
def on_reasoning_token(self, text):
pass
def on_content_token(self, text):
pass
def on_stream_end(self):
pass
def approve_tools(self, items):
return True, None
def on_tool_result(self, call_id, name, output):
pass
def on_tool_output_chunk(self, call_id, chunk):
pass
def on_status(self, usage, context_window, effort):
pass
def on_plan_review(self, content):
return ""
def on_info(self, message):
pass
def on_error(self, message):
pass
def on_state_change(self, state):
pass
def on_rename(self, name):
pass
def _make_session(**kwargs):
defaults = dict(
client=MagicMock(),
model="test-model",
ui=NullUI(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
defaults.update(kwargs)
return ChatSession(**defaults)
def _sys_content(session: ChatSession) -> str:
"""Extract the system message content."""
msgs = [m for m in session.system_messages if m["role"] == "system"]
assert msgs
return msgs[0]["content"]
def _create_template(db, template_id, name, content, is_default=False, **kwargs):
"""Helper to create a prompt template in storage."""
db.create_prompt_template(
template_id=template_id,
name=name,
category=kwargs.get("category", "general"),
content=content,
variables=kwargs.get("variables", "[]"),
is_default=is_default,
org_id=kwargs.get("org_id", ""),
created_by=kwargs.get("created_by", "test"),
origin=kwargs.get("origin", "manual"),
mcp_server=kwargs.get("mcp_server", ""),
readonly=kwargs.get("readonly", False),
)
# ---------------------------------------------------------------------------
# _render_template unit tests
# ---------------------------------------------------------------------------
class TestRenderTemplate:
def test_basic_substitution(self):
result = _render_template("Hello {{name}}", {"name": "world"})
assert result == "Hello world"
def test_multiple_variables(self):
result = _render_template(
"Model: {{model}}, WS: {{ws_id}}", {"model": "gpt-5", "ws_id": "abc123"}
)
assert result == "Model: gpt-5, WS: abc123"
def test_unresolvable_variable_kept(self):
result = _render_template("Hello {{unknown}}", {"model": "gpt-5"})
assert result == "Hello {{unknown}}"
def test_empty_context(self):
result = _render_template("No vars here", {})
assert result == "No vars here"
def test_duplicate_placeholder(self):
result = _render_template("{{x}} and {{x}}", {"x": "val"})
assert result == "val and val"
def test_no_cross_variable_injection(self):
# If model contains {{ws_id}}, it must NOT be expanded
result = _render_template("Model: {{model}}", {"model": "{{ws_id}}", "ws_id": "secret"})
assert result == "Model: {{ws_id}}"
assert "secret" not in result
# ---------------------------------------------------------------------------
# Default templates in system message
# ---------------------------------------------------------------------------
class TestDefaultTemplates:
def test_default_templates_in_system_message(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "alpha", "You are a helpful assistant.", is_default=True)
_create_template(db, "t2", "beta", "Always be concise.", is_default=True)
session = _make_session()
content = _sys_content(session)
assert "You are a helpful assistant." in content
assert "Always be concise." in content
def test_default_templates_ordered_by_name(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t2", "b-template", "SECOND", is_default=True)
_create_template(db, "t1", "a-template", "FIRST", is_default=True)
session = _make_session()
content = _sys_content(session)
first_pos = content.index("FIRST")
second_pos = content.index("SECOND")
assert first_pos < second_pos
def test_no_default_templates(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "alpha", "Not default.", is_default=False)
session = _make_session()
content = _sys_content(session)
assert "Not default." not in content
def test_templates_before_instructions(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "tpl", "TEMPLATE_CONTENT", is_default=True)
session = _make_session(instructions="USER_INSTRUCTIONS")
content = _sys_content(session)
tpl_pos = content.index("TEMPLATE_CONTENT")
instr_pos = content.index("USER_INSTRUCTIONS")
assert tpl_pos < instr_pos
# ---------------------------------------------------------------------------
# Explicit template selection
# ---------------------------------------------------------------------------
class TestExplicitTemplate:
def test_explicit_template_replaces_defaults(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "default-tpl", "DEFAULT_CONTENT", is_default=True)
_create_template(db, "t2", "specific-tpl", "SPECIFIC_CONTENT", is_default=False)
session = _make_session(template="specific-tpl")
content = _sys_content(session)
assert "SPECIFIC_CONTENT" in content
assert "DEFAULT_CONTENT" not in content
def test_explicit_template_not_found(self, tmp_db):
session = _make_session(template="nonexistent")
content = _sys_content(session)
# Graceful degradation — no template content injected
assert "nonexistent" not in content
# ---------------------------------------------------------------------------
# Variable substitution in templates
# ---------------------------------------------------------------------------
class TestTemplateVariables:
def test_model_and_ws_id_substituted(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "vars-tpl", "Model: {{model}}, WS: {{ws_id}}", is_default=True)
session = _make_session()
content = _sys_content(session)
assert "Model: test-model" in content
assert f"WS: {session.ws_id}" in content
def test_node_id_substituted(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "node-tpl", "Node: {{node_id}}", is_default=True)
session = _make_session(node_id="node-42")
content = _sys_content(session)
assert "Node: node-42" in content
def test_unknown_variable_preserved(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "unknown-tpl", "Val: {{unknown_var}}", is_default=True)
session = _make_session()
content = _sys_content(session)
assert "Val: {{unknown_var}}" in content
# ---------------------------------------------------------------------------
# Template persistence and resume
# ---------------------------------------------------------------------------
class TestTemplatePersistence:
def test_template_persisted_in_config(self, tmp_db):
from turnstone.core.memory import load_workstream_config
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "TPL_CONTENT", is_default=False)
session = _make_session(template="my-tpl")
config = load_workstream_config(session.ws_id)
assert config["template"] == "my-tpl"
def test_template_restored_on_resume(self, tmp_db):
from turnstone.core.memory import save_message
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "PERSISTED_TEMPLATE", is_default=False)
# Create session with template, save a message so resume has history
session1 = _make_session(template="my-tpl")
ws_id = session1.ws_id
save_message(ws_id, "user", "hello")
# New session without template, then resume
session2 = _make_session()
assert session2._template_name is None
resumed = session2.resume(ws_id)
assert resumed
assert session2._template_name == "my-tpl"
content = _sys_content(session2)
assert "PERSISTED_TEMPLATE" in content
def test_empty_template_config_means_defaults(self, tmp_db):
from turnstone.core.memory import load_workstream_config
session = _make_session()
config = load_workstream_config(session.ws_id)
assert config["template"] == ""
# ---------------------------------------------------------------------------
# /template slash command
# ---------------------------------------------------------------------------
class TestTemplateSlashCommand:
def test_template_set(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "SLASH_TEMPLATE", is_default=False)
session = _make_session()
content_before = _sys_content(session)
assert "SLASH_TEMPLATE" not in content_before
session.handle_command("/template my-tpl")
assert session._template_name == "my-tpl"
content_after = _sys_content(session)
assert "SLASH_TEMPLATE" in content_after
def test_template_clear(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "EXPLICIT_TEMPLATE", is_default=False)
_create_template(db, "t2", "default-tpl", "DEFAULT_TEMPLATE", is_default=True)
session = _make_session(template="my-tpl")
assert "EXPLICIT_TEMPLATE" in _sys_content(session)
assert "DEFAULT_TEMPLATE" not in _sys_content(session)
session.handle_command("/template clear")
assert session._template_name is None
assert "DEFAULT_TEMPLATE" in _sys_content(session)
assert "EXPLICIT_TEMPLATE" not in _sys_content(session)
def test_template_not_found(self, tmp_db):
ui = NullUI()
ui.on_error = MagicMock()
session = _make_session(ui=ui)
session.handle_command("/template nonexistent")
ui.on_error.assert_called_once()
assert "not found" in ui.on_error.call_args[0][0].lower()
def test_template_show_current(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(db, "t1", "my-tpl", "content", is_default=False)
ui = NullUI()
ui.on_info = MagicMock()
session = _make_session(ui=ui, template="my-tpl")
session.handle_command("/template")
ui.on_info.assert_called_once()
assert "my-tpl" in ui.on_info.call_args[0][0]
# ---------------------------------------------------------------------------
# MCP-origin templates
# ---------------------------------------------------------------------------
class TestMCPTemplates:
def test_mcp_readonly_template_as_default(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(
db,
"t1",
"mcp__server__prompt",
"MCP_CONTENT",
is_default=True,
origin="mcp",
mcp_server="server",
readonly=True,
)
session = _make_session()
content = _sys_content(session)
assert "MCP_CONTENT" in content
def test_mcp_template_selectable_explicitly(self, tmp_db):
from turnstone.core.storage import get_storage
db = get_storage()
_create_template(
db,
"t1",
"mcp__server__code",
"MCP_EXPLICIT",
is_default=False,
origin="mcp",
mcp_server="server",
readonly=True,
)
session = _make_session(template="mcp__server__code")
content = _sys_content(session)
assert "MCP_EXPLICIT" in content
+14
View File
@@ -209,6 +209,20 @@ def test_create_workstream_target_node():
assert restored.name == "debug-ws"
def test_create_workstream_template_field():
msg = CreateWorkstreamMessage(name="ws", template="code-review")
assert msg.template == "code-review"
raw = msg.to_json()
restored = InboundMessage.from_json(raw)
assert isinstance(restored, CreateWorkstreamMessage)
assert restored.template == "code-review"
def test_create_workstream_template_default_empty():
msg = CreateWorkstreamMessage(name="ws")
assert msg.template == ""
def test_list_nodes_round_trip():
msg = ListNodesMessage()
raw = msg.to_json()
+10
View File
@@ -1,6 +1,7 @@
"""Tests for turnstone.sdk.events — SSE event deserialization."""
from turnstone.sdk.events import (
ApprovalResolvedEvent,
ApproveRequestEvent,
BusyErrorEvent,
ClearUiEvent,
@@ -92,6 +93,15 @@ def test_approve_request_event():
assert len(e.items) == 1
def test_approval_resolved_event():
e = ServerEvent.from_dict(
{"type": "approval_resolved", "approved": False, "feedback": "Approval timed out"}
)
assert isinstance(e, ApprovalResolvedEvent)
assert e.approved is False
assert e.feedback == "Approval timed out"
def test_tool_result_event():
e = ServerEvent.from_dict(
{"type": "tool_result", "call_id": "c1", "name": "search", "output": "found it"}
+297 -7
View File
@@ -144,12 +144,21 @@ class TestChatSessionConstruction:
class TestPlanExec:
"""Tests for _exec_plan: unique session-scoped plan file and existing-plan injection."""
def _run_plan(self, session, prompt, agent_return="# Plan\n\nDo the thing."):
_VALID_PLAN = (
"## Goal\n\nDo the thing.\n\n"
"## Current State\n\nFile foo.py has bar().\n\n"
"## Plan\n\n1. Edit foo.py line 10.\n\n"
"## Risks\n\nNone."
)
def _run_plan(self, session, prompt, agent_return=None):
"""Invoke _exec_plan with _run_agent patched to avoid LLM calls.
Returns (call_id_returned, content_returned, captured_messages) where
captured_messages is the agent_messages list passed to _run_agent.
"""
if agent_return is None:
agent_return = self._VALID_PLAN
captured = {}
def fake_run_agent(messages, **kwargs):
@@ -175,10 +184,9 @@ class TestPlanExec:
"""Written plan file contains the agent's output verbatim."""
monkeypatch.chdir(tmp_path)
session = _make_session()
plan_content = "## Goal\n\nAdd a new endpoint."
self._run_plan(session, "add endpoint", agent_return=plan_content)
self._run_plan(session, "add endpoint")
plan_file = tmp_path / f".plan-{session._ws_id}.md"
assert plan_file.read_text() == plan_content
assert plan_file.read_text() == self._VALID_PLAN
def test_two_sessions_produce_different_files(self, tmp_db, tmp_path, monkeypatch):
"""Two ChatSession instances never collide on the same plan file."""
@@ -262,10 +270,292 @@ class TestPlanExec:
"""_exec_plan returns (call_id, agent_output)."""
monkeypatch.chdir(tmp_path)
session = _make_session()
agent_output = "## Goal\n\nBuild it."
call_id, content, _ = self._run_plan(session, "do stuff", agent_return=agent_output)
call_id, content, _ = self._run_plan(session, "do stuff")
assert call_id == "test-call-1"
assert content == agent_output
assert content == self._VALID_PLAN
def test_exec_plan_retries_on_garbage(self, tmp_db, tmp_path, monkeypatch):
"""When _run_agent returns garbage, _exec_plan retries once."""
monkeypatch.chdir(tmp_path)
session = _make_session()
good_plan = (
"## Goal\n\nAdd feature X.\n\n"
"## Current State\n\nFile foo.py has bar().\n\n"
"## Plan\n\n1. Edit foo.py:bar()\n\n"
"## Risks\n\nNone."
)
call_count = 0
def fake_run_agent(messages, **kwargs):
nonlocal call_count
call_count += 1
if call_count == 1:
return "Sure, do the thing."
return good_plan
item = {"call_id": "c1", "prompt": "add feature X"}
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
_, content = session._exec_plan(item)
assert call_count == 2
assert "## Goal" in content
def test_exec_plan_warning_on_double_failure(self, tmp_db, tmp_path, monkeypatch):
"""When both attempts produce garbage, content gets a warning prefix."""
monkeypatch.chdir(tmp_path)
session = _make_session()
def fake_run_agent(messages, **kwargs):
return "nope"
item = {"call_id": "c1", "prompt": "add feature X"}
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
_, content = session._exec_plan(item)
assert content.startswith("[Warning:")
def test_retry_continues_agent_conversation(self, tmp_db, tmp_path, monkeypatch):
"""Retry appends coaching to the same agent_messages list."""
monkeypatch.chdir(tmp_path)
session = _make_session()
captured_messages: list[list] = []
def fake_run_agent(messages, **kwargs):
captured_messages.append(list(messages))
if len(captured_messages) == 1:
return "garbage"
return (
"## Goal\n\nDone.\n\n## Current State\n\nx\n\n## Plan\n\n1. x\n\n## Risks\n\nNone."
)
item = {"call_id": "c1", "prompt": "add feature X"}
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._exec_plan(item)
assert len(captured_messages) == 2
# Second call should have more messages (coaching appended)
assert len(captured_messages[1]) > len(captured_messages[0])
# Last user message in second call is the coaching message
assert "did not follow" in captured_messages[1][-1]["content"]
# ---------------------------------------------------------------------------
# Plan validation
# ---------------------------------------------------------------------------
class TestPlanValidation:
"""Tests for ChatSession._validate_plan quality gate."""
GOOD_PLAN = (
"## Goal\n\nAdd authentication to the API.\n\n"
"## Current State\n\nFile server.py:45 has no auth middleware.\n\n"
"## Plan\n\n1. Add AuthMiddleware to server.py.\n"
"2. Create auth.py with JWT verification.\n\n"
"## Risks\n\nToken expiry handling may need tuning."
)
def test_valid_plan_passes(self):
valid, issues = ChatSession._validate_plan(self.GOOD_PLAN, "add auth")
assert valid
assert issues == []
def test_too_short_fails(self):
valid, issues = ChatSession._validate_plan("Do the thing.", "do stuff")
assert not valid
assert any("too short" in i for i in issues)
def test_no_sections_fails(self):
content = "A" * 150 # long enough but no sections
valid, issues = ChatSession._validate_plan(content, "build it")
assert not valid
assert any("missing plan sections" in i for i in issues)
def test_echo_detection(self):
goal = "deliver a simpsons quote from a specific episode"
content = "Deliver a Simpsons quote from a specific episode"
valid, issues = ChatSession._validate_plan(content, goal)
assert not valid
assert any("echo" in i for i in issues)
def test_refusal_detection(self):
content = "I cannot create a plan for this task because " + "x" * 100
valid, issues = ChatSession._validate_plan(content, "do stuff")
assert not valid
assert any("refusal" in i for i in issues)
def test_partial_sections_passes(self):
"""2 out of 4 sections is enough to pass."""
content = (
"## Goal\n\nFix the bug in parsing.\n\n"
"## Plan\n\n1. Edit parser.py line 42.\n"
"2. Add boundary check.\n"
"This is enough detail to proceed with confidence."
)
valid, issues = ChatSession._validate_plan(content, "fix bug")
assert valid
def test_one_section_fails(self):
"""Only 1 out of 4 sections is not enough."""
content = (
"## Goal\n\nFix the bug.\n\n"
"We should probably edit parser.py and add some checks "
"to the boundary handling code path for safety."
)
valid, issues = ChatSession._validate_plan(content, "fix bug")
assert not valid
assert any("missing plan sections" in i for i in issues)
# ---------------------------------------------------------------------------
# Plan refinement loop
# ---------------------------------------------------------------------------
class TestPlanRefinement:
"""Tests for the iterative plan refinement loop in _execute_tools."""
GOOD_PLAN = TestPlanValidation.GOOD_PLAN
def test_feedback_triggers_refinement(self, tmp_db, tmp_path, monkeypatch):
"""User feedback causes _refine_plan to run, then approval exits."""
monkeypatch.chdir(tmp_path)
session = _make_session()
refine_called = []
review_responses = iter(["add error handling", ""])
session.ui = MagicMock(spec_set=NullUI)
session.ui.on_plan_review.side_effect = lambda c: next(review_responses)
session.ui.on_info = MagicMock()
session.ui.on_state_change = MagicMock()
revised = self.GOOD_PLAN + "\n\n3. Add error handling."
def fake_refine(content, goal, feedback):
refine_called.append(feedback)
return revised
with patch.object(session, "_refine_plan", side_effect=fake_refine):
items = [
{
"func_name": "create_plan",
"call_id": "c1",
"prompt": "add auth",
}
]
results = [("c1", self.GOOD_PLAN)]
# Manually invoke the post-plan gate portion of _execute_tools.
# We test the loop by calling the gate code directly.
session.auto_approve = False
original_goal = items[0].get("prompt", "")
output = results[0][1]
refinement_round = 0
while refinement_round < session._MAX_PLAN_REFINEMENTS:
resp = session.ui.on_plan_review(output)
if resp.lower() in ("n", "no", "reject"):
break
elif resp:
output = session._refine_plan(output, original_goal, resp)
refinement_round += 1
else:
break
assert len(refine_called) == 1
assert refine_called[0] == "add error handling"
assert "error handling" in output
def test_reject_skips_refinement(self, tmp_db, tmp_path, monkeypatch):
"""Rejection exits immediately without calling _refine_plan."""
monkeypatch.chdir(tmp_path)
session = _make_session()
session.ui = MagicMock(spec_set=NullUI)
session.ui.on_plan_review.return_value = "reject"
with patch.object(session, "_refine_plan") as mock_refine:
output = self.GOOD_PLAN
resp = session.ui.on_plan_review(output)
if resp.lower() in ("n", "no", "reject"):
output += "\n\n---\nUser REJECTED"
elif resp:
output = session._refine_plan(output, "g", resp)
mock_refine.assert_not_called()
assert "REJECTED" in output
def test_approve_skips_refinement(self, tmp_db, tmp_path, monkeypatch):
"""Empty response (enter) approves without refinement."""
monkeypatch.chdir(tmp_path)
session = _make_session()
session.ui = MagicMock(spec_set=NullUI)
session.ui.on_plan_review.return_value = ""
with patch.object(session, "_refine_plan") as mock_refine:
output = self.GOOD_PLAN
resp = session.ui.on_plan_review(output)
if resp.lower() in ("n", "no", "reject"):
output += "\n\n---\nUser REJECTED"
elif resp:
output = session._refine_plan(output, "g", resp)
mock_refine.assert_not_called()
assert "REJECTED" not in output
def test_max_refinement_rounds(self, tmp_db, tmp_path, monkeypatch):
"""Loop stops after _MAX_PLAN_REFINEMENTS rounds with a final review."""
monkeypatch.chdir(tmp_path)
session = _make_session()
session.ui = MagicMock(spec_set=NullUI)
session.ui.on_plan_review.return_value = "more detail please"
session.ui.on_info = MagicMock()
refine_count = 0
def fake_refine(content, goal, feedback):
nonlocal refine_count
refine_count += 1
return content + f"\n(revision {refine_count})"
with patch.object(session, "_refine_plan", side_effect=fake_refine):
output = self.GOOD_PLAN
original_goal = "add auth"
refinement_round = 0
while True:
resp = session.ui.on_plan_review(output)
if (
resp.lower() in ("n", "no", "reject")
or not resp
or refinement_round >= session._MAX_PLAN_REFINEMENTS
):
break
output = session._refine_plan(output, original_goal, resp)
refinement_round += 1
assert refine_count == session._MAX_PLAN_REFINEMENTS
# User gets one extra review call after max rounds (the final prompt)
assert session.ui.on_plan_review.call_count == session._MAX_PLAN_REFINEMENTS + 1
def test_refine_plan_message_structure(self, tmp_db, tmp_path, monkeypatch):
"""_refine_plan passes system + prior plan + feedback to _run_agent."""
monkeypatch.chdir(tmp_path)
session = _make_session()
captured = {}
def fake_run_agent(messages, **kwargs):
captured["messages"] = list(messages)
return self.GOOD_PLAN
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
session._refine_plan(self.GOOD_PLAN, "add auth", "add tests too")
msgs = captured["messages"]
assert msgs[0]["role"] == "system"
assert msgs[1]["role"] == "assistant"
assert msgs[1]["tool_calls"][0]["function"]["name"] == "create_plan"
assert msgs[2]["role"] == "tool"
assert msgs[2]["content"] == self.GOOD_PLAN
assert msgs[3]["role"] == "user"
assert "add tests too" in msgs[3]["content"]
# ---------------------------------------------------------------------------
+81
View File
@@ -84,3 +84,84 @@ def test_first_match_wins(storage):
storage.create_tool_policy("p1", "deny-bash", "bash*", "deny", 100)
storage.create_tool_policy("p2", "allow-bash", "bash*", "allow", 50)
assert evaluate_tool_policy(storage, "bash_exec") == "deny"
# ---------------------------------------------------------------------------
# MCP resource and prompt policy patterns
# ---------------------------------------------------------------------------
def test_mcp_resource_wildcard_deny(storage):
"""Deny all MCP resource reads via glob pattern."""
storage.create_tool_policy("p1", "block-resources", "mcp_resource__*", "deny", 100)
assert evaluate_tool_policy(storage, "mcp_resource__file:///secret.txt") == "deny"
assert evaluate_tool_policy(storage, "mcp_resource__db://users") == "deny"
assert evaluate_tool_policy(storage, "read_file") is None # unrelated tool
def test_mcp_resource_per_server_pattern(storage):
"""Allow resources from a specific server, deny others."""
storage.create_tool_policy("p1", "block-all-resources", "mcp_resource__*", "deny", 50)
storage.create_tool_policy("p2", "allow-docs", "mcp_resource__file:///docs/*", "allow", 100)
assert evaluate_tool_policy(storage, "mcp_resource__file:///docs/readme.md") == "allow"
assert evaluate_tool_policy(storage, "mcp_resource__file:///etc/passwd") == "deny"
def test_mcp_prompt_wildcard_ask(storage):
"""Require approval for all MCP prompt invocations."""
storage.create_tool_policy("p1", "ask-prompts", "mcp__*", "ask", 100)
assert evaluate_tool_policy(storage, "mcp__github__code_review") == "ask"
assert evaluate_tool_policy(storage, "mcp__templates__greeting") == "ask"
assert evaluate_tool_policy(storage, "bash") is None
def test_mcp_prompt_per_server_allow(storage):
"""Auto-approve prompts from a trusted server."""
storage.create_tool_policy("p1", "ask-all-mcp", "mcp__*", "ask", 50)
storage.create_tool_policy("p2", "allow-trusted", "mcp__trusted__*", "allow", 100)
assert evaluate_tool_policy(storage, "mcp__trusted__greeting") == "allow"
assert evaluate_tool_policy(storage, "mcp__untrusted__evil") == "ask"
def test_mcp_batch_mixed(storage):
"""Batch evaluation with mixed MCP and built-in tools."""
storage.create_tool_policy("p1", "block-resources", "mcp_resource__*", "deny", 100)
storage.create_tool_policy("p2", "allow-prompts", "mcp__trusted__*", "allow", 100)
results = evaluate_tool_policies_batch(
storage,
["mcp_resource__file:///x", "mcp__trusted__greeting", "bash", "mcp__other__y"],
)
assert results["mcp_resource__file:///x"] == "deny"
assert results["mcp__trusted__greeting"] == "allow"
assert results["bash"] is None
assert results["mcp__other__y"] is None
def test_normalize_resource_uri_prevents_traversal():
"""URI normalization resolves .. segments to prevent policy traversal bypass."""
from turnstone.core.session import ChatSession
# Normal URI unchanged
assert ChatSession._normalize_resource_uri("file:///docs/readme.md") == "file:///docs/readme.md"
# Traversal resolved
assert ChatSession._normalize_resource_uri("file:///docs/../etc/passwd") == "file:///etc/passwd"
# Double traversal
assert ChatSession._normalize_resource_uri("file:///a/b/../../c") == "file:///c"
# Non-file scheme (netloc preserved, path normalized)
assert ChatSession._normalize_resource_uri("db://host/tables/../secrets") == "db://host/secrets"
# Percent-encoded traversal decoded before normalization
assert (
ChatSession._normalize_resource_uri("file:///docs/%2e%2e/etc/passwd")
== "file:///etc/passwd"
)
# Mixed percent-encoded and literal traversal
assert ChatSession._normalize_resource_uri("file:///a/%2e%2e/b/../c") == "file:///c"
def test_mcp_tool_granular_policy(storage):
"""MCP tool calls use their prefixed func_name for granular policy matching."""
storage.create_tool_policy("p1", "ask-all-mcp", "mcp__*", "ask", 50)
storage.create_tool_policy("p2", "allow-github", "mcp__github__*", "allow", 100)
# MCP tools now use func_name as approval_label
assert evaluate_tool_policy(storage, "mcp__github__search") == "allow"
assert evaluate_tool_policy(storage, "mcp__untrusted__exec") == "ask"
+14 -4
View File
@@ -72,16 +72,24 @@ class TestToolsMetadata:
"""Validate the metadata extracted from JSON files."""
def test_tool_count(self):
assert len(TOOLS) == 16
assert len(TOOLS) == 18
def test_agent_tools_count(self):
assert len(AGENT_TOOLS) == 7
assert len(AGENT_TOOLS) == 9
def test_task_agent_tools_count(self):
assert len(TASK_AGENT_TOOLS) == 10
assert len(TASK_AGENT_TOOLS) == 12
def test_auto_approve_sets_match(self):
expected = {"read_file", "search", "math", "man", "web_fetch", "web_search", "notify"}
expected = {
"read_file",
"search",
"math",
"man",
"web_fetch",
"web_search",
"notify",
}
assert expected == AGENT_AUTO_TOOLS
assert expected == TASK_AUTO_TOOLS
@@ -103,6 +111,8 @@ class TestToolsMetadata:
"forget": "key",
"notify": "message",
"watch": "command",
"read_resource": "uri",
"use_prompt": "name",
}
assert expected == PRIMARY_KEY_MAP
+25
View File
@@ -665,6 +665,31 @@ class TestWebUI:
assert ui._approval_result == (True, "looks good")
t.join()
def test_resolve_approval_emits_event(self):
"""resolve_approval should enqueue an approval_resolved SSE event."""
from turnstone.server import WebUI
ui = WebUI(ws_id="test-emit")
listener = ui._register_listener()
# Drain any init events
while not listener.empty():
listener.get_nowait()
ui.resolve_approval(False, "Approval timed out")
# Collect events from the listener
events = []
while not listener.empty():
events.append(listener.get_nowait())
ui._unregister_listener(listener)
resolved = [e for e in events if e.get("type") == "approval_resolved"]
assert len(resolved) == 1
assert resolved[0]["approved"] is False
assert resolved[0]["feedback"] == "Approval timed out"
def test_resolve_plan(self):
from turnstone.server import WebUI
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "0.5.4"
__version__ = "0.5.5"
+27
View File
@@ -136,6 +136,9 @@ class ConsoleCreateWsRequest(BaseModel):
initial_message: str = Field(
default="", description="Optional first message sent after creation"
)
template: str = Field(
default="", description="Prompt template name (replaces default templates)"
)
class ConsoleCreateWsResponse(BaseModel):
@@ -286,6 +289,9 @@ class PromptTemplateInfo(BaseModel):
is_default: bool
org_id: str
created_by: str
origin: str = "manual"
mcp_server: str = ""
readonly: bool = False
created: str
updated: str
@@ -347,4 +353,25 @@ class AuditEventInfo(BaseModel):
class ListAuditEventsResponse(BaseModel):
events: list[AuditEventInfo]
# ---------------------------------------------------------------------------
# Channels
# ---------------------------------------------------------------------------
class ChannelUserInfo(BaseModel):
channel_type: str
channel_user_id: str
user_id: str
created: str
class ListChannelUsersResponse(BaseModel):
channels: list[ChannelUserInfo]
class CreateChannelUserRequest(BaseModel):
channel_type: str = Field(..., description="Channel type (e.g. discord, slack)")
channel_user_id: str = Field(..., description="External channel user identifier")
total: int
+31
View File
@@ -10,6 +10,7 @@ if TYPE_CHECKING:
from turnstone.api.console_schemas import (
AssignRoleRequest,
AuditEventInfo,
ChannelUserInfo,
ClusterNodesResponse,
ClusterOverviewResponse,
ClusterSnapshotResponse,
@@ -17,10 +18,12 @@ from turnstone.api.console_schemas import (
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateChannelUserRequest,
CreatePromptTemplateRequest,
CreateRoleRequest,
CreateToolPolicyRequest,
ListAuditEventsResponse,
ListChannelUsersResponse,
ListOrgsResponse,
ListPromptTemplatesResponse,
ListRolesResponse,
@@ -219,6 +222,31 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
error_codes=[404],
tags=["Admin"],
),
# --- Channels ---
EndpointSpec(
"/v1/api/admin/users/{user_id}/channels",
"GET",
"List channel links for a user",
response_model=ListChannelUsersResponse,
tags=["Admin"],
),
EndpointSpec(
"/v1/api/admin/users/{user_id}/channels",
"POST",
"Link a channel account to a user",
request_model=CreateChannelUserRequest,
response_model=ChannelUserInfo,
error_codes=[400, 404, 409],
tags=["Admin"],
),
EndpointSpec(
"/v1/api/admin/channels/{channel_type}/{channel_user_id}",
"DELETE",
"Unlink a channel account",
response_model=StatusResponse,
error_codes=[404],
tags=["Admin"],
),
# --- Schedules ---
EndpointSpec(
"/v1/api/admin/schedules",
@@ -483,6 +511,9 @@ _ALL_MODELS: list[type[BaseModel]] = [
CreateTokenRequest,
CreateTokenResponse,
ListTokensResponse,
ChannelUserInfo,
CreateChannelUserRequest,
ListChannelUsersResponse,
ClusterOverviewResponse,
ClusterNodesResponse,
ClusterWorkstreamsResponse,
+3
View File
@@ -175,6 +175,7 @@ class CreateScheduleRequest(BaseModel):
initial_message: str = Field(description="Message sent to the new workstream")
auto_approve: bool = Field(default=False)
auto_approve_tools: list[str] = Field(default_factory=list)
template: str = Field(default="", description="Prompt template name")
enabled: bool = Field(default=True)
@@ -191,6 +192,7 @@ class UpdateScheduleRequest(BaseModel):
initial_message: str | None = None
auto_approve: bool | None = None
auto_approve_tools: list[str] | None = None
template: str | None = None
enabled: bool | None = None
@@ -208,6 +210,7 @@ class ScheduleInfo(BaseModel):
initial_message: str
auto_approve: bool = False
auto_approve_tools: list[str] = Field(default_factory=list)
template: str = ""
enabled: bool = True
created_by: str = ""
last_run: str | None = None
+10
View File
@@ -47,6 +47,9 @@ class CreateWorkstreamRequest(BaseModel):
default="",
description="Workstream ID to resume atomically during creation (empty = fresh start)",
)
template: str = Field(
default="", description="Prompt template name (replaces default templates)"
)
class CreateWorkstreamResponse(BaseModel):
@@ -143,6 +146,12 @@ class WorkstreamCounts(BaseModel):
error: int = 0
class McpStatus(BaseModel):
servers: int = 0
resources: int = 0
prompts: int = 0
class HealthResponse(BaseModel):
status: str = Field(examples=["ok", "degraded"])
version: str = ""
@@ -150,3 +159,4 @@ class HealthResponse(BaseModel):
model: str = ""
workstreams: WorkstreamCounts = WorkstreamCounts()
backend: BackendStatus | None = None
mcp: McpStatus | None = None
File diff suppressed because it is too large Load Diff
+1
View File
@@ -21,3 +21,4 @@ class ChannelConfig:
model: str = ""
auto_approve: bool = False
auto_approve_tools: list[str] = field(default_factory=list)
template: str = ""
+3
View File
@@ -49,11 +49,13 @@ class ChannelRouter:
*,
auto_approve: bool = False,
auto_approve_tools: list[str] | None = None,
template: str = "",
) -> None:
self._broker = broker
self._storage = storage
self._auto_approve = auto_approve
self._auto_approve_tools: list[str] = auto_approve_tools or []
self._template = template
self._pending: dict[str, asyncio.Event] = {}
self._pending_results: dict[str, str] = {}
self._global_task: asyncio.Task[None] | None = None
@@ -172,6 +174,7 @@ class ChannelRouter:
resume_ws=resume_ws,
auto_approve=self._auto_approve,
auto_approve_tools=list(self._auto_approve_tools),
template=self._template,
)
cid = msg.correlation_id
waiter = asyncio.Event()
+1
View File
@@ -141,6 +141,7 @@ class TurnstoneBot:
storage,
auto_approve=config.auto_approve,
auto_approve_tools=list(config.auto_approve_tools),
template=config.template,
)
self._subscribed_ws: set[str] = set()
+11 -1
View File
@@ -204,7 +204,8 @@ class TerminalUI(SessionUI):
try:
prompt_text = (
f" \001{BOLD}\002Plan ready.\001{RESET}\002 "
f"\001{DIM}\002[enter to approve, or give feedback]\001{RESET}\002 "
f"\001{DIM}\002[enter to approve, feedback to amend, "
f"ctrl-c to reject]\001{RESET}\002 "
)
resp = input(prompt_text).strip()
except EOFError:
@@ -724,6 +725,11 @@ def main() -> None:
default=None,
help="Developer instructions injected as developer message",
)
parser.add_argument(
"--template",
default=None,
help="Prompt template name (replaces default templates)",
)
parser.add_argument(
"--temperature",
type=float,
@@ -951,6 +957,7 @@ def main() -> None:
tool_search=args.tool_search,
tool_search_threshold=args.tool_search_threshold,
tool_search_max_results=args.tool_search_max_results,
template=args.template,
)
# Create workstream manager and initial workstream
@@ -1001,6 +1008,9 @@ def main() -> None:
mcp_tools = mcp_client.get_tools()
if mcp_tools:
print(f"MCP tools: {len(mcp_tools)} from {mcp_client.server_count} server(s)")
from turnstone.core.storage import get_storage as _cli_get_storage
mcp_client.set_storage(_cli_get_storage())
print("Type /help for commands, /ws for workstreams, /exit or Ctrl+D to quit.\n")
# Prompt string -- use a short display name
+37 -12
View File
@@ -320,6 +320,9 @@ class ClusterCollector:
total_tokens = 0
total_tool_calls = 0
total_ws = 0
mcp_servers = 0
mcp_resources = 0
mcp_prompts = 0
versions: set[str] = set()
with self._lock:
for node in self._nodes.values():
@@ -332,8 +335,12 @@ class ClusterCollector:
ver = node.health.get("version", "")
if ver:
versions.add(ver)
mcp = node.health.get("mcp", {})
mcp_servers += mcp.get("servers", 0)
mcp_resources += mcp.get("resources", 0)
mcp_prompts += mcp.get("prompts", 0)
node_count = len(self._nodes)
return {
result: dict[str, Any] = {
"nodes": node_count,
"workstreams": total_ws,
"states": states,
@@ -344,6 +351,11 @@ class ClusterCollector:
"version_drift": len(versions) > 1,
"versions": sorted(versions),
}
if mcp_servers:
result["mcp_servers"] = mcp_servers
result["mcp_resources"] = mcp_resources
result["mcp_prompts"] = mcp_prompts
return result
def get_version_info(self) -> dict[str, Any]:
"""Return per-node version map and drift flag."""
@@ -515,6 +527,9 @@ class ClusterCollector:
total_tokens = 0
total_tool_calls = 0
total_ws = 0
mcp_servers = 0
mcp_resources = 0
mcp_prompts = 0
versions: set[str] = set()
for node in self._nodes.values():
@@ -530,6 +545,10 @@ class ClusterCollector:
ver = node.health.get("version", "")
if ver:
versions.add(ver)
mcp = node.health.get("mcp", {})
mcp_servers += mcp.get("servers", 0)
mcp_resources += mcp.get("resources", 0)
mcp_prompts += mcp.get("prompts", 0)
nodes_out.append(
{
@@ -546,19 +565,25 @@ class ClusterCollector:
node_count = len(self._nodes)
overview: dict[str, Any] = {
"nodes": node_count,
"workstreams": total_ws,
"states": states,
"aggregate": {
"total_tokens": total_tokens,
"total_tool_calls": total_tool_calls,
},
"version_drift": len(versions) > 1,
"versions": sorted(versions),
}
if mcp_servers:
overview["mcp_servers"] = mcp_servers
overview["mcp_resources"] = mcp_resources
overview["mcp_prompts"] = mcp_prompts
return {
"nodes": nodes_out,
"overview": {
"nodes": node_count,
"workstreams": total_ws,
"states": states,
"aggregate": {
"total_tokens": total_tokens,
"total_tool_calls": total_tool_calls,
},
"version_drift": len(versions) > 1,
"versions": sorted(versions),
},
"overview": overview,
"timestamp": time.time(),
}
+2
View File
@@ -208,6 +208,7 @@ class TaskScheduler:
auto_approve=bool(task.get("auto_approve", 0)),
auto_approve_tools=self._parse_tools(task),
user_id=task.get("created_by", ""),
template=task.get("template", ""),
)
self._broker.push_inbound(msg.to_json(), node_id=node_id)
@@ -233,6 +234,7 @@ class TaskScheduler:
auto_approve=bool(task.get("auto_approve", 0)),
auto_approve_tools=self._parse_tools(task),
user_id=task.get("created_by", ""),
template=task.get("template", ""),
)
self._broker.push_inbound(msg.to_json())
+32 -5
View File
@@ -349,6 +349,7 @@ async def create_workstream(request: Request) -> JSONResponse:
raw_name = body.get("name", "")
raw_model = body.get("model", "")
raw_initial_message = body.get("initial_message", "")
raw_template = body.get("template", "")
if not isinstance(raw_node_id, str):
raw_node_id = "" if raw_node_id is None else None
if not isinstance(raw_name, str):
@@ -357,20 +358,32 @@ async def create_workstream(request: Request) -> JSONResponse:
raw_model = "" if raw_model is None else None
if not isinstance(raw_initial_message, str):
raw_initial_message = "" if raw_initial_message is None else None
if raw_node_id is None or raw_name is None or raw_model is None or raw_initial_message is None:
if not isinstance(raw_template, str):
raw_template = "" if raw_template is None else None
if (
raw_node_id is None
or raw_name is None
or raw_model is None
or raw_initial_message is None
or raw_template is None
):
return JSONResponse(
{"error": "node_id, name, model, and initial_message must be strings"}, status_code=400
{"error": "node_id, name, model, initial_message, and template must be strings"},
status_code=400,
)
node_id = raw_node_id
name = raw_name[:256]
model = raw_model[:128]
initial_message = raw_initial_message[:4096]
template = raw_template[:256]
from turnstone.mq.protocol import CreateWorkstreamMessage
# General pool — push to shared queue, any bridge picks it up
if node_id == "pool":
msg = CreateWorkstreamMessage(name=name, model=model, initial_message=initial_message)
msg = CreateWorkstreamMessage(
name=name, model=model, initial_message=initial_message, template=template
)
broker.push_inbound(msg.to_json())
log.debug("Pool dispatch: correlation_id=%s name=%r", msg.correlation_id, name)
return JSONResponse(
@@ -397,6 +410,7 @@ async def create_workstream(request: Request) -> JSONResponse:
model=model,
target_node=node_id,
initial_message=initial_message,
template=template,
)
broker.push_inbound(msg.to_json(), node_id=node_id)
@@ -1130,12 +1144,15 @@ async def admin_create_schedule(request: Request) -> JSONResponse:
auto_approve = bool(body.get("auto_approve", False))
raw_tools = body.get("auto_approve_tools", [])
auto_approve_tools = raw_tools if isinstance(raw_tools, list) else []
template = str(body.get("template", "")).strip()[:256]
enabled = bool(body.get("enabled", True))
if not name:
return JSONResponse({"error": "name is required"}, status_code=400)
if not initial_message:
return JSONResponse({"error": "initial_message is required"}, status_code=400)
if template and not storage.get_prompt_template_by_name(template):
return JSONResponse({"error": f"Template not found: {template}"}, status_code=400)
validation_err = _validate_schedule_fields(schedule_type, cron_expr, at_time)
if validation_err:
@@ -1170,6 +1187,7 @@ async def admin_create_schedule(request: Request) -> JSONResponse:
auto_approve_tools=auto_approve_tools,
created_by=created_by,
next_run=next_run if enabled else "",
template=template,
)
if not enabled:
@@ -1244,6 +1262,11 @@ async def admin_update_schedule(request: Request) -> JSONResponse:
if "auto_approve_tools" in body:
raw = body["auto_approve_tools"]
updates["auto_approve_tools"] = raw if isinstance(raw, list) else []
if "template" in body:
tpl_name = str(body["template"]).strip()[:256]
if tpl_name and not storage.get_prompt_template_by_name(tpl_name):
return JSONResponse({"error": f"Template not found: {tpl_name}"}, status_code=400)
updates["template"] = tpl_name
if "enabled" in body:
updates["enabled"] = bool(body["enabled"])
@@ -2014,7 +2037,7 @@ async def admin_create_template(request: Request) -> JSONResponse:
return body
name = str(body.get("name", "")).strip()[:256]
content = str(body.get("content", "")).strip()
content = str(body.get("content", "")).strip()[:32768]
category = str(body.get("category", "general")).strip()[:64]
variables = str(body.get("variables", "[]")).strip()
try:
@@ -2074,6 +2097,8 @@ async def admin_update_template(request: Request) -> JSONResponse:
existing = storage.get_prompt_template(template_id)
if existing is None:
return JSONResponse({"error": "Template not found"}, status_code=404)
if existing.get("readonly"):
return JSONResponse({"error": "MCP-sourced templates are read-only"}, status_code=403)
body = await read_json_or_400(request)
if isinstance(body, JSONResponse):
@@ -2083,7 +2108,7 @@ async def admin_update_template(request: Request) -> JSONResponse:
if "name" in body:
updates["name"] = str(body["name"]).strip()[:256]
if "content" in body:
updates["content"] = str(body["content"]).strip()
updates["content"] = str(body["content"]).strip()[:32768]
if "category" in body:
updates["category"] = str(body["category"]).strip()[:64]
if "variables" in body:
@@ -2130,6 +2155,8 @@ async def admin_delete_template(request: Request) -> JSONResponse:
existing = storage.get_prompt_template(template_id)
if existing is None:
return JSONResponse({"error": "Template not found"}, status_code=404)
if existing.get("readonly"):
return JSONResponse({"error": "MCP-sourced templates are read-only"}, status_code=403)
storage.delete_prompt_template(template_id)
+5
View File
@@ -661,6 +661,7 @@ function showCreateScheduleModal() {
document.getElementById("cs-target").value = "auto";
document.getElementById("cs-node").value = "";
document.getElementById("cs-model").value = "";
document.getElementById("cs-template").value = "";
document.getElementById("cs-message").value = "";
document.getElementById("cs-autoapprove").checked = false;
toggleScheduleTypeFields();
@@ -693,6 +694,7 @@ function submitCreateSchedule() {
var nodeId = (document.getElementById("cs-node").value || "").trim();
var model = (document.getElementById("cs-model").value || "").trim();
var message = (document.getElementById("cs-message").value || "").trim();
var template = (document.getElementById("cs-template").value || "").trim();
var autoApprove = document.getElementById("cs-autoapprove").checked;
var errEl = document.getElementById("create-schedule-error");
@@ -729,6 +731,7 @@ function submitCreateSchedule() {
model: model,
initial_message: message,
auto_approve: autoApprove,
template: template,
}),
})
.then(function (r) {
@@ -795,6 +798,7 @@ function showEditScheduleModal(taskId) {
? s.target_mode
: "";
document.getElementById("es-model").value = s.model || "";
document.getElementById("es-template").value = s.template || "";
document.getElementById("es-message").value = s.initial_message || "";
document.getElementById("es-autoapprove").checked = !!s.auto_approve;
document.getElementById("es-enabled").checked = !!s.enabled;
@@ -867,6 +871,7 @@ function submitEditSchedule() {
at_time: atTime,
target_mode: targetMode,
model: (document.getElementById("es-model").value || "").trim(),
template: (document.getElementById("es-template").value || "").trim(),
initial_message: (
document.getElementById("es-message").value || ""
).trim(),
+95 -2
View File
@@ -140,6 +140,9 @@ function recomputeOverview() {
var totalTokens = 0,
totalToolCalls = 0,
totalWs = 0;
var mcpServers = 0,
mcpResources = 0,
mcpPrompts = 0;
var versions = {};
Object.keys(clusterState.nodes).forEach(function (nid) {
var node = clusterState.nodes[nid];
@@ -154,6 +157,10 @@ function recomputeOverview() {
totalTokens += aggTokens || nodeWsTokens;
totalToolCalls += (node.aggregate || {}).total_tool_calls || 0;
if (node.version) versions[node.version] = true;
var mcp = (node.health || {}).mcp || {};
mcpServers += mcp.servers || 0;
mcpResources += mcp.resources || 0;
mcpPrompts += mcp.prompts || 0;
});
var versionList = Object.keys(versions).sort();
clusterState.overview = {
@@ -167,6 +174,11 @@ function recomputeOverview() {
version_drift: versionList.length > 1,
versions: versionList,
};
if (mcpServers > 0) {
clusterState.overview.mcp_servers = mcpServers;
clusterState.overview.mcp_resources = mcpResources;
clusterState.overview.mcp_prompts = mcpPrompts;
}
}
function buildNodeInfoFromSnapshot(node) {
@@ -227,6 +239,23 @@ function renderFromState() {
}).length;
document.getElementById("node-ws-summary").textContent =
active + " active \u00b7 " + wsList.length + " total";
var mcpSumEl = document.getElementById("node-mcp-summary");
if (mcpSumEl) {
var mcpInfo = snapNode.health && snapNode.health.mcp;
if (mcpInfo && mcpInfo.servers > 0) {
mcpSumEl.textContent =
mcpInfo.servers +
" MCP server" +
(mcpInfo.servers !== 1 ? "s" : "") +
" \u00b7 " +
mcpInfo.resources +
" resources \u00b7 " +
mcpInfo.prompts +
" prompts";
} else {
mcpSumEl.textContent = "";
}
}
renderWsTable(document.getElementById("node-ws-table"), wsList);
}
} else if (currentView === "filtered") {
@@ -470,6 +499,43 @@ function renderStatusBar(overview) {
verEl.appendChild(verLbl);
metricsContainer.appendChild(verEl);
}
// MCP aggregate metrics
if (overview.mcp_servers && overview.mcp_servers > 0) {
var mcpDivider = document.createElement("span");
mcpDivider.className = "csb-divider";
mcpDivider.setAttribute("aria-hidden", "true");
metricsContainer.appendChild(mcpDivider);
var mcpTitles = {
mcp: "MCP servers",
rsrc: "MCP resources",
pmpt: "MCP prompts",
};
var mcpMetrics = [
{ value: overview.mcp_servers, label: "mcp" },
{ value: overview.mcp_resources, label: "rsrc" },
{ value: overview.mcp_prompts, label: "pmpt" },
];
mcpMetrics.forEach(function (m) {
var el = document.createElement("span");
el.className = "csb-metric";
el.title = mcpTitles[m.label] || "";
if (m.label === "mcp") {
var dot = document.createElement("span");
dot.className = "csb-mcp-dot";
dot.setAttribute("aria-hidden", "true");
el.appendChild(dot);
}
var valSpan = document.createElement("span");
valSpan.className = "csb-metric-value";
valSpan.textContent = formatCount(m.value);
var labelSpan = document.createElement("span");
labelSpan.className = "csb-metric-label";
labelSpan.textContent = m.label;
el.appendChild(valSpan);
el.appendChild(labelSpan);
metricsContainer.appendChild(el);
});
}
}
// --- Node Grouping ---
@@ -1174,6 +1240,27 @@ function showNewWsModal() {
.catch(function () {
/* ignore — auto is always available */
});
// Populate template dropdown
var tplSelect = document.getElementById("new-ws-template");
tplSelect.innerHTML = '<option value="">Use defaults</option>';
authFetch("/v1/api/admin/templates")
.then(function (r) {
return r.json();
})
.then(function (data) {
(data.templates || []).forEach(function (t) {
var opt = document.createElement("option");
opt.value = t.name;
var label = t.name;
if (t.is_default) label += " (default)";
if (t.origin === "mcp") label += " [MCP]";
opt.textContent = label;
tplSelect.appendChild(opt);
});
})
.catch(function () {
/* ignore — defaults still work */
});
document.getElementById("new-ws-name").value = "";
document.getElementById("new-ws-model").value = "";
document.getElementById("new-ws-task").value = "";
@@ -1190,7 +1277,7 @@ function showNewWsModal() {
_newWsTrapHandler = function (e) {
if (e.key === "Tab") {
var box = document.getElementById("new-ws-box");
var focusable = box.querySelectorAll("select, input, button");
var focusable = box.querySelectorAll("select, input, textarea, button");
var first = focusable[0];
var last = focusable[focusable.length - 1];
if (e.shiftKey) {
@@ -1228,6 +1315,7 @@ function submitNewWs() {
var nodeId = document.getElementById("new-ws-node").value;
var name = document.getElementById("new-ws-name").value.trim();
var model = document.getElementById("new-ws-model").value.trim();
var template = document.getElementById("new-ws-template").value;
var task = document.getElementById("new-ws-task").value.trim();
var errEl = document.getElementById("new-ws-error");
var btn = document.getElementById("new-ws-submit");
@@ -1241,6 +1329,7 @@ function submitNewWs() {
if (name) body.name = name;
if (model) body.model = model;
if (task) body.initial_message = task;
if (template) body.template = template;
authFetch("/v1/api/cluster/workstreams/new", {
method: "POST",
@@ -1281,7 +1370,11 @@ document.addEventListener("keydown", function (e) {
e.preventDefault();
hideNewWsModal();
}
if (e.key === "Enter" && e.target.tagName !== "SELECT") {
if (
e.key === "Enter" &&
e.target.tagName !== "SELECT" &&
e.target.tagName !== "TEXTAREA"
) {
e.preventDefault();
var btn = document.getElementById("new-ws-submit");
if (btn && !btn.disabled) submitNewWs();
+47 -22
View File
@@ -689,14 +689,23 @@ function _renderGovTemplates(items) {
var defBadge = t.is_default
? '<span class="scope-badge scope-approve">default</span>'
: "";
var originBadge =
t.origin === "mcp"
? ' <span class="scope-badge scope-deny">mcp:' +
escapeHtml(t.mcp_server) +
"</span>"
: "";
var catBadge =
'<span class="scope-badge">' + escapeHtml(t.category) + "</span>";
var editDisabled = t.readonly ? " disabled" : "";
var deleteDisabled = t.readonly ? " disabled" : "";
html +=
'<div class="admin-row" role="listitem">' +
'<span class="admin-col admin-col-tmname">' +
escapeHtml(t.name) +
" " +
defBadge +
originBadge +
"</span>" +
'<span class="admin-col admin-col-tmcat">' +
catBadge +
@@ -707,12 +716,16 @@ function _renderGovTemplates(items) {
'<span class="admin-col admin-col-actions">' +
'<button class="admin-btn-action" data-edit-tmpl="' +
escapeHtml(t.template_id) +
'">edit</button>' +
'"' +
editDisabled +
">edit</button>" +
'<button class="admin-btn-danger" data-delete-tmpl="' +
escapeHtml(t.template_id) +
'" data-tmpl-name="' +
escapeHtml(t.name) +
'">delete</button>' +
'"' +
deleteDisabled +
">delete</button>" +
"</span></div>";
}
el.innerHTML = html;
@@ -748,6 +761,28 @@ function _renderGovTemplates(items) {
});
}
function _detectTemplateVars(content) {
var matches = content.match(/\{\{(\w+)\}\}/g) || [];
var seen = {};
var result = [];
for (var i = 0; i < matches.length; i++) {
var v = matches[i].replace(/[{}]/g, "");
if (!seen[v]) {
seen[v] = true;
result.push(v);
}
}
return result;
}
function _updateVarsDisplay(contentId, displayId) {
var content = document.getElementById(contentId).value || "";
var vars = _detectTemplateVars(content);
document.getElementById(displayId).textContent = vars.length
? vars.join(", ")
: "(none)";
}
function showCreateTemplateModal() {
_ctmTriggerEl = document.activeElement;
var ov = document.getElementById("create-template-overlay");
@@ -755,7 +790,10 @@ function showCreateTemplateModal() {
document.getElementById("ctm-name").value = "";
document.getElementById("ctm-category").value = "general";
document.getElementById("ctm-content").value = "";
document.getElementById("ctm-variables").value = "";
document.getElementById("ctm-variables").textContent = "(none)";
document.getElementById("ctm-content").oninput = function () {
_updateVarsDisplay("ctm-content", "ctm-variables");
};
document.getElementById("ctm-default").checked = false;
document.getElementById("create-template-error").style.display = "none";
document.getElementById("ctm-name").focus();
@@ -783,12 +821,7 @@ function submitCreateTemplate() {
e.style.display = "";
return;
}
var vars = document.getElementById("ctm-variables").value.trim();
var varList = vars
? vars.split(",").map(function (s) {
return s.trim();
})
: [];
var varList = _detectTemplateVars(content);
document.getElementById("ctm-submit").disabled = true;
authFetch("/v1/api/admin/templates", {
method: "POST",
@@ -839,13 +872,10 @@ function showEditTemplateModal(tmplId) {
document.getElementById("etm-name").value = tmpl.name;
document.getElementById("etm-category").value = tmpl.category;
document.getElementById("etm-content").value = tmpl.content;
var vars = "";
try {
vars = JSON.parse(tmpl.variables || "[]").join(", ");
} catch (e) {
vars = tmpl.variables;
}
document.getElementById("etm-variables").value = vars;
_updateVarsDisplay("etm-content", "etm-variables");
document.getElementById("etm-content").oninput = function () {
_updateVarsDisplay("etm-content", "etm-variables");
};
document.getElementById("etm-default").checked = tmpl.is_default;
document.getElementById("edit-template-error").style.display = "none";
_etmTrapHandler = _installTrap("edit-template-overlay", "edit-template-box");
@@ -863,12 +893,7 @@ function hideEditTemplateModal() {
function submitEditTemplate() {
var id = document.getElementById("etm-id").value;
var content = document.getElementById("etm-content").value;
var vars = document.getElementById("etm-variables").value.trim();
var varList = vars
? vars.split(",").map(function (s) {
return s.trim();
})
: [];
var varList = _detectTemplateVars(content);
document.getElementById("etm-submit").disabled = true;
authFetch("/v1/api/admin/templates/" + id, {
method: "PUT",
+15 -6
View File
@@ -41,6 +41,7 @@
<div class="dash-header">
<span class="dash-header-title">WORKSTREAMS</span>
<span class="dash-header-summary" id="node-ws-summary"></span>
<span id="node-mcp-summary" aria-label="MCP status"></span>
</div>
<div class="dash-colheaders" aria-hidden="true">
<span class="dash-col dash-col-state">STATE</span>
@@ -348,6 +349,10 @@ window.TURNSTONE_KB_SHORTCUTS = [
<input id="new-ws-name" type="text" placeholder="Auto-generated if empty" autocomplete="off">
<label for="new-ws-model">Model <span class="label-hint">optional</span></label>
<input id="new-ws-model" type="text" placeholder="Default model" autocomplete="off">
<label for="new-ws-template">Template <span class="label-hint">optional</span></label>
<select id="new-ws-template">
<option value="">Use defaults</option>
</select>
<label for="new-ws-task">Task <span class="label-hint">optional &mdash; sent as first message</span></label>
<textarea id="new-ws-task" rows="3" placeholder="What should this workstream work on?"></textarea>
<div id="new-ws-buttons">
@@ -483,6 +488,8 @@ window.TURNSTONE_KB_SHORTCUTS = [
</div>
<label for="cs-model">Model <span class="label-hint">optional</span></label>
<input id="cs-model" type="text" placeholder="Default model" autocomplete="off">
<label for="cs-template">Template <span class="label-hint">optional</span></label>
<input id="cs-template" type="text" placeholder="Prompt template name" autocomplete="off">
<label for="cs-message">Initial message</label>
<textarea id="cs-message" rows="3" placeholder="What should the workstream do?"></textarea>
<label class="admin-checkbox"><input id="cs-autoapprove" type="checkbox"> Auto-approve tool calls</label>
@@ -529,6 +536,8 @@ window.TURNSTONE_KB_SHORTCUTS = [
</div>
<label for="es-model">Model</label>
<input id="es-model" type="text" autocomplete="off">
<label for="es-template">Template <span class="label-hint">optional</span></label>
<input id="es-template" type="text" autocomplete="off">
<label for="es-message">Initial message</label>
<textarea id="es-message" rows="3"></textarea>
<label class="admin-checkbox"><input id="es-autoapprove" type="checkbox"> Auto-approve tool calls</label>
@@ -666,10 +675,10 @@ window.TURNSTONE_KB_SHORTCUTS = [
<option value="support">Support</option>
<option value="custom">Custom</option>
</select>
<label for="ctm-content">Content <span class="label-hint">system message text, use {{variable}} for placeholders</span></label>
<textarea id="ctm-content" rows="6" placeholder="You are a helpful assistant for {{project_name}}..."></textarea>
<label for="ctm-variables">Variables <span class="label-hint">comma-separated list</span></label>
<input id="ctm-variables" type="text" placeholder="project_name, review_focus" autocomplete="off">
<label for="ctm-content">Content <span class="label-hint">system message text, use {{model}}, {{ws_id}}, {{node_id}} for placeholders</span></label>
<textarea id="ctm-content" rows="6" placeholder="You are a code reviewer using {{model}}..."></textarea>
<label>Variables <span class="label-hint">auto-detected from content &mdash; available: model, ws_id, node_id</span></label>
<div id="ctm-variables" class="label-hint" style="padding:4px 0;min-height:1.2em"></div>
<label class="admin-checkbox"><input id="ctm-default" type="checkbox"> Set as default for new workstreams</label>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideCreateTemplateModal()">Cancel</button>
@@ -695,8 +704,8 @@ window.TURNSTONE_KB_SHORTCUTS = [
</select>
<label for="etm-content">Content</label>
<textarea id="etm-content" rows="6"></textarea>
<label for="etm-variables">Variables</label>
<input id="etm-variables" type="text" autocomplete="off">
<label>Variables <span class="label-hint">auto-detected from content &mdash; available: model, ws_id, node_id</span></label>
<div id="etm-variables" class="label-hint" style="padding:4px 0;min-height:1.2em"></div>
<label class="admin-checkbox"><input id="etm-default" type="checkbox"> Set as default</label>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideEditTemplateModal()">Cancel</button>
+22
View File
@@ -166,6 +166,17 @@
opacity: 0.7;
}
/* MCP indicator dot — LED effect with magenta glow */
.csb-mcp-dot {
width: 7px;
height: 7px;
border-radius: 50%;
background: var(--magenta);
box-shadow: 0 0 4px var(--magenta-glow);
display: inline-block;
flex-shrink: 0;
}
.csb-loading { color: var(--fg-dim); font-size: 11px; font-style: italic; opacity: 0.8; }
#cluster-status-bar.stale { border-top-color: var(--yellow); }
@@ -454,6 +465,16 @@
}
.dash-cell-node:hover { text-decoration: underline; color: var(--fg-bright); }
/* ==========================================================================
MCP summary in node detail
========================================================================== */
#node-mcp-summary {
color: var(--magenta);
font-size: 11px;
font-family: var(--font-mono);
margin-left: 12px;
}
/* ==========================================================================
Node link
========================================================================== */
@@ -665,6 +686,7 @@
.node-group-header .node-group-cell:last-child { display: none; }
.ncol-version, .node-cell-version { display: none; }
.ncol-health, .node-cell-health { display: none; }
#node-mcp-summary { display: none; }
#main { padding: 16px; padding-bottom: 60px; }
}
@media (max-width: 480px) {
+611 -28
View File
@@ -1,16 +1,16 @@
"""MCP (Model Context Protocol) client manager.
Connects to external MCP tool servers and exposes their tools alongside
turnstone's built-in tools.
Connects to external MCP tool servers and exposes their tools, resources,
and prompts alongside turnstone's built-in capabilities.
Architecture: the MCP SDK is fully async, but turnstone's ChatSession is
synchronous. We bridge the two by running a dedicated asyncio event loop
in a daemon thread. ``call_tool_sync`` dispatches coroutines onto that loop
via ``asyncio.run_coroutine_threadsafe``.
Tool refresh: three mechanisms keep tool lists up-to-date without restart:
1. Push notifications servers declaring ``tools.listChanged`` trigger
immediate refresh via ``ToolListChangedNotification``.
Refresh: three mechanisms keep tool/resource/prompt lists up-to-date:
1. Push notifications servers declaring ``listChanged`` on the
respective capability trigger immediate refresh.
2. Periodic timer servers *without* push support are polled on a
staggered interval (configurable, default 4 h, seeded at launch).
3. Manual ``/mcp refresh [server]`` triggers ``refresh_sync()``.
@@ -19,6 +19,7 @@ Tool refresh: three mechanisms keep tool lists up-to-date without restart:
from __future__ import annotations
import asyncio
import concurrent.futures
import contextlib
import json
import logging
@@ -26,6 +27,7 @@ import os
import random
import threading
import time
import uuid
from contextlib import AsyncExitStack
from pathlib import Path
from typing import TYPE_CHECKING, Any
@@ -112,6 +114,31 @@ class MCPClientManager:
self._listeners: list[Callable[[], None]] = []
self._listeners_lock = threading.Lock()
# Resources — parallel to tools
self._per_server_resources: dict[str, list[dict[str, Any]]] = {}
self._resources: list[dict[str, Any]] = []
self._resource_map: dict[str, tuple[str, str]] = {} # uri → (server, uri)
self._supports_resources: dict[str, bool] = {} # server has resources capability
self._supports_resource_list_changed: dict[str, bool] = {}
self._resource_listeners: list[Callable[[], None]] = []
self._resource_listeners_lock = threading.Lock()
# Prompts — parallel to tools
self._per_server_prompts: dict[str, list[dict[str, Any]]] = {}
self._prompts: list[dict[str, Any]] = []
self._prompt_map: dict[str, tuple[str, str]] = {} # prefixed → (server, original)
self._supports_prompts: dict[str, bool] = {} # server has prompts capability
self._supports_prompt_list_changed: dict[str, bool] = {}
self._prompt_listeners: list[Callable[[], None]] = []
self._prompt_listeners_lock = threading.Lock()
# Template prefix → (server_name, full_template_uri) for URI expansion
self._template_prefixes: dict[str, tuple[str, str]] = {}
# Governance storage (optional — set via set_storage())
self._storage: Any = None
self._sync_lock = threading.Lock()
# Periodic refresh for servers without push notifications
self._refresh_interval = refresh_interval
self._refresh_task: asyncio.Task[None] | None = None
@@ -146,7 +173,16 @@ class MCPClientManager:
# Start periodic refresh for servers without push notifications
needs_periodic = any(
not self._supports_list_changed.get(name, False) for name in self._sessions
not self._supports_list_changed.get(name, False)
or (
self._supports_resources.get(name, False)
and not self._supports_resource_list_changed.get(name, False)
)
or (
self._supports_prompts.get(name, False)
and not self._supports_prompt_list_changed.get(name, False)
)
for name in self._sessions
)
if needs_periodic and self._refresh_interval > 0:
self._refresh_task = asyncio.get_running_loop().create_task(self._periodic_refresh())
@@ -178,20 +214,26 @@ class MCPClientManager:
)
read, write = await self._exit_stack.enter_async_context(stdio_client(params))
# Register notification handler — lightweight; only acts on
# ToolListChangedNotification, which is a no-op if the server
# never sends it.
# Register notification handler — dispatches tool, resource, and
# prompt list-change notifications to the appropriate refresh method.
async def _on_notification(
msg: Any, # RequestResponder | ServerNotification | Exception
) -> None:
if isinstance(msg, mcp_types.ServerNotification) and isinstance(
msg.root, mcp_types.ToolListChangedNotification
):
log.info("Received tools/list_changed from '%s'", name)
try:
await self._refresh_server(name)
except Exception:
log.warning("Refresh after notification failed for '%s'", name, exc_info=True)
if not isinstance(msg, mcp_types.ServerNotification):
return
root = msg.root
try:
if isinstance(root, mcp_types.ToolListChangedNotification):
log.info("Received tools/list_changed from '%s'", name)
await self._refresh_server_tools(name)
elif isinstance(root, mcp_types.ResourceListChangedNotification):
log.info("Received resources/list_changed from '%s'", name)
await self._refresh_server_resources(name)
elif isinstance(root, mcp_types.PromptListChangedNotification):
log.info("Received prompts/list_changed from '%s'", name)
await self._refresh_server_prompts(name)
except Exception:
log.warning("Refresh after notification failed for '%s'", name, exc_info=True)
session = await self._exit_stack.enter_async_context(
ClientSession(read, write, message_handler=_on_notification) # type: ignore[arg-type]
@@ -199,11 +241,22 @@ class MCPClientManager:
await session.initialize()
self._sessions[name] = session
# Check push notification support
# Check push notification support for each capability
caps = session.get_server_capabilities()
tools_cap = getattr(caps, "tools", None) if caps else None
self._supports_list_changed[name] = bool(getattr(tools_cap, "listChanged", False))
resources_cap = getattr(caps, "resources", None) if caps else None
self._supports_resources[name] = resources_cap is not None
self._supports_resource_list_changed[name] = bool(
getattr(resources_cap, "listChanged", False)
)
prompts_cap = getattr(caps, "prompts", None) if caps else None
self._supports_prompts[name] = prompts_cap is not None
self._supports_prompt_list_changed[name] = bool(getattr(prompts_cap, "listChanged", False))
# Discover tools
result = await session.list_tools()
server_tools: list[dict[str, Any]] = []
@@ -213,14 +266,88 @@ class MCPClientManager:
self._per_server_tools[name] = server_tools
self._rebuild_tools()
push_status = " (push)" if self._supports_list_changed[name] else ""
# Discover resources
resource_count = 0
if resources_cap is not None:
server_resources: list[dict[str, Any]] = []
res_result = await session.list_resources()
for r in res_result.resources:
server_resources.append(
{
"uri": str(r.uri),
"name": r.name or "",
"description": r.description or "",
"mimeType": r.mimeType or "",
"server": name,
}
)
# Also include resource templates (catalog-only — not directly
# readable via read_resource since they contain URI placeholders)
tmpl_result = await session.list_resource_templates()
for t in tmpl_result.resourceTemplates:
server_resources.append(
{
"uri": str(t.uriTemplate),
"name": t.name or "",
"description": t.description or "",
"mimeType": t.mimeType or "",
"server": name,
"template": True,
}
)
resource_count = len(server_resources)
self._per_server_resources[name] = server_resources
self._rebuild_resources()
# Discover prompts
prompt_count = 0
if prompts_cap is not None:
server_prompts: list[dict[str, Any]] = []
prompt_result = await session.list_prompts()
for p in prompt_result.prompts:
server_prompts.append(
{
"name": f"mcp__{name}__{p.name}",
"original_name": p.name,
"server": name,
"description": p.description or "",
"arguments": [
{
"name": a.name,
"description": a.description or "",
"required": a.required or False,
}
for a in (p.arguments or [])
],
}
)
prompt_count = len(server_prompts)
self._per_server_prompts[name] = server_prompts
self._rebuild_prompts()
push_parts: list[str] = []
if self._supports_list_changed[name]:
push_parts.append("tools")
if self._supports_resource_list_changed[name]:
push_parts.append("resources")
if self._supports_prompt_list_changed[name]:
push_parts.append("prompts")
push_status = f" (push: {','.join(push_parts)})" if push_parts else ""
log.info(
"Connected MCP server '%s'%d tool(s)%s",
"Connected MCP server '%s'%d tool(s), %d resource(s), %d prompt(s)%s",
name,
len(result.tools),
resource_count,
prompt_count,
push_status,
)
# Sync discovered prompts into governance storage
try:
self.sync_prompts_to_storage()
except Exception:
log.warning("Prompt sync after connect failed for '%s'", name, exc_info=True)
# -- tool refresh --------------------------------------------------------
def _rebuild_tools(self) -> None:
@@ -242,7 +369,7 @@ class MCPClientManager:
self._tool_map = new_map
self._notify_listeners()
async def _refresh_server(self, name: str) -> tuple[list[str], list[str]]:
async def _refresh_server_tools(self, name: str) -> tuple[list[str], list[str]]:
"""Re-fetch tools for one server. Returns ``(added, removed)`` names."""
session = self._sessions.get(name)
if session is None:
@@ -268,10 +395,21 @@ class MCPClientManager:
)
return added, removed
async def _refresh_server(self, name: str) -> tuple[list[str], list[str]]:
"""Re-fetch tools, resources, and prompts for one server.
Returns ``(added_tools, removed_tools)`` names (tool diff only,
for backward compatibility with ``/mcp refresh`` output).
"""
added, removed = await self._refresh_server_tools(name)
await self._refresh_server_resources(name)
await self._refresh_server_prompts(name)
return added, removed
async def _refresh_all(
self, server_name: str | None = None
) -> dict[str, tuple[list[str], list[str]]]:
"""Refresh tools for one or all servers.
"""Refresh tools, resources, and prompts for one or all servers.
For disconnected servers (in config but not connected), attempts
reconnect. Returns ``{server: (added, removed)}`` per server.
@@ -297,6 +435,13 @@ class MCPClientManager:
except Exception:
log.warning("Refresh failed for MCP server '%s'", name, exc_info=True)
results[name] = ([], [])
# Final sync to clean up templates from servers that are no longer connected
try:
self.sync_prompts_to_storage()
except Exception:
log.warning("Prompt sync after refresh_all failed", exc_info=True)
return results
def refresh_sync(
@@ -319,16 +464,169 @@ class MCPClientManager:
await asyncio.sleep(initial_delay)
while True:
for name in list(self._server_configs):
if self._supports_list_changed.get(name, False):
continue # has push — skip
if name not in self._sessions:
continue # not connected — skip (reconnect on manual refresh)
try:
await self._refresh_server(name)
if not self._supports_list_changed.get(name, False):
await self._refresh_server_tools(name)
if not self._supports_resource_list_changed.get(name, False):
await self._refresh_server_resources(name)
if not self._supports_prompt_list_changed.get(name, False):
await self._refresh_server_prompts(name)
except Exception:
log.warning("Periodic refresh failed for '%s'", name, exc_info=True)
await asyncio.sleep(self._refresh_interval)
# -- resource refresh ----------------------------------------------------
def _rebuild_resources(self) -> None:
"""Rebuild merged ``_resources`` and ``_resource_map`` from per-server state.
Uses copy-on-write: builds new objects, then assigns atomically.
"""
new_resources: list[dict[str, Any]] = []
new_map: dict[str, tuple[str, str]] = {}
for srv_name, srv_resources in self._per_server_resources.items():
for res in srv_resources:
uri: str = res["uri"]
new_resources.append(res)
if res.get("template"):
continue # templates are catalog-only, not directly readable
if uri in new_map:
log.warning(
"Resource URI collision: '%s' from '%s' overrides '%s'",
uri,
srv_name,
new_map[uri][0],
)
new_map[uri] = (srv_name, uri)
# Build template prefix map for URI expansion fallback
new_prefixes: dict[str, tuple[str, str]] = {}
for srv_name, srv_resources in self._per_server_resources.items():
for res in srv_resources:
if res.get("template"):
tmpl_uri = res["uri"]
brace = tmpl_uri.find("{")
prefix = tmpl_uri[:brace] if brace >= 0 else tmpl_uri
if prefix:
if prefix in new_prefixes:
existing_srv, existing_tmpl = new_prefixes[prefix]
if len(tmpl_uri) > len(existing_tmpl):
log.warning(
"Template prefix collision: '%s' from '%s' overrides '%s'"
" (keeping more specific template)",
prefix,
srv_name,
existing_srv,
)
new_prefixes[prefix] = (srv_name, tmpl_uri)
else:
log.warning(
"Template prefix collision: '%s' from '%s' ignored in"
" favor of '%s' (keeping more specific template)",
prefix,
srv_name,
existing_srv,
)
else:
new_prefixes[prefix] = (srv_name, tmpl_uri)
self._resources = new_resources
self._resource_map = new_map
self._template_prefixes = new_prefixes
self._notify_resource_listeners()
async def _refresh_server_resources(self, name: str) -> None:
"""Re-fetch resources for one server."""
if not self._supports_resources.get(name, False):
return
session = self._sessions.get(name)
if session is None:
return
server_resources: list[dict[str, Any]] = []
res_result = await session.list_resources()
for r in res_result.resources:
server_resources.append(
{
"uri": str(r.uri),
"name": r.name or "",
"description": r.description or "",
"mimeType": r.mimeType or "",
"server": name,
}
)
tmpl_result = await session.list_resource_templates()
for t in tmpl_result.resourceTemplates:
server_resources.append(
{
"uri": str(t.uriTemplate),
"name": t.name or "",
"description": t.description or "",
"mimeType": t.mimeType or "",
"server": name,
"template": True,
}
)
self._per_server_resources[name] = server_resources
self._rebuild_resources()
# -- prompt refresh ------------------------------------------------------
def _rebuild_prompts(self) -> None:
"""Rebuild merged ``_prompts`` and ``_prompt_map`` from per-server state.
Uses copy-on-write: builds new objects, then assigns atomically.
"""
new_prompts: list[dict[str, Any]] = []
new_map: dict[str, tuple[str, str]] = {}
for srv_name, srv_prompts in self._per_server_prompts.items():
for prompt in srv_prompts:
prefixed: str = prompt["name"]
new_prompts.append(prompt)
new_map[prefixed] = (srv_name, prompt["original_name"])
self._prompts = new_prompts
self._prompt_map = new_map
self._notify_prompt_listeners()
async def _refresh_server_prompts(self, name: str) -> None:
"""Re-fetch prompts for one server."""
if not self._supports_prompts.get(name, False):
return
session = self._sessions.get(name)
if session is None:
return
server_prompts: list[dict[str, Any]] = []
prompt_result = await session.list_prompts()
for p in prompt_result.prompts:
server_prompts.append(
{
"name": f"mcp__{name}__{p.name}",
"original_name": p.name,
"server": name,
"description": p.description or "",
"arguments": [
{
"name": a.name,
"description": a.description or "",
"required": a.required or False,
}
for a in (p.arguments or [])
],
}
)
self._per_server_prompts[name] = server_prompts
self._rebuild_prompts()
# Sync discovered prompts into governance storage
try:
self.sync_prompts_to_storage()
except Exception:
log.warning("Prompt sync after refresh failed for '%s'", name, exc_info=True)
# -- listener infrastructure ---------------------------------------------
def add_listener(self, callback: Callable[[], None]) -> None:
@@ -342,7 +640,7 @@ class MCPClientManager:
self._listeners.remove(callback)
def _notify_listeners(self) -> None:
"""Invoke all registered listeners (runs on MCP background thread)."""
"""Invoke all registered tool-change listeners."""
with self._listeners_lock:
listeners = list(self._listeners)
for cb in listeners:
@@ -351,6 +649,156 @@ class MCPClientManager:
except Exception:
log.warning("Tool-change listener raised", exc_info=True)
def add_resource_listener(self, callback: Callable[[], None]) -> None:
"""Register a callback invoked when the resource list changes."""
with self._resource_listeners_lock:
self._resource_listeners.append(callback)
def remove_resource_listener(self, callback: Callable[[], None]) -> None:
"""Unregister a resource-change callback."""
with self._resource_listeners_lock, contextlib.suppress(ValueError):
self._resource_listeners.remove(callback)
def _notify_resource_listeners(self) -> None:
"""Invoke all registered resource-change listeners."""
with self._resource_listeners_lock:
listeners = list(self._resource_listeners)
for cb in listeners:
try:
cb()
except Exception:
log.warning("Resource-change listener raised", exc_info=True)
def add_prompt_listener(self, callback: Callable[[], None]) -> None:
"""Register a callback invoked when the prompt list changes."""
with self._prompt_listeners_lock:
self._prompt_listeners.append(callback)
def remove_prompt_listener(self, callback: Callable[[], None]) -> None:
"""Unregister a prompt-change callback."""
with self._prompt_listeners_lock, contextlib.suppress(ValueError):
self._prompt_listeners.remove(callback)
def _notify_prompt_listeners(self) -> None:
"""Invoke all registered prompt-change listeners."""
with self._prompt_listeners_lock:
listeners = list(self._prompt_listeners)
for cb in listeners:
try:
cb()
except Exception:
log.warning("Prompt-change listener raised", exc_info=True)
# -- governance storage sync ---------------------------------------------
def set_storage(self, storage: Any) -> None:
"""Inject governance storage backend for prompt template sync.
If MCP servers are already connected, triggers an immediate sync
so prompts discovered during startup appear in governance storage
(``start()`` completes before ``set_storage()`` is called).
"""
self._storage = storage
if self._connected.is_set():
try:
self.sync_prompts_to_storage()
except Exception:
log.warning("Prompt sync after set_storage failed", exc_info=True)
def sync_prompts_to_storage(self) -> dict[str, Any]:
"""Sync discovered MCP prompts into the prompt_templates governance table.
Returns ``{"added": [...], "removed": [...], "skipped": [...]}``.
Thread-safe: serialized via ``_sync_lock`` to prevent races
between ``set_storage()`` (main thread) and MCP background thread.
"""
if self._storage is None:
return {"added": [], "removed": [], "skipped": []}
with self._sync_lock:
return self._sync_prompts_locked()
def _sync_prompts_locked(self) -> dict[str, Any]:
"""Inner sync logic — must be called under ``_sync_lock``."""
storage = self._storage
added: list[str] = []
removed: list[str] = []
skipped: list[str] = []
# Current MCP prompt names (the prefixed names used as template names)
current_names: set[str] = set()
for prompt in list(self._prompts):
name: str = prompt["name"][:256]
server: str = prompt["server"][:128]
current_names.add(name)
# Build content from description + argument schema
desc = prompt.get("description", "")[:4096]
args_list = prompt.get("arguments", [])
content_parts = [desc] if desc else []
if args_list:
content_parts.append("\nArguments:")
for arg in args_list:
req = " (required)" if arg.get("required") else ""
arg_desc = arg.get("description", "")[:512]
content_parts.append(f" - {arg['name'][:128]}{req}: {arg_desc}")
content = "\n".join(content_parts) if content_parts else name
# Variables = JSON list of argument names
variables = json.dumps([a["name"] for a in args_list])
existing = storage.get_prompt_template_by_name(name)
if existing is not None:
if existing.get("origin") == "manual":
log.info(
"Skipping MCP prompt '%s' — manual template with same name exists", name
)
skipped.append(name)
continue
# Existing MCP template — update content/variables.
# Reset is_default to prevent a compromised MCP server from
# injecting content into a previously admin-promoted default.
storage.update_prompt_template(
existing["template_id"],
content=content,
variables=variables,
is_default=False,
)
else:
# Create new MCP-sourced template
template_id = str(uuid.uuid4())
storage.create_prompt_template(
template_id=template_id,
name=name,
category="mcp",
content=content,
variables=variables,
is_default=False,
org_id="",
created_by="",
origin="mcp",
mcp_server=server,
readonly=True,
)
added.append(name)
# Remove MCP templates whose prompts no longer exist
existing_mcp = storage.list_prompt_templates_by_origin("mcp")
for tpl in existing_mcp:
if tpl["name"] not in current_names:
storage.delete_prompt_template(tpl["template_id"])
removed.append(tpl["name"])
if added or removed:
log.info(
"MCP prompt sync: +%d added, -%d removed, %d skipped",
len(added),
len(removed),
len(skipped),
)
return {"added": added, "removed": removed, "skipped": skipped}
# -- lifecycle (shutdown) ------------------------------------------------
def shutdown(self) -> None:
@@ -371,18 +819,62 @@ class MCPClientManager:
if self._thread:
self._thread.join(timeout=5)
# Clear all state
self._sessions.clear()
self._tools = []
self._tool_map = {}
self._per_server_tools.clear()
self._supports_list_changed.clear()
self._resources = []
self._resource_map = {}
self._template_prefixes = {}
self._per_server_resources.clear()
self._supports_resources.clear()
self._supports_resource_list_changed.clear()
self._prompts = []
self._prompt_map = {}
self._per_server_prompts.clear()
self._supports_prompts.clear()
self._supports_prompt_list_changed.clear()
# Clear listener lists to release callback references
self._listeners.clear()
self._resource_listeners.clear()
self._prompt_listeners.clear()
log.info("MCP client shut down")
# -- query methods -------------------------------------------------------
def get_tools(self) -> list[dict[str, Any]]:
"""Return MCP tools in OpenAI function-calling format."""
return list(self._tools)
return [dict(t) for t in self._tools]
def get_resources(self) -> list[dict[str, Any]]:
"""Return discovered MCP resources (shallow-copied dicts)."""
return [dict(r) for r in self._resources]
def get_prompts(self) -> list[dict[str, Any]]:
"""Return discovered MCP prompts (shallow-copied dicts)."""
return [dict(p) for p in self._prompts]
@property
def resource_count(self) -> int:
"""Number of discovered resources (no allocation)."""
return len(self._resources)
@property
def prompt_count(self) -> int:
"""Number of discovered prompts (no allocation)."""
return len(self._prompts)
def is_mcp_tool(self, func_name: str) -> bool:
"""Check whether *func_name* belongs to an MCP server."""
return func_name in self._tool_map
def is_mcp_prompt(self, name: str) -> bool:
"""Check whether *name* is a known MCP prompt."""
return name in self._prompt_map
@property
def server_count(self) -> int:
return len(self._sessions)
@@ -417,7 +909,10 @@ class MCPClientManager:
future = asyncio.run_coroutine_threadsafe(
session.call_tool(original_name, arguments), self._loop
)
result = future.result(timeout=timeout)
try:
result = future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
raise TimeoutError(f"MCP tool call timed out after {timeout}s") from None
# Extract text from the content array
texts: list[str] = []
@@ -435,6 +930,94 @@ class MCPClientManager:
output = f"Error: {output}"
return output
# -- resource read -------------------------------------------------------
def _match_template(self, uri: str) -> tuple[str, str] | None:
"""Find the longest matching template prefix for an expanded URI.
Returns ``(server_name, template_uri)`` or *None* if no match.
The match uses the longest static prefix stored in
``_template_prefixes`` (the portion of each template URI before
the first ``{``), with simple ``startswith`` matching.
"""
best: tuple[str, str] | None = None
best_len = 0
for prefix, mapping in self._template_prefixes.items():
if uri.startswith(prefix) and len(prefix) > best_len:
best = mapping
best_len = len(prefix)
return best
def read_resource_sync(self, uri: str, timeout: int = 120) -> str:
"""Read a resource by URI synchronously (blocks the calling thread).
Returns text content for ``TextResourceContents``, or base64 data
for ``BlobResourceContents``.
"""
mapping = self._resource_map.get(uri)
if mapping is None:
# Fall back to template prefix matching for expanded URIs
mapping = self._match_template(uri)
if mapping is None:
raise ValueError(f"Unknown MCP resource: {uri}")
server_name, _ = mapping
session = self._sessions.get(server_name)
if session is None:
raise RuntimeError(f"MCP server '{server_name}' is not connected")
assert self._loop is not None
future = asyncio.run_coroutine_threadsafe(session.read_resource(uri), self._loop)
try:
result = future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
raise TimeoutError(f"MCP resource read timed out after {timeout}s") from None
parts: list[str] = []
for item in result.contents:
if hasattr(item, "text"):
parts.append(item.text)
elif hasattr(item, "blob"):
parts.append(item.blob)
else:
parts.append(str(item))
return "\n".join(parts) if parts else "(empty resource)"
# -- prompt invocation ---------------------------------------------------
def get_prompt_sync(
self,
prefixed_name: str,
arguments: dict[str, str] | None = None,
timeout: int = 30,
) -> list[dict[str, Any]]:
"""Invoke an MCP prompt synchronously and return expanded messages.
Returns a list of ``{role: str, content: str}`` dicts.
"""
mapping = self._prompt_map.get(prefixed_name)
if mapping is None:
raise ValueError(f"Unknown MCP prompt: {prefixed_name}")
server_name, original_name = mapping
session = self._sessions.get(server_name)
if session is None:
raise RuntimeError(f"MCP server '{server_name}' is not connected")
assert self._loop is not None
future = asyncio.run_coroutine_threadsafe(
session.get_prompt(original_name, arguments=arguments), self._loop
)
try:
result = future.result(timeout=timeout)
except concurrent.futures.TimeoutError:
raise TimeoutError(f"MCP prompt retrieval timed out after {timeout}s") from None
messages: list[dict[str, Any]] = []
for msg in result.messages:
content = msg.content
text = content.text if hasattr(content, "text") else str(content)
messages.append({"role": msg.role, "content": text})
return messages
# ---------------------------------------------------------------------------
# Config loading
+19
View File
@@ -143,6 +143,25 @@ def load_workstream_config(ws_id: str) -> dict[str, str]:
return {}
# -- Prompt templates ---------------------------------------------------------
def list_default_templates(org_id: str = "") -> list[dict[str, Any]]:
"""Return all templates where is_default=True, ordered by name."""
try:
return get_storage().list_default_templates(org_id)
except Exception:
return []
def get_prompt_template_by_name(name: str) -> dict[str, Any] | None:
"""Lookup prompt template by name."""
try:
return get_storage().get_prompt_template_by_name(name)
except Exception:
return None
# -- Workstream metadata ------------------------------------------------------
+19
View File
@@ -102,6 +102,7 @@ class MetricsCollector:
workstream_states: dict[str, int],
total_workstreams: int,
workstream_metrics: list[dict[str, Any]] | None = None,
mcp_info: dict[str, int] | None = None,
) -> str:
"""Return Prometheus text exposition format (v0.0.4)."""
lines: list[str] = []
@@ -322,6 +323,24 @@ class MetricsCollector:
f"turnstone_workstream_context_ratio{lstr} {_fmt_value(wm['context_ratio'])}"
)
# MCP gauges (optional)
if mcp_info:
gauge(
"turnstone_mcp_servers",
"Number of connected MCP servers",
mcp_info.get("servers", 0),
)
gauge(
"turnstone_mcp_resources",
"Number of MCP resources available",
mcp_info.get("resources", 0),
)
gauge(
"turnstone_mcp_prompts",
"Number of MCP prompts available",
mcp_info.get("prompts", 0),
)
lines.append("") # trailing newline
return "\n".join(lines)
+572 -32
View File
@@ -24,6 +24,7 @@ import textwrap
import threading
import time
import uuid
from html import escape as _html_escape
from typing import TYPE_CHECKING, Any, Protocol
import httpx
@@ -34,7 +35,9 @@ from turnstone.core.log import get_logger
from turnstone.core.memory import (
delete_memory,
delete_workstream,
get_prompt_template_by_name,
get_workstream_display_name,
list_default_templates,
list_workstreams_with_history,
load_memories,
load_messages,
@@ -104,6 +107,26 @@ _IMAGE_EXTENSIONS: frozenset[str] = frozenset(
# 4 MB raw → ~5.3 MB base64, safely under Anthropic's per-block limit
_IMAGE_SIZE_CAP: int = 4 * 1024 * 1024
# Upper bound on total prompt template content injected into system messages
_MAX_TEMPLATE_CONTENT: int = 32768
_TEMPLATE_VAR_RE = re.compile(r"\{\{(\w+)\}\}")
def _render_template(content: str, context: dict[str, str]) -> str:
"""Replace ``{{variable}}`` placeholders in a single pass.
Unresolvable placeholders are kept as-is. Single-pass avoids
cross-variable injection (e.g. a model name containing ``{{ws_id}}``).
"""
def _replace(m: re.Match[str]) -> str:
return context.get(m.group(1), m.group(0))
return _TEMPLATE_VAR_RE.sub(_replace, content)
# ---------------------------------------------------------------------------
# SessionUI protocol — the contract every frontend must implement
# ---------------------------------------------------------------------------
@@ -193,6 +216,7 @@ class ChatSession:
tool_search: str = "auto",
tool_search_threshold: int = 20,
tool_search_max_results: int = 5,
template: str | None = None,
):
self.client = client
self.model = model
@@ -246,6 +270,8 @@ class ChatSession:
# MCP tool integration: merge external tools with built-in
self._mcp_client = mcp_client
self._mcp_refresh_cb: Any = None # Callable | None (avoid import)
self._mcp_resource_cb: Any = None
self._mcp_prompt_cb: Any = None
if mcp_client:
mcp_tools = mcp_client.get_tools()
self._tools = merge_mcp_tools(TOOLS, mcp_tools)
@@ -254,6 +280,12 @@ class ChatSession:
# Register for tool-change notifications from MCP servers
self._mcp_refresh_cb = self._on_mcp_tools_changed
mcp_client.add_listener(self._mcp_refresh_cb)
# Register for resource-change notifications
self._mcp_resource_cb = self._on_mcp_resources_changed
mcp_client.add_resource_listener(self._mcp_resource_cb)
# Register for prompt-change notifications
self._mcp_prompt_cb = self._on_mcp_prompts_changed
mcp_client.add_prompt_listener(self._mcp_prompt_cb)
else:
self._tools = TOOLS
self._task_tools = TASK_AGENT_TOOLS
@@ -272,6 +304,10 @@ class ChatSession:
threshold=tool_search_threshold,
max_results=tool_search_max_results,
)
# Prompt template: explicit name overrides is_default templates
self._template_name: str | None = template
self._template_content: str | None = None
self._load_templates()
self._init_system_messages()
self._save_config()
@@ -305,9 +341,39 @@ class ChatSession:
"max_tokens": str(self.max_tokens),
"instructions": self.instructions or "",
"creative_mode": str(self.creative_mode),
"template": self._template_name or "",
},
)
def _load_templates(self) -> None:
"""Load prompt templates from storage. Called once at init and on /template."""
context = {
"model": self.model,
"ws_id": self._ws_id,
"node_id": self._node_id or "",
}
if self._template_name:
tpl = get_prompt_template_by_name(self._template_name)
if tpl:
self._template_content = _render_template(tpl["content"], context)
else:
log.warning("prompt_template.not_found", name=self._template_name)
self._template_content = None
else:
defaults = list_default_templates()
if defaults:
parts = [_render_template(t["content"], context) for t in defaults]
self._template_content = "\n\n".join(parts)
else:
self._template_content = None
def set_template(self, name: str | None) -> None:
"""Set or clear the active prompt template."""
self._template_name = name
self._load_templates()
self._init_system_messages()
self._save_config()
# -- MCP tool refresh ----------------------------------------------------
def _on_mcp_tools_changed(self) -> None:
@@ -333,6 +399,22 @@ class ChatSession:
self._agent_tools = merge_mcp_tools(AGENT_TOOLS, mcp_tools)
self._rebuild_tool_search()
def _on_mcp_resources_changed(self) -> None:
"""Callback from MCPClientManager when the resource list changes.
Rebuilds the system message to update the resource catalog.
Called on the MCP background thread.
"""
self._init_system_messages()
def _on_mcp_prompts_changed(self) -> None:
"""Callback from MCPClientManager when the prompt list changes.
Rebuilds the system message to update the prompt catalog.
Called on the MCP background thread.
"""
self._init_system_messages()
def _rebuild_tool_search(self) -> None:
"""Reconstruct ToolSearchManager, preserving expanded tools."""
old_expanded = self._tool_search.get_expanded_names() if self._tool_search else []
@@ -375,6 +457,12 @@ class ChatSession:
if self._mcp_client and self._mcp_refresh_cb:
self._mcp_client.remove_listener(self._mcp_refresh_cb)
self._mcp_refresh_cb = None
if self._mcp_client and self._mcp_resource_cb:
self._mcp_client.remove_resource_listener(self._mcp_resource_cb)
self._mcp_resource_cb = None
if self._mcp_client and self._mcp_prompt_cb:
self._mcp_client.remove_prompt_listener(self._mcp_prompt_cb)
self._mcp_prompt_cb = None
if self._watch_runner:
self._watch_runner.remove_dispatch_fn(self._ws_id)
@@ -512,6 +600,9 @@ class ChatSession:
self.instructions = config["instructions"] or None
if "creative_mode" in config:
self.creative_mode = config["creative_mode"] == "True"
if "template" in config:
self._template_name = config["template"] or None
self._load_templates()
self._init_system_messages()
return True
@@ -521,8 +612,12 @@ class ChatSession:
Developer message contains tool patterns (or creative writing
instructions when creative_mode is on), plus any user-supplied
instructions and memory reminders.
Uses copy-on-write: builds new lists locally, then assigns
atomically so concurrent readers (e.g. background thread
callbacks) never see a partially-built system message.
"""
self.system_messages: list[dict[str, Any]] = []
new_system_messages: list[dict[str, Any]] = []
# -- Chat template kwargs --
self._chat_template_kwargs_base: dict[str, Any] = {
@@ -583,6 +678,55 @@ class ChatSession:
"\n\nAdditional tools are available via tool_search. "
"Use it when you need a capability not in your current tool set."
)
# MCP resource catalog (lets the model know what's available for read_resource)
if self._mcp_client:
all_resources = self._mcp_client.get_resources()
concrete = [r for r in all_resources if not r.get("template")]
templates = [r for r in all_resources if r.get("template")]
if concrete or templates:
lines = ["\n<mcp-resources>"]
for r in concrete[:50]:
safe_uri = _html_escape(r["uri"])
desc = r.get("description", "")
if desc:
desc = f" {_html_escape(desc[:100])}"
lines.append(f" {safe_uri}{desc}")
if templates:
lines.append("")
lines.append("Resource templates (construct a URI and use read_resource):")
for t in templates[:20]:
safe_uri = _html_escape(t["uri"])
desc = t.get("description", "")
if desc:
desc = f" {_html_escape(desc[:100])}"
lines.append(f" {safe_uri}{desc}")
lines.append("</mcp-resources>")
lines.append("Use read_resource(uri='...') to access the resources listed above.")
dev_parts.append("\n".join(lines))
# MCP prompt catalog (lets the model know what's available for use_prompt)
if self._mcp_client:
prompts = self._mcp_client.get_prompts()
if prompts:
lines = ["<mcp-prompts>"]
for p in prompts[:30]:
# Names/args are NOT escaped — model must use exact strings
# in use_prompt(). Only description (display-only) is escaped.
arg_names = ", ".join(a["name"] for a in p.get("arguments", []))
desc = _html_escape(p.get("description", "")[:100])
lines.append(f" {p['name']}({arg_names}) {desc}")
lines.append("</mcp-prompts>")
lines.append(
"Use use_prompt(name='...', arguments={...}) "
"to invoke the prompts listed above."
)
dev_parts.append("\n".join(lines))
if self._template_content:
tpl = self._template_content
if len(tpl) > _MAX_TEMPLATE_CONTENT:
log.warning("template_content.truncated", length=len(tpl))
tpl = tpl[:_MAX_TEMPLATE_CONTENT]
dev_parts.append("")
dev_parts.append(tpl)
if self.instructions:
dev_parts.append("")
dev_parts.append(self.instructions)
@@ -593,9 +737,11 @@ class ChatSession:
f"REMINDER: You currently have {len(memories)} memories stored. "
"Use recall to see them."
)
self.system_messages.append({"role": "system", "content": "\n".join(dev_parts)})
new_system_messages.append({"role": "system", "content": "\n".join(dev_parts)})
# Atomic swap — readers see either old or new, never partial
self.system_messages = new_system_messages
# Agent prefix: system + developer only (no memories)
self._agent_system_messages = list(self.system_messages)
self._agent_system_messages = list(new_system_messages)
def _full_messages(self) -> list[dict[str, Any]]:
"""System messages + conversation history."""
@@ -1111,6 +1257,11 @@ class ChatSession:
# Handle tool call deltas
if chunk.tool_call_deltas:
_stop_spinner_once()
# Flush any buffered content — model has moved to tool calls,
# so pending text cannot be a partial <think> tag.
if pending:
_flush_text(pending, in_think)
pending = ""
# Close reasoning if transitioning from reasoning
if in_think:
in_think = False
@@ -1551,28 +1702,73 @@ class ChatSession:
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
results = list(pool.map(run_one, items))
# Post-plan gate: prompt user on main thread after plan completes
# Post-plan gate: iterative review loop. When the user gives
# feedback the plan agent re-runs and the revised plan is shown
# again, up to _MAX_PLAN_REFINEMENTS rounds.
for i, item in enumerate(items):
if (
item.get("func_name") == "create_plan"
and not item.get("error")
and not item.get("denied")
and not self.auto_approve
):
cid, output = results[i]
assert isinstance(output, str) # plan always returns text
# Let the UI present the plan for review
self._emit_state("attention")
resp = self.ui.on_plan_review(output)
self._emit_state("running")
if resp.lower() in ("n", "no", "reject"):
output += (
"\n\n---\nUser REJECTED this plan. Do not proceed "
"with implementation. Ask the user what they want instead."
)
elif resp:
output += f"\n\n---\nUser feedback on this plan: {resp}"
results[i] = (cid, output)
if item.get("func_name") != "create_plan" or item.get("error") or item.get("denied"):
continue
cid, output = results[i]
assert isinstance(output, str) # plan always returns text
plan_path = f".plan-{self._ws_id}.md"
if not self.auto_approve:
original_goal = item.get("prompt", "")
refinement_round = 0
while True:
self._emit_state("attention")
resp = self.ui.on_plan_review(output)
self._emit_state("running")
if resp.lower() in ("n", "no", "reject"):
output += (
"\n\n---\nUser REJECTED this plan. Do not "
"proceed with implementation. Ask the user "
"what they want instead."
)
break
elif not resp:
break # empty response = approve
elif refinement_round >= self._MAX_PLAN_REFINEMENTS:
self.ui.on_info("[plan] max refinement rounds reached")
break
else:
# Re-run plan agent with user feedback.
# Strip any internal warning prefix so the
# agent sees the raw plan content.
raw = output
_warn = "[Warning: plan may be incomplete or poorly structured]\n\n"
if raw.startswith(_warn):
raw = raw[len(_warn) :]
try:
output = self._refine_plan(
raw,
original_goal,
resp,
)
refinement_round += 1
except (KeyboardInterrupt, GenerationCancelled):
output += "\n\n---\n(plan refinement interrupted)"
break
except Exception as e:
self.ui.on_info(f"[plan refinement error] {e}")
output += f"\n\n---\nUser feedback: {resp}"
break
# Loop continues → show revised plan to user
# Write final version to disk (overwrites initial write)
try:
with open(plan_path, "w") as f:
f.write(output)
except OSError:
pass
# Always include file path in the tool result so the
# outer model knows where the plan lives on disk.
output += f"\n\n---\nPlan saved to `{plan_path}`"
results[i] = (cid, output)
return results, user_feedback
@@ -1591,11 +1787,13 @@ class ChatSession:
"command",
"code",
"content",
"name",
"page",
"path",
"pattern",
"prompt",
"query",
"uri",
"url",
):
m = re.search(rf'"{key}"\s*:\s*"((?:[^"\\]|\\.)*)"', raw_args)
@@ -1640,6 +1838,8 @@ class ChatSession:
"forget": self._prepare_forget,
"notify": self._prepare_notify,
"watch": self._prepare_watch,
"read_resource": self._prepare_read_resource,
"use_prompt": self._prepare_use_prompt,
}
preparer = preparers.get(func_name)
if not preparer:
@@ -2296,7 +2496,7 @@ class ChatSession:
"header": f"\u2699 mcp:{display}",
"preview": f"{DIM}{preview}{RESET}",
"needs_approval": True,
"approval_label": "mcp_tool",
"approval_label": func_name,
"execute": self._exec_mcp_tool,
"mcp_func_name": func_name,
"mcp_args": args,
@@ -2322,6 +2522,160 @@ class ChatSession:
self.ui.on_tool_result(call_id, func_name, output)
return call_id, output
@staticmethod
def _normalize_resource_uri(uri: str) -> str:
"""Normalize a resource URI for policy matching.
Decodes percent-encoded path segments (e.g. ``%2e%2e`` ``..``)
then resolves ``..`` to prevent traversal bypasses where
``file:///docs/%2e%2e/etc/passwd`` would match a policy
allowing ``mcp_resource__file:///docs/*``.
"""
import posixpath
from urllib.parse import quote, unquote, urlparse, urlunparse
parsed = urlparse(uri)
if parsed.path:
decoded = unquote(parsed.path)
normalized = posixpath.normpath(decoded)
if parsed.path.startswith("/") and not normalized.startswith("/"):
normalized = "/" + normalized
parsed = parsed._replace(path=quote(normalized, safe="/"))
return urlunparse(parsed)
def _prepare_read_resource(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
"""Prepare an MCP resource read."""
uri = args.get("uri", "")
if not uri:
return {
"call_id": call_id,
"func_name": "read_resource",
"header": "\u2717 read_resource: missing uri",
"preview": "",
"needs_approval": False,
"error": "Missing required parameter: uri",
}
if not self._mcp_client:
return {
"call_id": call_id,
"func_name": "read_resource",
"header": "\u2717 read_resource: no MCP servers",
"preview": "",
"needs_approval": False,
"error": "No MCP servers configured",
}
return {
"call_id": call_id,
"func_name": "read_resource",
"header": "\u2699 read_resource",
"preview": f"{DIM} uri: {uri}{RESET}",
"needs_approval": True,
"approval_label": f"mcp_resource__{self._normalize_resource_uri(uri)}",
"execute": self._exec_read_resource,
"resource_uri": uri,
}
def _exec_read_resource(self, item: dict[str, Any]) -> tuple[str, str]:
"""Read an MCP resource by URI."""
call_id: str = item["call_id"]
uri: str = item["resource_uri"]
assert self._mcp_client is not None
try:
output = self._mcp_client.read_resource_sync(uri, timeout=self.tool_timeout)
except TimeoutError:
output = f"MCP resource read timed out after {self.tool_timeout}s"
self.ui.on_error(output)
except Exception:
log.warning("MCP resource read failed for %s", uri, exc_info=True)
output = "MCP resource error: failed to read resource"
self.ui.on_error(output)
output = self._truncate_output(output)
self.ui.on_tool_result(call_id, "read_resource", output)
return call_id, output
def _prepare_use_prompt(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
"""Prepare an MCP prompt invocation."""
name = args.get("name", "")
if not name:
return {
"call_id": call_id,
"func_name": "use_prompt",
"header": "\u2717 use_prompt: missing name",
"preview": "",
"needs_approval": False,
"error": "Missing required parameter: name",
}
if not self._mcp_client:
return {
"call_id": call_id,
"func_name": "use_prompt",
"header": "\u2717 use_prompt: no MCP servers",
"preview": "",
"needs_approval": False,
"error": "No MCP servers configured",
}
if not self._mcp_client.is_mcp_prompt(name):
return {
"call_id": call_id,
"func_name": "use_prompt",
"header": f"\u2717 use_prompt: unknown prompt '{name}'",
"preview": "",
"needs_approval": False,
"error": f"Unknown MCP prompt: {name}",
}
raw_arguments = args.get("arguments") or {}
if not isinstance(raw_arguments, dict):
return {
"call_id": call_id,
"func_name": "use_prompt",
"header": "\u2717 use_prompt: arguments must be an object",
"preview": "",
"needs_approval": False,
"error": "arguments must be a JSON object with string values",
}
arguments = {str(k): str(v) for k, v in raw_arguments.items()}
preview_parts = [f" {DIM}name: {name}"]
if arguments:
preview_parts.append(f" arguments: {arguments}")
preview_parts.append(RESET)
return {
"call_id": call_id,
"func_name": "use_prompt",
"header": "\u2699 use_prompt",
"preview": "\n".join(preview_parts),
"needs_approval": True,
"approval_label": name,
"execute": self._exec_use_prompt,
"prompt_name": name,
"prompt_arguments": arguments,
}
def _exec_use_prompt(self, item: dict[str, Any]) -> tuple[str, str]:
"""Invoke an MCP prompt and return expanded messages."""
call_id: str = item["call_id"]
name: str = item["prompt_name"]
arguments: dict[str, str] = item["prompt_arguments"]
assert self._mcp_client is not None
try:
messages = self._mcp_client.get_prompt_sync(
name, arguments or None, timeout=self.tool_timeout
)
output = "\n\n".join(f"[{m['role']}]: {m['content']}" for m in messages)
except TimeoutError:
output = f"MCP prompt timed out after {self.tool_timeout}s"
self.ui.on_error(output)
except Exception:
log.warning("MCP prompt invocation failed for %s", name, exc_info=True)
output = "MCP prompt error: failed to invoke prompt"
self.ui.on_error(output)
output = self._truncate_output(output)
self.ui.on_tool_result(call_id, "use_prompt", output)
return call_id, output
# -- Execute methods (do the work, report output via UI) -------------------
def _exec_bash(self, item: dict[str, Any]) -> tuple[str, str]:
@@ -2811,6 +3165,61 @@ class ChatSession:
"and functions in every step."
)
_MIN_PLAN_LENGTH = 100
_PLAN_REQUIRED_SECTIONS = ("## goal", "## current state", "## plan", "## risks")
_MIN_PLAN_SECTIONS = 2
_MAX_PLAN_REFINEMENTS = 5
@staticmethod
def _validate_plan(content: str, goal: str) -> tuple[bool, list[str]]:
"""Check if plan output meets minimum quality bar.
Returns ``(valid, issues)`` where *issues* is a list of
human-readable problem descriptions (empty when valid).
"""
issues: list[str] = []
stripped = content.strip()
stripped_lower = stripped.lower()
# 1. Minimum length
if len(stripped) < ChatSession._MIN_PLAN_LENGTH:
issues.append(
f"too short ({len(stripped)} chars, minimum {ChatSession._MIN_PLAN_LENGTH})"
)
# 2. Section structure
found_sections = sum(
1 for section in ChatSession._PLAN_REQUIRED_SECTIONS if section in stripped_lower
)
if found_sections < ChatSession._MIN_PLAN_SECTIONS:
issues.append(
f"missing plan sections (found {found_sections}/"
f"{len(ChatSession._PLAN_REQUIRED_SECTIONS)}, "
f"need at least {ChatSession._MIN_PLAN_SECTIONS})"
)
# 3. Echo detection: plan is basically just the goal repeated
goal_stripped = goal.strip().lower()
if (
goal_stripped
and len(stripped) < len(goal_stripped) * 2
and goal_stripped in stripped_lower
):
issues.append("plan appears to echo the goal without elaboration")
# 4. Refusal detection
refusal_starts = (
"i cannot",
"i'm sorry",
"i am sorry",
"error:",
"i can't",
)
if any(stripped_lower.startswith(r) for r in refusal_starts):
issues.append("plan appears to be a refusal or error")
return (len(issues) == 0, issues)
def _exec_plan(self, item: dict[str, Any]) -> tuple[str, str]:
"""Run a planning agent and write the result to .plan-<ws_id>.md."""
call_id, prompt = item["call_id"], item["prompt"]
@@ -2853,6 +3262,41 @@ class ChatSession:
self.ui.on_info(f"[plan error] {e}")
return call_id, f"Plan error: {e}"
# Validate plan quality — retry once with coaching on failure
valid, issues = self._validate_plan(content, prompt)
if not valid:
self.ui.on_info(f"[plan] quality issues: {', '.join(issues)}")
preview = content[:200] + ("..." if len(content) > 200 else "")
coaching = (
"Your previous response did not follow the required plan "
"format. A valid plan should include at least two of "
"these markdown sections:\n"
"## Goal (1-2 sentences)\n"
"## Current State (files/line numbers found)\n"
"## Plan (numbered steps with file names and functions)\n"
"## Risks (edge cases and unknowns)\n\n"
f'Your previous response was: "{preview}"\n\n'
"Please try again. Explore the codebase first, then write "
"the plan."
)
agent_messages.append({"role": "user", "content": coaching})
try:
content = self._run_agent(
agent_messages,
label="plan",
reasoning_effort="high",
)
except (KeyboardInterrupt, GenerationCancelled):
return call_id, "(plan interrupted by user)"
except Exception as e:
self.ui.on_info(f"[plan retry error] {e}")
return call_id, f"Plan error: {e}"
valid2, issues2 = self._validate_plan(content, prompt)
if not valid2:
self.ui.on_info(f"[plan] still has issues after retry: {', '.join(issues2)}")
content = "[Warning: plan may be incomplete or poorly structured]\n\n" + content
# Write to file separately — always return content even if write fails
try:
with open(plan_path, "w") as f:
@@ -2863,6 +3307,60 @@ class ChatSession:
return call_id, content
def _refine_plan(
self,
original_content: str,
original_goal: str,
feedback: str,
) -> str:
"""Re-run the plan agent incorporating user feedback."""
tc_id = f"plan_refine_{uuid.uuid4().hex[:8]}"
agent_messages: list[dict[str, Any]] = [
{"role": "system", "content": self._PLAN_IDENTITY},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": tc_id,
"type": "function",
"function": {
"name": "create_plan",
"arguments": json.dumps({"goal": original_goal}),
},
}
],
},
{
"role": "tool",
"tool_call_id": tc_id,
"content": original_content,
},
{
"role": "user",
"content": (
"The user reviewed this plan and provided feedback:\n\n"
f"{feedback}\n\n"
"Please revise the plan accordingly. Keep the same "
"format (## Goal, ## Current State, ## Plan, ## Risks) "
"and address the feedback."
),
},
]
self.ui.on_info("[plan] revising based on feedback...")
content = self._run_agent(
agent_messages,
label="plan",
reasoning_effort="high",
)
valid, issues = self._validate_plan(content, original_goal)
if not valid:
self.ui.on_info(f"[plan] revised plan has issues: {', '.join(issues)}")
return content
def _exec_remember(self, item: dict[str, Any]) -> tuple[str, str]:
"""Save a persistent memory."""
call_id, key, value = item["call_id"], item["key"], item["value"]
@@ -3674,6 +4172,25 @@ class ChatSession:
self._save_config()
self.ui.on_info("Instructions updated.")
elif cmd == "/template":
if not arg:
if self._template_name:
self.ui.on_info(f"Active template: {self._template_name}")
else:
self.ui.on_info(
"Using default templates. Usage: /template <name> or /template clear"
)
elif arg.strip().lower() == "clear":
self.set_template(None)
self.ui.on_info("Template cleared; using defaults.")
else:
tpl = get_prompt_template_by_name(arg.strip())
if tpl:
self.set_template(tpl["name"])
self.ui.on_info(f"Template set: {tpl['name']}")
else:
self.ui.on_error(f"Template not found: {arg.strip()}")
elif cmd == "/clear":
self.messages.clear()
self._read_files.clear()
@@ -3868,15 +4385,37 @@ class ChatSession:
self._handle_mcp_refresh(arg)
else:
tools = self._mcp_client.get_tools()
if not tools:
self.ui.on_info("MCP client connected but no tools available.")
else:
lines = [f"MCP tools ({len(tools)}):"]
resources = self._mcp_client.get_resources()
prompts = self._mcp_client.get_prompts()
mcp_lines = []
if tools:
mcp_lines.append(f"MCP tools ({len(tools)}):")
for t in tools:
name = t["function"]["name"]
desc = t["function"].get("description", "")[:80]
lines.append(f" {name} {dim(desc)}")
self.ui.on_info("\n".join(lines))
mcp_lines.append(f" {name} {dim(desc)}")
if resources:
if mcp_lines:
mcp_lines.append("")
mcp_lines.append(f"MCP resources ({len(resources)}):")
for r in resources:
prefix = "[template] " if r.get("template") else ""
desc = r.get("description", "")[:80]
mcp_lines.append(f" {prefix}{r['uri']} {dim(desc)}")
if prompts:
if mcp_lines:
mcp_lines.append("")
mcp_lines.append(f"MCP prompts ({len(prompts)}):")
for p in prompts:
arg_names = ", ".join(a["name"] for a in p.get("arguments", []))
desc = p.get("description", "")[:60]
mcp_lines.append(f" {p['name']}({arg_names}) {dim(desc)}")
if not mcp_lines:
self.ui.on_info(
"MCP client connected but no tools, resources, or prompts available."
)
else:
self.ui.on_info("\n".join(mcp_lines))
elif cmd == "/help":
self.ui.on_info(
@@ -3884,6 +4423,7 @@ class ChatSession:
[
"── Slash Commands ─────────────────────────────────────",
" /instructions <text> Set developer instructions",
" /template [name|clear] Set/show/clear prompt template",
" /clear Clear context (workstream preserved in database)",
" /new Start a new workstream (old one stays resumable)",
"",
@@ -3900,7 +4440,7 @@ class ChatSession:
" /reason [low|med|high] Set/show reasoning effort",
" /creative Toggle creative writing mode (no tools)",
" /debug Toggle raw SSE delta logging",
" /mcp [refresh [server]] List or refresh MCP tools",
" /mcp [refresh [server]] List or refresh MCP tools, resources, and prompts",
" /help Show this help",
" /exit Exit (also: Ctrl+D)",
"────────────────────────────────────────────────────────",
+42 -17
View File
@@ -863,6 +863,7 @@ class PostgreSQLBackend:
auto_approve_tools: list[str],
created_by: str,
next_run: str,
template: str = "",
) -> None:
from sqlalchemy.dialects import postgresql
@@ -884,6 +885,7 @@ class PostgreSQLBackend:
initial_message=initial_message,
auto_approve=1 if auto_approve else 0,
auto_approve_tools=",".join(auto_approve_tools),
template=template,
enabled=1,
created_by=created_by,
next_run=next_run,
@@ -926,6 +928,7 @@ class PostgreSQLBackend:
"initial_message",
"auto_approve",
"auto_approve_tools",
"template",
"enabled",
"last_run",
"next_run",
@@ -1363,21 +1366,7 @@ class PostgreSQLBackend:
.select_from(user_roles.join(roles, user_roles.c.role_id == roles.c.role_id))
.where(user_roles.c.user_id == user_id)
).fetchall()
return [
{
"role_id": r[0],
"name": r[1],
"display_name": r[2],
"permissions": r[3],
"builtin": bool(r[4]),
"org_id": r[5],
"created": r[6],
"updated": r[7],
"assigned_by": r[8],
"assignment_created": r[9],
}
for r in rows
]
return [_row_to_dict(r, "builtin") for r in rows]
def get_user_permissions(self, user_id: str) -> set[str]:
with self._engine.connect() as conn:
@@ -1526,6 +1515,9 @@ class PostgreSQLBackend:
is_default: bool = False,
org_id: str = "",
created_by: str = "",
origin: str = "manual",
mcp_server: str = "",
readonly: bool = False,
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
@@ -1540,6 +1532,9 @@ class PostgreSQLBackend:
"is_default": 1 if is_default else 0,
"org_id": org_id,
"created_by": created_by,
"origin": origin,
"mcp_server": mcp_server,
"readonly": 1 if readonly else 0,
"created": now,
"updated": now,
},
@@ -1552,7 +1547,16 @@ class PostgreSQLBackend:
sa.select(prompt_templates).where(prompt_templates.c.template_id == template_id)
).fetchone()
if row:
return _row_to_dict(row, "is_default")
return _row_to_dict(row, "is_default", "readonly")
return None
def get_prompt_template_by_name(self, name: str) -> dict[str, Any] | None:
with self._engine.connect() as conn:
row = conn.execute(
sa.select(prompt_templates).where(prompt_templates.c.name == name)
).fetchone()
if row:
return _row_to_dict(row, "is_default", "readonly")
return None
def list_prompt_templates(self, org_id: str = "") -> list[dict[str, Any]]:
@@ -1561,7 +1565,28 @@ class PostgreSQLBackend:
if org_id:
q = q.where(prompt_templates.c.org_id == org_id)
rows = conn.execute(q).fetchall()
return [_row_to_dict(r, "is_default") for r in rows]
return [_row_to_dict(r, "is_default", "readonly") for r in rows]
def list_default_templates(self, org_id: str = "") -> list[dict[str, Any]]:
with self._engine.connect() as conn:
q = (
sa.select(prompt_templates)
.where(prompt_templates.c.is_default == 1)
.order_by(prompt_templates.c.name)
)
if org_id:
q = q.where(prompt_templates.c.org_id == org_id)
rows = conn.execute(q).fetchall()
return [_row_to_dict(r, "is_default", "readonly") for r in rows]
def list_prompt_templates_by_origin(self, origin: str) -> list[dict[str, Any]]:
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(prompt_templates)
.where(prompt_templates.c.origin == origin)
.order_by(prompt_templates.c.name)
).fetchall()
return [_row_to_dict(r, "is_default", "readonly") for r in rows]
def update_prompt_template(self, template_id: str, **fields: Any) -> bool:
dropped = set(fields) - _TEMPLATE_MUTABLE
+16
View File
@@ -247,6 +247,7 @@ class StorageBackend(Protocol):
auto_approve_tools: list[str],
created_by: str,
next_run: str,
template: str = "",
) -> None:
"""Create a scheduled task. No-op if task_id already exists."""
...
@@ -471,6 +472,9 @@ class StorageBackend(Protocol):
is_default: bool,
org_id: str,
created_by: str,
origin: str = "manual",
mcp_server: str = "",
readonly: bool = False,
) -> None:
"""Create a prompt template."""
...
@@ -479,10 +483,22 @@ class StorageBackend(Protocol):
"""Return prompt template dict or None."""
...
def get_prompt_template_by_name(self, name: str) -> dict[str, Any] | None:
"""Lookup prompt template by name. Returns same dict as get_prompt_template or None."""
...
def list_prompt_templates(self, org_id: str = "") -> list[dict[str, Any]]:
"""Return all prompt templates ordered by name."""
...
def list_default_templates(self, org_id: str = "") -> list[dict[str, Any]]:
"""Return all templates where is_default=True, ordered by name."""
...
def list_prompt_templates_by_origin(self, origin: str) -> list[dict[str, Any]]:
"""Return all prompt templates with the given origin, ordered by name."""
...
def update_prompt_template(self, template_id: str, **fields: Any) -> bool:
"""Update specified fields on a prompt template. Returns True if found."""
...
+4
View File
@@ -140,6 +140,7 @@ scheduled_tasks = sa.Table(
sa.Column("initial_message", sa.Text, nullable=False),
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
sa.Column("auto_approve_tools", sa.Text, nullable=False, server_default=""),
sa.Column("template", sa.Text, nullable=False, server_default=""),
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
sa.Column("created_by", sa.Text, nullable=False, server_default=""),
sa.Column("last_run", sa.Text),
@@ -288,6 +289,9 @@ prompt_templates = sa.Table(
sa.Column("is_default", sa.Integer, nullable=False, server_default="0"),
sa.Column("org_id", sa.Text, nullable=False, server_default=""),
sa.Column("created_by", sa.Text, nullable=False, server_default=""),
sa.Column("origin", sa.Text, nullable=False, server_default="manual"),
sa.Column("mcp_server", sa.Text, nullable=False, server_default=""),
sa.Column("readonly", sa.Integer, nullable=False, server_default="0"),
sa.Column("created", sa.Text, nullable=False),
sa.Column("updated", sa.Text, nullable=False),
)
+42 -17
View File
@@ -916,6 +916,7 @@ class SQLiteBackend:
auto_approve_tools: list[str],
created_by: str,
next_run: str,
template: str = "",
) -> None:
from turnstone.core.storage._schema import scheduled_tasks
@@ -935,6 +936,7 @@ class SQLiteBackend:
"initial_message": initial_message,
"auto_approve": 1 if auto_approve else 0,
"auto_approve_tools": ",".join(auto_approve_tools),
"template": template,
"enabled": 1,
"created_by": created_by,
"next_run": next_run,
@@ -976,6 +978,7 @@ class SQLiteBackend:
"initial_message",
"auto_approve",
"auto_approve_tools",
"template",
"enabled",
"last_run",
"next_run",
@@ -1401,21 +1404,7 @@ class SQLiteBackend:
.select_from(user_roles.join(roles, user_roles.c.role_id == roles.c.role_id))
.where(user_roles.c.user_id == user_id)
).fetchall()
return [
{
"role_id": r[0],
"name": r[1],
"display_name": r[2],
"permissions": r[3],
"builtin": bool(r[4]),
"org_id": r[5],
"created": r[6],
"updated": r[7],
"assigned_by": r[8],
"assignment_created": r[9],
}
for r in rows
]
return [_row_to_dict(r, "builtin") for r in rows]
def get_user_permissions(self, user_id: str) -> set[str]:
with self._engine.connect() as conn:
@@ -1560,6 +1549,9 @@ class SQLiteBackend:
is_default: bool = False,
org_id: str = "",
created_by: str = "",
origin: str = "manual",
mcp_server: str = "",
readonly: bool = False,
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
@@ -1574,6 +1566,9 @@ class SQLiteBackend:
"is_default": 1 if is_default else 0,
"org_id": org_id,
"created_by": created_by,
"origin": origin,
"mcp_server": mcp_server,
"readonly": 1 if readonly else 0,
"created": now,
"updated": now,
},
@@ -1586,7 +1581,16 @@ class SQLiteBackend:
sa.select(prompt_templates).where(prompt_templates.c.template_id == template_id)
).fetchone()
if row:
return _row_to_dict(row, "is_default")
return _row_to_dict(row, "is_default", "readonly")
return None
def get_prompt_template_by_name(self, name: str) -> dict[str, Any] | None:
with self._engine.connect() as conn:
row = conn.execute(
sa.select(prompt_templates).where(prompt_templates.c.name == name)
).fetchone()
if row:
return _row_to_dict(row, "is_default", "readonly")
return None
def list_prompt_templates(self, org_id: str = "") -> list[dict[str, Any]]:
@@ -1595,7 +1599,28 @@ class SQLiteBackend:
if org_id:
q = q.where(prompt_templates.c.org_id == org_id)
rows = conn.execute(q).fetchall()
return [_row_to_dict(r, "is_default") for r in rows]
return [_row_to_dict(r, "is_default", "readonly") for r in rows]
def list_default_templates(self, org_id: str = "") -> list[dict[str, Any]]:
with self._engine.connect() as conn:
q = (
sa.select(prompt_templates)
.where(prompt_templates.c.is_default == 1)
.order_by(prompt_templates.c.name)
)
if org_id:
q = q.where(prompt_templates.c.org_id == org_id)
rows = conn.execute(q).fetchall()
return [_row_to_dict(r, "is_default", "readonly") for r in rows]
def list_prompt_templates_by_origin(self, origin: str) -> list[dict[str, Any]]:
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(prompt_templates)
.where(prompt_templates.c.origin == origin)
.order_by(prompt_templates.c.name)
).fetchall()
return [_row_to_dict(r, "is_default", "readonly") for r in rows]
def update_prompt_template(self, template_id: str, **fields: Any) -> bool:
dropped = set(fields) - _TEMPLATE_MUTABLE
@@ -0,0 +1,28 @@
"""Add MCP origin tracking columns to prompt_templates.
Revision ID: 009
Revises: 008
Create Date: 2026-03-12
"""
import sqlalchemy as sa
from alembic import op
revision = "009"
down_revision = "008"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.add_column(sa.Column("origin", sa.Text, nullable=False, server_default="manual"))
batch_op.add_column(sa.Column("mcp_server", sa.Text, nullable=False, server_default=""))
batch_op.add_column(sa.Column("readonly", sa.Integer, nullable=False, server_default="0"))
def downgrade() -> None:
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.drop_column("readonly")
batch_op.drop_column("mcp_server")
batch_op.drop_column("origin")
@@ -0,0 +1,24 @@
"""Add template column to scheduled_tasks.
Revision ID: 010
Revises: 009
Create Date: 2026-03-12
"""
import sqlalchemy as sa
from alembic import op
revision = "010"
down_revision = "009"
branch_labels = None
depends_on = None
def upgrade() -> None:
with op.batch_alter_table("scheduled_tasks") as batch_op:
batch_op.add_column(sa.Column("template", sa.Text, nullable=False, server_default=""))
def downgrade() -> None:
with op.batch_alter_table("scheduled_tasks") as batch_op:
batch_op.drop_column("template")
+37 -17
View File
@@ -139,12 +139,17 @@ class Bridge:
# -- public entry point --------------------------------------------------
def _fetch_node_id(self) -> str:
"""Retrieve node_id from server /health with exponential backoff.
"""Retrieve node_id from server /health with capped exponential backoff.
Raises ``SystemExit`` if the server is unreachable after 5 attempts.
Retries indefinitely so the bridge recovers when a server comes
back after a transient outage. 4xx responses (auth/config errors)
still fail fast.
"""
delays = [1, 2, 4, 8, 16]
for attempt, delay in enumerate(delays, 1):
attempt = 0
delay = 1.0
max_delay = 60.0
while True:
attempt += 1
try:
resp = self._http.get("/health")
if 400 <= resp.status_code < 500:
@@ -155,20 +160,17 @@ class Bridge:
nid = data.get("node_id", "")
if nid:
return str(nid)
log.warning("Server /health missing node_id (attempt %d/%d)", attempt, len(delays))
log.warning("Server /health missing node_id (attempt %d)", attempt)
except SystemExit:
raise
except Exception as exc:
log.warning(
"Failed to fetch node_id from server (attempt %d/%d): %s",
"Failed to fetch node_id from server (attempt %d): %s",
attempt,
len(delays),
exc,
)
if attempt < len(delays):
time.sleep(delay)
log.critical(
"Could not retrieve node_id from server after %d attempts — exiting", len(delays)
)
raise SystemExit(1)
time.sleep(delay)
delay = min(delay * 2, max_delay)
def run(self) -> None:
"""Block until shutdown (KeyboardInterrupt)."""
@@ -386,6 +388,7 @@ class Bridge:
initial_message = getattr(msg, "initial_message", "")
resume_ws = getattr(msg, "resume_ws", "")
user_id = getattr(msg, "user_id", "")
template = getattr(msg, "template", "")
if user_id:
log.info("bridge.create_ws user_id=%s name=%s model=%s", user_id, name, model)
ws_id, resumed = self._create_ws_on_server(
@@ -395,6 +398,7 @@ class Bridge:
correlation_id=msg.correlation_id,
model=model,
resume_ws=resume_ws,
template=template,
)
# Send initial_message only when no workstream was actually resumed.
# Use the server's `resumed` response (not just the intent) so that
@@ -462,6 +466,7 @@ class Bridge:
correlation_id: str,
model: str = "",
resume_ws: str = "",
template: str = "",
) -> tuple[str, bool]:
"""Create a workstream on the server. Returns (ws_id, resumed)."""
try:
@@ -470,6 +475,8 @@ class Bridge:
payload["model"] = model
if resume_ws:
payload["resume_ws"] = resume_ws
if template:
payload["template"] = template
resp = self._http.post(
"/v1/api/workstreams/new",
json=payload,
@@ -709,6 +716,11 @@ class Bridge:
def _wait_plan() -> None:
try:
raw_resp = self._broker.pop_response(request_id, timeout=self._approval_timeout)
# Clear pending entry *before* posting response so that
# a subsequent plan review event (from the refinement
# loop) is not skipped by the duplicate guard.
with self._lock:
self._pending_plan_reviews.pop(ws_id, None)
if raw_resp:
resp_msg = InboundMessage.from_json(raw_resp)
feedback = getattr(resp_msg, "feedback", "")
@@ -716,9 +728,16 @@ class Bridge:
else:
log.warning("Plan review timeout for ws %s — rejecting", ws_id)
self._http.post("/v1/api/plan", json={"feedback": "reject", "ws_id": ws_id})
finally:
except Exception:
with self._lock:
self._pending_plan_reviews.pop(ws_id, None)
# Best-effort rejection so the server doesn't hang
with contextlib.suppress(Exception):
self._http.post(
"/v1/api/plan",
json={"feedback": "reject", "ws_id": ws_id},
)
raise
threading.Thread(target=self._run_in_context(_wait_plan), daemon=True).start()
@@ -775,12 +794,13 @@ class Bridge:
)
)
# Completion detection
# Completion detection — emit for all idle transitions so
# channel adapters can finalize streaming messages even when
# the turn was initiated from the server UI (no correlation_id).
if state == "idle":
with self._lock:
cid = self._active_sends.pop(ws_id, None)
if cid:
self._publish_ws(ws_id, TurnCompleteEvent(ws_id=ws_id, correlation_id=cid))
self._publish_ws(ws_id, TurnCompleteEvent(ws_id=ws_id, correlation_id=cid or ""))
elif etype == "ws_rename":
self._publish_global(WorkstreamRenameEvent(ws_id=ws_id, name=data.get("name", "")))
+2
View File
@@ -126,6 +126,7 @@ class TurnstoneClient:
auto_approve_tools: list[str] | None = None,
target_node: str = "",
initial_message: str = "",
template: str = "",
) -> str:
"""Create a workstream. Returns correlation_id."""
msg = CreateWorkstreamMessage(
@@ -134,6 +135,7 @@ class TurnstoneClient:
auto_approve_tools=auto_approve_tools or [],
target_node=target_node,
initial_message=initial_message,
template=template,
)
self._broker.push_inbound(msg.to_json(), node_id=target_node)
return msg.correlation_id
+3 -1
View File
@@ -97,6 +97,7 @@ class CreateWorkstreamMessage(InboundMessage):
initial_message: str = ""
resume_ws: str = ""
user_id: str = ""
template: str = ""
@dataclass
@@ -264,7 +265,8 @@ class TurnCompleteEvent(OutboundEvent):
"""Emitted when a workstream finishes processing (returns to IDLE).
This is a synthetic event produced by the bridge when it detects
the ws_state transition to 'idle' after a send.
the ws_state transition to 'idle'. ``correlation_id`` is set for
MQ-initiated turns and empty for turns initiated from the server UI.
"""
type: str = "turn_complete"
+9 -1
View File
@@ -126,6 +126,7 @@ class AsyncTurnstoneConsole(_BaseClient):
name: str = "",
model: str = "",
initial_message: str = "",
template: str = "",
) -> ConsoleCreateWsResponse:
body: dict[str, Any] = {}
if node_id:
@@ -136,6 +137,8 @@ class AsyncTurnstoneConsole(_BaseClient):
body["model"] = model
if initial_message:
body["initial_message"] = initial_message
if template:
body["template"] = template
return await self._request(
"POST",
"/v1/api/cluster/workstreams/new",
@@ -574,10 +577,15 @@ class TurnstoneConsole:
name: str = "",
model: str = "",
initial_message: str = "",
template: str = "",
) -> ConsoleCreateWsResponse:
return self._runner.run(
self._async.create_workstream(
node_id=node_id, name=name, model=model, initial_message=initial_message
node_id=node_id,
name=name,
model=model,
initial_message=initial_message,
template=template,
)
)
+8
View File
@@ -94,6 +94,13 @@ class ApproveRequestEvent(ServerEvent):
items: list[dict[str, Any]] = field(default_factory=list)
@dataclass
class ApprovalResolvedEvent(ServerEvent):
type: str = "approval_resolved"
approved: bool = False
feedback: str = ""
@dataclass
class ToolResultEvent(ServerEvent):
type: str = "tool_result"
@@ -285,6 +292,7 @@ _SERVER_REGISTRY: dict[str, type[ServerEvent]] = {
StreamEndEvent,
ToolInfoEvent,
ApproveRequestEvent,
ApprovalResolvedEvent,
ToolResultEvent,
ToolOutputChunkEvent,
StatusEvent,
+9 -1
View File
@@ -77,6 +77,7 @@ class AsyncTurnstoneServer(_BaseClient):
model: str = "",
auto_approve: bool = False,
resume_ws: str = "",
template: str = "",
) -> CreateWorkstreamResponse:
body: dict[str, Any] = {}
if name:
@@ -87,6 +88,8 @@ class AsyncTurnstoneServer(_BaseClient):
body["auto_approve"] = True
if resume_ws:
body["resume_ws"] = resume_ws
if template:
body["template"] = template
return await self._request(
"POST",
"/v1/api/workstreams/new",
@@ -318,10 +321,15 @@ class TurnstoneServer:
model: str = "",
auto_approve: bool = False,
resume_ws: str = "",
template: str = "",
) -> CreateWorkstreamResponse:
return self._runner.run(
self._async.create_workstream(
name=name, model=model, auto_approve=auto_approve, resume_ws=resume_ws
name=name,
model=model,
auto_approve=auto_approve,
resume_ws=resume_ws,
template=template,
)
)
+59 -6
View File
@@ -97,6 +97,7 @@ class WebUI:
self._ws_completion_tokens: int = 0
self._ws_messages: int = 0
self._ws_tool_calls: dict[str, int] = {}
self._ws_tool_calls_reported: int = 0 # last cumulative total sent to usage
self._ws_context_ratio: float = 0.0
# Activity tracking for dashboard (current tool / thinking / approval)
self._ws_current_activity: str = ""
@@ -208,17 +209,21 @@ class WebUI:
storage = get_storage()
if storage is not None:
tool_names = [it.get("func_name", "") for it in pending if it.get("func_name")]
tool_names = [
it.get("approval_label", "") or it.get("func_name", "")
for it in pending
if it.get("func_name")
]
if tool_names:
verdicts = evaluate_tool_policies_batch(storage, tool_names)
still_pending = []
for it in pending:
fname = it.get("func_name", "")
verdict = verdicts.get(fname)
policy_name = it.get("approval_label", "") or it.get("func_name", "")
verdict = verdicts.get(policy_name)
if verdict == "deny":
it["denied"] = True
it["denial_msg"] = (
f"Blocked by tool policy (pattern match for '{fname}')"
f"Blocked by tool policy (pattern match for '{policy_name}')"
)
elif verdict == "allow":
it["needs_approval"] = False
@@ -308,7 +313,9 @@ class WebUI:
self._ws_prompt_tokens += usage["prompt_tokens"]
self._ws_completion_tokens += usage["completion_tokens"]
self._ws_context_ratio = total_tok / context_window if context_window > 0 else 0.0
tool_count = sum(self._ws_tool_calls.values())
tool_total = sum(self._ws_tool_calls.values())
tool_count = tool_total - self._ws_tool_calls_reported
self._ws_tool_calls_reported = tool_total
self._enqueue(
{
"type": "status",
@@ -371,8 +378,17 @@ class WebUI:
WebUI._global_queue.put({"type": "ws_rename", "ws_id": self.ws_id, "name": name})
def resolve_approval(self, approved: bool, feedback: str | None = None) -> None:
"""Called by the HTTP handler when the user approves/denies."""
"""Resolve a pending approval, whether triggered by the HTTP handler
(user approves/denies in the browser) or by server-initiated flows
such as cancellations or timeouts."""
self._approval_result = (approved, feedback)
self._enqueue(
{
"type": "approval_resolved",
"approved": approved,
"feedback": feedback or "",
}
)
self._approval_event.set()
def resolve_plan(self, feedback: str) -> None:
@@ -752,6 +768,13 @@ async def health(request: Request) -> JSONResponse:
"circuit_state": monitor.circuit_state.value if monitor else "closed",
},
}
mc = getattr(request.app.state, "mcp_client", None)
if mc:
data["mcp"] = {
"servers": mc.server_count,
"resources": mc.resource_count,
"prompts": mc.prompt_count,
}
return JSONResponse(data)
@@ -775,10 +798,19 @@ async def metrics_endpoint(request: Request) -> Response:
"context_ratio": ui._ws_context_ratio,
}
)
mcp_info = None
mc = getattr(request.app.state, "mcp_client", None)
if mc:
mcp_info = {
"servers": mc.server_count,
"resources": mc.resource_count,
"prompts": mc.prompt_count,
}
content = _metrics.generate_text(
workstream_states=states,
total_workstreams=len(wss),
workstream_metrics=ws_data,
mcp_info=mcp_info,
)
return Response(content, media_type="text/plain; version=0.0.4; charset=utf-8")
@@ -981,6 +1013,7 @@ async def create_workstream(request: Request) -> JSONResponse:
skip: bool = request.app.state.skip_permissions
auth = getattr(getattr(request, "state", None), "auth_result", None)
uid: str = getattr(auth, "user_id", "") or ""
body_template = body.get("template", "")
try:
ws = mgr.create(
name=body.get("name", ""),
@@ -1028,6 +1061,19 @@ async def create_workstream(request: Request) -> JSONResponse:
if history:
ui._enqueue({"type": "history", "messages": history})
# Per-workstream template override — only when not resumed (resumed
# workstreams restore their own template from workstream_config).
if body_template and not resumed and ws.session:
from turnstone.core.memory import get_prompt_template_by_name
if not get_prompt_template_by_name(body_template):
# Workstream already created — close it and return error
mgr.close(ws.id)
return JSONResponse(
{"error": f"Template not found: {body_template}"}, status_code=400
)
ws.session.set_template(body_template)
return JSONResponse(
{
"ws_id": ws.id,
@@ -1344,6 +1390,11 @@ def main() -> None:
default=None,
help="Developer instructions injected as developer message",
)
parser.add_argument(
"--template",
default=None,
help="Prompt template name (replaces default templates)",
)
parser.add_argument(
"--temperature",
type=float,
@@ -1688,6 +1739,7 @@ def main() -> None:
tool_search=args.tool_search,
tool_search_threshold=args.tool_search_threshold,
tool_search_max_results=args.tool_search_max_results,
template=args.template,
)
# Create WatchRunner (periodic command polling, server-level)
@@ -1801,6 +1853,7 @@ def main() -> None:
mcp_tools = mcp_client.get_tools()
if mcp_tools:
log.info("MCP tools: %d from %d server(s)", len(mcp_tools), mcp_client.server_count)
mcp_client.set_storage(get_storage())
log.info(
"Health monitor: probe every %ss, circuit breaker threshold=%s",
args.health_probe_interval,
+4
View File
@@ -28,6 +28,7 @@
--yellow: #fbbf24;
--cyan: #67e8f9;
--magenta: #c084fc;
--on-color: var(--bg);
/* Glow variants for LED effects */
--green-glow: rgba(52, 211, 153, 0.25);
@@ -35,6 +36,7 @@
--yellow-glow: rgba(251, 191, 36, 0.25);
--accent-glow-strong: rgba(229, 160, 66, 0.3);
--cyan-glow: rgba(103, 232, 249, 0.2);
--magenta-glow: rgba(192, 132, 252, 0.25);
/* Structure */
--border: rgba(255, 255, 255, 0.06);
@@ -66,11 +68,13 @@
--yellow: #b45309;
--cyan: #0e7490;
--magenta: #7c3aed;
--on-color: #ffffff;
--green-glow: rgba(4, 120, 87, 0.25);
--red-glow: rgba(220, 38, 38, 0.25);
--yellow-glow: rgba(180, 83, 9, 0.25);
--accent-glow-strong: rgba(140, 94, 27, 0.15);
--cyan-glow: rgba(14, 116, 144, 0.2);
--magenta-glow: rgba(124, 58, 237, 0.2);
--border: rgba(0, 0, 0, 0.08);
--border-strong: rgba(0, 0, 0, 0.12);
--code-bg: #f0f1f5;
+18
View File
@@ -0,0 +1,18 @@
{
"name": "read_resource",
"description": "Read a resource from a connected MCP server by URI. Returns the resource content (text or base64-encoded binary). Use this to access data, files, or content exposed by connected MCP servers. Available resource URIs are listed in your system context.",
"parameters": {
"type": "object",
"properties": {
"uri": {
"type": "string",
"description": "The resource URI to read (e.g. 'file:///path', 'db://table/row')."
}
},
"required": ["uri"]
},
"agent": true,
"task_agent": true,
"auto_approve": false,
"primary_key": "uri"
}
+23
View File
@@ -0,0 +1,23 @@
{
"name": "use_prompt",
"description": "Invoke an MCP prompt template by name, expanding it into messages. Returns the expanded prompt content that can be used as context or instructions.",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The prompt name (e.g. 'mcp__server__prompt_name')."
},
"arguments": {
"type": "object",
"description": "Key-value argument pairs for the prompt.",
"additionalProperties": { "type": "string" }
}
},
"required": ["name"]
},
"agent": true,
"task_agent": true,
"auto_approve": false,
"primary_key": "name"
}
+132 -17
View File
@@ -42,6 +42,26 @@ function pollHealth() {
.then(function (data) {
pollHealth._failCount = 0;
_lastHealth = data;
var mcpEl = document.getElementById("mcp-status");
if (mcpEl) {
if (data.mcp && data.mcp.servers > 0) {
mcpEl.textContent =
"MCP: " +
data.mcp.servers +
" server" +
(data.mcp.servers !== 1 ? "s" : "");
mcpEl.title =
data.mcp.resources +
" resources \u00b7 " +
data.mcp.prompts +
" prompts";
mcpEl.style.opacity = "1";
} else {
mcpEl.textContent = "";
mcpEl.title = "";
mcpEl.style.opacity = "0";
}
}
var el = document.getElementById("health-indicator");
if (!el) return;
if (data.status === "degraded") {
@@ -1071,6 +1091,10 @@ function handleEvent(evt) {
showInlineToolBlock(evt.items, false);
break;
case "approval_resolved":
resolveInlineApproval(evt.approved, false, evt.feedback, true);
break;
case "tool_output_chunk":
appendToolOutputChunk(evt.call_id || "", evt.chunk);
break;
@@ -1378,7 +1402,7 @@ function showInlineToolBlock(items, autoApproved) {
scrollToBottom();
}
function resolveInlineApproval(approved, always, feedback) {
function resolveInlineApproval(approved, always, feedback, skipPost) {
if (!approvalBlockEl) return;
pendingApproval = false;
@@ -1406,19 +1430,21 @@ function resolveInlineApproval(approved, always, feedback) {
sendBtn.disabled = busy;
inputEl.focus();
// POST to server with ws_id
authFetch("/v1/api/approve", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
approved: approved,
feedback: feedback || null,
always: !!always,
ws_id: currentWsId,
}),
}).catch(function (err) {
addErrorMessage("Connection error: " + err.message);
});
// POST to server with ws_id (skip when server already resolved, e.g. timeout)
if (!skipPost) {
authFetch("/v1/api/approve", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
approved: approved,
feedback: feedback || null,
always: !!always,
ws_id: currentWsId,
}),
}).catch(function (err) {
addErrorMessage("Connection error: " + err.message);
});
}
scrollToBottom();
}
@@ -1570,20 +1596,45 @@ function scrollToBottom(force) {
}
// --- Plan review dialog ---
var _planContent = "";
function showPlanDialog(content) {
_planContent = content;
document.getElementById("plan-content").textContent = content;
document.getElementById("plan-feedback").value = "";
var feedbackEl = document.getElementById("plan-feedback");
feedbackEl.value = "";
_updatePlanRejectBtn();
inputEl.disabled = true;
sendBtn.disabled = true;
document.getElementById("plan-overlay").classList.add("active");
setTimeout(function () {
document.getElementById("plan-feedback").focus();
feedbackEl.focus();
}, 50);
}
function _updatePlanRejectBtn() {
var btn = document.getElementById("btn-plan-reject");
var hasFeedback =
document.getElementById("plan-feedback").value.trim().length > 0;
btn.innerHTML = hasFeedback
? '<span class="key">Esc</span> Amend'
: '<span class="key">Esc</span> Reject';
btn.style.background = hasFeedback ? "var(--accent)" : "";
btn.style.color = hasFeedback ? "var(--on-color)" : "";
btn.onclick = function () {
resolvePlan(hasFeedback ? "" : "reject");
};
}
function resolvePlan(defaultFeedback) {
let feedback = document.getElementById("plan-feedback").value.trim();
if (!feedback && defaultFeedback) feedback = defaultFeedback;
document.getElementById("plan-overlay").classList.remove("active");
inputEl.disabled = false;
sendBtn.disabled = false;
inputEl.focus();
// Critical: fire the API call first — this unblocks the server.
// The inline rendering below is cosmetic and must never prevent it.
authFetch("/v1/api/plan", {
method: "POST",
headers: { "Content-Type": "application/json" },
@@ -1591,6 +1642,65 @@ function resolvePlan(defaultFeedback) {
}).catch(function (err) {
addErrorMessage("Connection error: " + err.message);
});
// Render plan inline in the chat (best-effort)
try {
var isReject = feedback === "reject";
var isAmend = feedback && !isReject;
var action = isReject ? "rejected" : isAmend ? "amending" : "approved";
_addInlinePlan(_planContent, action, feedback);
} catch (err) {
console.error("Failed to render inline plan:", err);
addInfoMessage("Plan " + action);
}
// Show spinner while the model processes the plan result
setBusy(true);
addThinkingIndicator();
}
function _addInlinePlan(content, action, feedback) {
if (!content) return;
var wrapper = document.createElement("div");
wrapper.className = "plan-inline";
var header = document.createElement("div");
header.className = "plan-inline-header";
var label =
action === "rejected"
? "Plan rejected"
: action === "amending"
? "Plan — amending"
: "Plan approved";
header.innerHTML =
'<span class="plan-inline-label plan-' + action + '">' + label + "</span>";
wrapper.appendChild(header);
var body = document.createElement("div");
body.className = "plan-inline-body";
try {
body.innerHTML = renderMarkdown(content);
} catch (e) {
body.textContent = content;
}
if (content.split("\n").length > 12) {
makeCollapsible(body);
body.setAttribute(
"aria-label",
"Plan content (collapsed). Activate to expand.",
);
}
wrapper.appendChild(body);
if (feedback && action === "amending") {
var fb = document.createElement("div");
fb.className = "plan-inline-feedback";
fb.textContent = "Feedback: " + feedback;
wrapper.appendChild(fb);
}
messagesEl.appendChild(wrapper);
scrollToBottom();
}
// --- Send message ---
@@ -1647,6 +1757,9 @@ function autoResize() {
}
inputEl.addEventListener("input", autoResize);
document
.getElementById("plan-feedback")
.addEventListener("input", _updatePlanRejectBtn);
inputEl.addEventListener("keydown", function (e) {
if (e.key === "Enter" && !e.shiftKey) {
e.preventDefault();
@@ -1749,7 +1862,9 @@ document.addEventListener("keydown", function (e) {
resolvePlan("");
} else if (e.key === "Escape") {
e.preventDefault();
resolvePlan("reject");
var hasFb =
document.getElementById("plan-feedback").value.trim().length > 0;
resolvePlan(hasFb ? "" : "reject");
} else if (e.key === "Tab") {
var focusable = document.querySelectorAll(
"#plan-dialog input, #plan-dialog button",
+4 -3
View File
@@ -28,6 +28,7 @@
</div>
<h1>turnstone</h1>
<span id="model-name"></span>
<span id="mcp-status" role="status" aria-live="polite"></span>
<span id="status-bar"></span>
<span id="health-indicator" class="health-ok" role="status" aria-live="polite" aria-atomic="true"></span>
<button id="logout-btn" class="header-btn" onclick="logout()" style="display:none">logout</button>
@@ -76,10 +77,10 @@
<div id="plan-dialog" role="dialog" aria-modal="true" aria-labelledby="plan-dialog-title">
<h3 id="plan-dialog-title">Plan Review</h3>
<div id="plan-content"></div>
<input type="text" id="plan-feedback" placeholder="Feedback (empty = approve)...">
<input type="text" id="plan-feedback" placeholder="feedback (optional)" aria-label="Plan feedback">
<div id="plan-buttons">
<button id="btn-plan-reject" onclick="resolvePlan('reject')">Reject</button>
<button id="btn-plan-approve" onclick="resolvePlan('')">Approve</button>
<button id="btn-plan-reject"><span class="key">Esc</span> Reject</button>
<button id="btn-plan-approve" onclick="resolvePlan('')"><span class="key">&crarr;</span> Approve</button>
</div>
</div>
</div>
+75 -4
View File
@@ -12,6 +12,14 @@
font-family: var(--font-display);
letter-spacing: 0.02em;
}
#mcp-status {
color: var(--magenta);
font-size: 11px;
font-family: var(--font-mono);
opacity: 0;
transition: opacity 0.3s;
cursor: default;
}
#health-indicator {
font-size: 11px;
padding: 2px 8px;
@@ -497,11 +505,73 @@
cursor: pointer;
font-weight: 600;
letter-spacing: 0.02em;
transition: filter 0.15s;
display: inline-flex;
align-items: center;
gap: 5px;
transition: filter 0.15s, background 0.2s;
}
#plan-buttons button:hover { filter: brightness(1.1); }
#btn-plan-approve { background: var(--green); color: var(--bg); }
#btn-plan-reject { background: var(--red); color: var(--bg); }
#plan-buttons button:focus-visible { outline: 2px solid var(--fg-bright); outline-offset: 2px; }
#plan-buttons button .key {
display: inline-block;
background: rgba(0,0,0,0.2);
border-radius: var(--radius-sm);
padding: 0 4px;
font-size: 10px;
font-weight: 600;
}
#btn-plan-approve { background: var(--green); color: var(--on-color); }
#btn-plan-reject { background: var(--red); color: var(--on-color); }
/* Inline plan block (rendered in chat after plan review) */
.plan-inline {
margin: 8px 0;
border: 1px solid var(--border);
border-radius: var(--radius);
overflow: hidden;
}
.plan-inline-header {
padding: 6px 12px;
background: var(--bg-surface);
border-bottom: 1px solid var(--border);
font-size: 12px;
font-family: var(--font-display);
font-weight: 600;
letter-spacing: 0.02em;
}
.plan-inline-label.plan-approved { color: var(--green); }
.plan-inline-label.plan-rejected { color: var(--red); }
.plan-inline-label.plan-amending { color: var(--accent); }
.plan-inline-body {
padding: 10px 14px;
background: var(--code-bg);
font-size: 12px;
line-height: 1.6;
max-height: 300px;
overflow-y: auto;
}
.plan-inline-body.collapsed { max-height: 150px; position: relative; }
.plan-inline-body.collapsed::after {
content: 'click to expand';
position: absolute;
bottom: 0; left: 0; right: 0;
text-align: center;
padding: 8px 0 4px;
background: linear-gradient(transparent, var(--code-bg));
color: var(--fg-dim);
font-size: 11px;
cursor: pointer;
}
.plan-inline-body h2, .plan-inline-body h3 { font-size: 13px; margin: 10px 0 4px; color: var(--accent); }
.plan-inline-body p { margin: 4px 0; }
.plan-inline-body ol, .plan-inline-body ul { margin: 4px 0; padding-left: 20px; }
.plan-inline-feedback {
padding: 6px 12px;
border-top: 1px solid var(--border);
font-size: 12px;
color: var(--accent);
font-style: italic;
}
/* ==========================================================================
Focus indicators server-specific overrides
@@ -637,5 +707,6 @@
.approval-btn, .approval-feedback-input,
#plan-buttons button, #input-area button,
.dashboard-new-btn, .dashboard-input,
#health-indicator, #hamburger-btn { transition: none; }
#health-indicator, #hamburger-btn,
#mcp-status { transition: none; }
}