Compare commits

...

23 Commits

Author SHA1 Message Date
Patrick Buckley 136b75fdef Bump version to 0.5.0 2026-03-08 04:47:10 -07:00
Patrick Buckley 4d1107839b refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing

- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service

* Refactor SSE proxy to raw byte passthrough

- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
  httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
  events, and comments now flow through verbatim. Added per-request timeout
  override (read=None, pool=None) for long-lived SSE streams.

- tests/test_console.py: Add 3 new tests for SSE proxy:
  - Ping and event preservation
  - Upstream error status handling
  - Client disconnect handling

- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
  approach.
2026-03-08 04:46:34 -07:00
Patrick Buckley 7d66bc2159 Bump version to 0.4.6 2026-03-08 03:44:22 -07:00
Patrick Buckley 165cbb2d29 Bump version to 0.4.5 2026-03-08 03:29:44 -07:00
Patrick Buckley c79c47b940 Add MCP dynamic tool refresh with push notifications and periodic pol… (#31)
* Add MCP dynamic tool refresh with push notifications and periodic polling

MCP tool lists now stay up-to-date without restart via three mechanisms:
push notifications (ToolListChangedNotification) for servers that support
it, staggered periodic polling for servers that don't, and manual
/mcp refresh [server] command. MCPClientManager tracks tools per-server
with copy-on-write rebuild, notifies ChatSession listeners which rebuild
tool lists and ToolSearchManager (preserving expanded tools).

* Address Copilot review feedback on MCP refresh PR

- Fix /mcp refresh typo matching (startswith → exact token check)
- Validate --mcp-refresh-interval >= 0 at parse time via shared
  nonneg_float in config.py (deduplicated from cli.py + server.py)
- Clamp negative refresh_interval to 0 in MCPClientManager constructor
- Fix periodic refresh first poll timing (was initial_delay + interval,
  now initial_delay then immediate first poll)
- Clarify _on_mcp_tools_changed docstring re: O(n) BM25 build cost
2026-03-08 03:28:38 -07:00
Patrick Buckley 660c273e8e remove old demo.svg 2026-03-08 01:47:34 -08:00
Patrick Buckley c7586abd0a Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI

When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:

- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool

New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.

Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.

43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.

* Fix Copilot review feedback on tool search

- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
2026-03-08 01:43:38 -08:00
Patrick Buckley 14d57176ce Bump version to 0.4.4 2026-03-07 15:43:56 -08:00
Patrick Buckley 96084ca5f3 Fix PostgreSQL migration race condition with advisory lock
Multiple containers starting simultaneously race on Alembic migrations
against shared PostgreSQL. Use pg_advisory_lock so they wait in line.
Also update SQLite bootstrap to detect post-migration databases.
2026-03-07 15:41:29 -08:00
Patrick Buckley 8b92302247 Bump version to 0.4.3 2026-03-07 13:11:42 -08:00
Patrick Buckley e195ca54a6 new high level arch abstract 2026-03-07 13:01:19 -08:00
Patrick Buckley 50277cd4de Add 10-node cluster profile to Docker Compose 2026-03-07 12:54:41 -08:00
Patrick Buckley fb190f8977 Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity

Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).

Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id

Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days

Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.

* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec

- docs/channels.md: "resumes the session" → "resumes the workstream",
  "Session resumed:" → "Resumed:", "old session was pruned" → "old
  workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
  object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
  models — removes all stale session_id properties from WorkstreamInfo,
  DashboardWorkstream, CreateWorkstreamResponse schemas
2026-03-07 12:49:07 -08:00
Patrick Buckley 25b5e32089 Bump version to 0.4.2 2026-03-05 20:18:46 -08:00
Patrick Buckley 339981a258 Add GPT-5.3, GPT-5.4, and pro model capabilities (#28)
* Add GPT-5.3, GPT-5.4, and pro model capabilities

Add capability entries for gpt-5-pro (272k output, high-only reasoning),
gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro.

* Validate reasoning_effort against model capabilities

_apply_model_params now falls back to caps.default_reasoning_effort when
the requested value is not in caps.reasoning_effort_values. Prevents
sending unsupported effort levels to models like gpt-5-pro (high only).
2026-03-05 20:17:03 -08:00
Patrick Buckley 06de9ff83b Bump version to 0.4.1 2026-03-05 18:13:42 -08:00
Patrick Buckley 924b976f1f Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
2026-03-05 18:11:36 -08:00
Patrick Buckley d5db817391 Fix channel gateway Docker networking and console proxy approval scope
Two runtime bugs:

1. Channel gateway advertised http://127.0.0.1:8091 which is
   unreachable from other Docker containers. Add
   TURNSTONE_CHANNEL_ADVERTISE_URL env var override for Docker/K8s
   environments, set to http://channel:8091 in compose.yaml, and
   pass --http-host=0.0.0.0 so the gateway listens on all interfaces.

2. Console proxy service JWT had only "write" scope but the approval
   endpoint requires "approve". Tool approval buttons in the server
   web UI silently failed when accessed through the console proxy.
   Changed proxy token scopes to read+write+approve.
2026-03-05 18:01:52 -08:00
Patrick Buckley fc8ceb4c72 Update README: 3-node cluster diagram, remove directory tree
Replace single-node Mermaid diagram with a 3-node cluster layout
showing bridge+server pairs per node, shared Redis MQ, console, and
channel gateway. Remove the verbose directory tree listing.
2026-03-05 17:44:33 -08:00
Patrick Buckley 07234dec4d Replace ASCII architecture diagram with Mermaid in README
GitHub renders Mermaid natively as an interactive SVG. The new diagram
shows all client entry points (CLI, browser, SDK, Discord), the full
cluster topology including the channel gateway and notify path, and
the LLM provider layer.
2026-03-05 17:34:42 -08:00
Patrick Buckley dd4cc0b30d Add PROGRESS.md and .coverage to .gitignore 2026-03-05 17:31:55 -08:00
Patrick Buckley e7fe8fca9d Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening

Implements the `notify` tool allowing the LLM to send notifications to
Discord channels/users via the channel gateway. Includes fixes for 11
review findings: JWT auth on the gateway endpoint, first-healthy gateway
delivery with retry+backoff, rate limiting only on success, SSRF URL
scheme validation, Discord mention sanitization, SQLite ON CONFLICT
upsert preserving created timestamps, advertise URL resolution for
0.0.0.0 bind, randomized service IDs, generic error messages to prevent
internal state leakage, and partial direct-target validation.

Service registry with heartbeat-based health filtering (migration 005).
Channel gateway registers on startup, heartbeats every 30s, deregisters
on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint
auth (static + JWT), storage CRUD, and retry behavior.

* Add notify documentation, diagrams, and review fixes

Documentation:
- New sequence diagram 17-notify-flow.puml showing end-to-end delivery
- Updated 16-channel-architecture.puml with services table, notify HTTP
  path, and Notification Flow note
- channels.md: Notifications section (targeting, delivery flow, service
  registry, security) and new config table entries
- tools.md: notify tool reference, updated counts/tables (14→15 tools)
- security.md: channel gateway row in service-to-service auth table
- architecture.md: notification subsystem paragraph

Review fixes (copilot):
- _http.py: fail closed when auth unconfigured (401 instead of pass-
  through), strip whitespace on message/title, generic error messages
  for user-not-found vs no-linked-channels
- session.py: parse gateway response JSON and require at least one
  result with status=="sent" before counting as success
- _postgresql.py: use index_elements instead of constraint for upsert
2026-03-05 17:24:00 -08:00
Patrick Buckley 42b9f89988 Add scheduled task system with cron/at scheduling (#26)
* Add scheduled task system with cron/at scheduling, admin API, and console UI

Console-integrated background scheduler dispatches workstreams on recurring
cron expressions or one-shot ISO8601 timestamps. Four target modes: auto
(best node by headroom), pool (shared queue), all (fan-out), or specific
node. Redis distributed lock with unique owner + Lua conditional release
prevents duplicate dispatch in multi-console deployments.

Storage: scheduled_tasks + scheduled_task_runs tables (migration 004),
9 protocol methods on both SQLite and PostgreSQL backends, field allowlist
on updates, run history auto-pruned at 90 days.

API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter
validation, ISO8601 future-time checks, field length bounds, schedule
count cap (200), and OpenAPI spec entries with Pydantic models.

UI: Schedules tab in admin panel with create/edit/delete modals, run
history modal, cron/at type toggle, target mode select, status dots for
accessibility, responsive grid, keyboard navigation, and focus management.

Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches
logged at WARNING with created_by attribution, user_id propagated in
CreateWorkstreamMessage for audit trail.

46 tests across storage, scheduler engine, and API endpoints.

* Add croniter to test extras for CI compatibility

CI installs [test] extras but not [console], so croniter was missing
when schedule API tests import console/server.py validation functions.

* Address Copilot review: timezone validation, focus trap, enabled flag

- Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z)
- UI appends +00:00 to datetime-local values for explicit UTC
- Fix datetime-local normalization: check length before appending seconds
- Add textarea to modal focus trap selector (prevents focus escape)
- Fix _normalize_task_dict not called in update response
- Persist enabled=false on create (storage defaults to enabled=1)
- Validate at_time is still in future when re-enabling a one-shot task
- broker._redis coupling acknowledged as tracked tech debt
2026-03-05 16:05:00 -08:00
110 changed files with 9402 additions and 1611 deletions
+2
View File
@@ -17,3 +17,5 @@ venv/
.plan.md
.plan-*.md
.hypothesis/
PROGRESS.md
.coverage
+15 -74
View File
@@ -11,7 +11,7 @@ Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone)
## What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. It runs as:
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. Native deferred tool loading for Anthropic and OpenAI APIs reduces token overhead and improves tool selection accuracy when MCP servers expose many tools; local models (vLLM, llama.cpp) get a transparent client-side BM25 fallback. It runs as:
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
@@ -19,13 +19,9 @@ Turnstone gives LLMs tools — shell, files, search, web, planning — and orche
- **Cluster dashboard** — real-time view of all nodes and workstreams, workstream creation with node targeting, reverse proxy for server UIs (only the console port needs network access)
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
```
External System → Message Queue → Bridge (per node) → Turnstone Server → LLM + Tools
Pub/Sub → Progress Events → External System
turnstone-console → Cluster Dashboard (browser)
```
<p align="center">
<img src="docs/diagrams/architecture-overview.svg" alt="Turnstone system architecture — data flow from clients through gateways, Redis MQ, cluster nodes, to LLM providers" width="960"/>
</p>
## Quickstart
@@ -111,69 +107,7 @@ All frontends connect to any OpenAI-compatible API (vLLM, NVIDIA NIM/NGC, llama.
## Architecture
```
turnstone/
├── core/ # UI-agnostic engine
│ ├── session.py # ChatSession — multi-turn loop, tool dispatch, agents
│ ├── providers/ # LLM provider adapters (OpenAI, Anthropic)
│ │ ├── _protocol.py # LLMProvider protocol, ModelCapabilities, StreamChunk
│ │ ├── _openai.py # OpenAI-compatible (OpenAI, vLLM, llama.cpp)
│ │ └── _anthropic.py # Anthropic Messages API (native streaming, thinking)
│ ├── tools.py # Tool definitions (auto-loaded from JSON)
│ ├── workstream.py # WorkstreamManager — parallel independent sessions
│ ├── mcp_client.py # MCP client manager (external tool servers)
│ ├── model_registry.py # ModelRegistry — named models, fallback routing, per-workstream selection
│ ├── config.py # Unified TOML config (~/.config/turnstone/config.toml)
│ ├── memory.py # Persistence facade (delegates to storage/)
│ ├── storage/ # Pluggable storage backend (SQLite + PostgreSQL)
│ ├── metrics.py # Prometheus-compatible metrics collector
│ ├── healthcheck.py # Backend health monitor + circuit breaker
│ ├── ratelimit.py # Per-IP token-bucket rate limiter
│ ├── edit.py # File editing (fuzzy match, indentation)
│ ├── safety.py # Path validation, sandbox checks
│ ├── sandbox.py # Command sandboxing
│ └── web.py # Web fetch/search helpers
├── mq/ # Message queue integration
│ ├── protocol.py # Typed message dataclasses (JSON serialization)
│ ├── broker.py # Abstract MessageBroker + RedisBroker
│ ├── bridge.py # Bridge service (queue ↔ HTTP API, multi-node routing)
│ └── client.py # TurnstoneClient — Python API for external systems
├── console/ # Cluster dashboard
│ ├── collector.py # ClusterCollector — aggregates all nodes via Redis + HTTP
│ ├── server.py # Dashboard Starlette/ASGI server + SSE
│ └── static/ # Cluster dashboard web UI
├── tools/ # Tool schemas (one JSON file per tool)
├── ui/ # Frontend assets and terminal rendering
│ └── static/ # Web UI (HTML, CSS, JS)
├── sim/ # Cluster simulator
│ ├── cluster.py # SimCluster — orchestrates N nodes + dispatchers
│ ├── node.py # SimNode + SimWorkstream — protocol-compatible node
│ ├── engine.py # LLM + tool execution simulation
│ ├── scenario.py # 5 workload scenarios (steady, burst, node_failure, …)
│ ├── metrics.py # Latency, throughput, utilization collection
│ └── cli.py # CLI entry point (turnstone-sim)
├── cli.py # Terminal frontend (+ /cluster commands for console)
├── server.py # Web frontend (Starlette/ASGI + SSE)
└── eval.py # Evaluation and prompt optimization harness
├── api/ # OpenAPI spec generation (Pydantic v2 models)
├── sdk/ # Client SDKs (sync + async, Python)
docs/
├── architecture.md # System architecture and threading model
├── api-reference.md # Web server API and SSE event reference
├── sdk.md # Client SDK reference (Python + TypeScript)
├── console.md # Cluster dashboard service (turnstone-console)
├── docker.md # Docker Compose deployment and configuration
├── simulator.md # Cluster simulator usage and scenarios
├── tools.md # Tool schemas, execution pipeline, approval flow
├── eval.md # Evaluation harness internals
└── diagrams/ # UML architecture diagrams (PlantUML sources + PNGs)
└── png/ # Pre-rendered diagram images
deploy/
├── helm/turnstone/ # Helm chart for Kubernetes
└── terraform/ # Terraform modules (AWS ECS/Fargate)
```
### Architecture Diagrams
### Diagrams
Detailed UML diagrams are available in [`docs/diagrams/`](docs/diagrams/):
@@ -217,7 +151,7 @@ Bridges BLPOP from their per-node queue (priority) then the shared queue. Direct
## Tools
14 built-in tools, 2 agent tools, plus external tools via MCP:
15 built-in tools, 2 agent tools, plus external tools via MCP:
| Tool | Description | Auto-approved |
|------|-------------|:---:|
@@ -233,13 +167,16 @@ Bridges BLPOP from their per-node queue (priority) then the shared queue. Direct
| `remember` | Save persistent facts | yes |
| `recall` | Search memories and history | yes |
| `forget` | Remove a memory | yes |
| `notify` | Send notifications to linked channels | yes |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
| `mcp__*` | External tools from MCP servers | |
When the total tool count exceeds a configurable threshold (default 20), MCP tools are automatically deferred using native `defer_loading` on Anthropic and OpenAI APIs, or a transparent client-side BM25 search for local models. The LLM discovers deferred tools on demand via a `tool_search` capability — no configuration needed beyond `--tool-search auto` (the default).
### MCP Tool Servers
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions.
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions. Tool lists stay fresh via push notifications (`tools.listChanged`), periodic polling for servers without push, and manual `/mcp refresh`.
Configure via `config.toml` or `--mcp-config`:
@@ -259,7 +196,7 @@ turnstone --mcp-config ~/.config/turnstone/mcp.json
turnstone-server --mcp-config ~/.config/turnstone/mcp.json
```
Use `/mcp` in the REPL to list connected tools. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
Use `/mcp` in the REPL to list connected tools, `/mcp refresh` to re-fetch tool lists from servers. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
### Multi-Model and Multi-Provider Support
@@ -314,6 +251,9 @@ agent_model = "" # model alias for plan/task sub-agents
[tools]
timeout = 30
skip_permissions = false
search = "auto" # "auto" (enable when >threshold tools), "on", "off"
search_threshold = 20 # min tools before tool search activates
search_max_results = 5 # max tools returned per search query
[server]
host = "0.0.0.0"
@@ -354,6 +294,7 @@ path = ".turnstone.db" # SQLite file path (relative to working directory)
[mcp]
config_path = "" # path to MCP JSON config file (alternative to TOML sections)
refresh_interval = 14400 # periodic refresh for servers without push notifications (seconds, 0 to disable)
[mcp.servers.example] # one section per MCP server
command = "npx"
+221 -1
View File
@@ -5,8 +5,8 @@
# Default (SQLite): docker compose up
# Production (PG): DB_BACKEND=postgresql docker compose --profile production up
# (or set DB_BACKEND=postgresql in .env)
# 10-node cluster: docker compose --profile cluster up
# With simulator: docker compose --profile sim up
# Scale bridges: docker compose up --scale bridge=3
# =============================================================================
name: turnstone
@@ -28,6 +28,7 @@ services:
image: postgres:17-alpine
profiles:
- production
- cluster
environment:
POSTGRES_DB: turnstone
POSTGRES_USER: ${POSTGRES_USER:-turnstone}
@@ -206,6 +207,7 @@ services:
dockerfile: Dockerfile
profiles:
- production
- cluster
command:
- sh
- -c
@@ -213,6 +215,7 @@ services:
turnstone-channel
--redis-host=redis
--redis-port=6379
--http-host=0.0.0.0
$${TURNSTONE_DISCORD_GUILD:+--discord-guild $$TURNSTONE_DISCORD_GUILD}
environment:
- TURNSTONE_DISCORD_TOKEN=${TURNSTONE_DISCORD_TOKEN:-}
@@ -221,6 +224,7 @@ services:
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
- TURNSTONE_CHANNEL_ADVERTISE_URL=http://channel:8091
networks:
- turnstone-net
depends_on:
@@ -270,3 +274,219 @@ services:
redis:
condition: service_healthy
restart: "no"
# ===================================================================
# 10-node cluster (profile: cluster)
#
# Each node is a server + bridge pair. All share the same PostgreSQL
# and Redis instances. Access via console at :8090.
#
# Start: docker compose --profile cluster up
# ===================================================================
# -- cluster servers ------------------------------------------------
server-1: &cluster-server
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster]
command: &cluster-server-cmd
- sh
- -c
- >-
turnstone-server
--host 0.0.0.0
--port 8080
--base-url "$${LLM_BASE_URL}"
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
volumes: [turnstone-data:/data]
environment: &cluster-server-env
LLM_BASE_URL: ${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
OPENAI_API_KEY: ${OPENAI_API_KEY:-dummy}
TAVILY_API_KEY: ${TAVILY_API_KEY:-}
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_AUTH_ENABLED: ${TURNSTONE_AUTH_ENABLED:-}
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
MODEL: ${MODEL:-}
TURNSTONE_DB_BACKEND: ${DB_BACKEND:-postgresql}
TURNSTONE_DB_URL: ${DATABASE_URL:-postgresql://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:?}@postgres:5432/turnstone}
TURNSTONE_NODE_ID: node-1
extra_hosts: ["host.docker.internal:host-gateway"]
networks: [turnstone-net]
depends_on:
redis: { condition: service_healthy }
postgres: { condition: service_healthy }
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
deploy:
resources:
limits: { memory: 384M, cpus: '0.5' }
restart: unless-stopped
server-2:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-2 }
server-3:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-3 }
server-4:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-4 }
server-5:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-5 }
server-6:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-6 }
server-7:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-7 }
server-8:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-8 }
server-9:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-9 }
server-10:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-10 }
# -- cluster bridges ------------------------------------------------
bridge-1: &cluster-bridge
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster]
command:
- turnstone-bridge
- --server-url=http://server-1:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
environment: &cluster-bridge-env
REDIS_PASSWORD: ${REDIS_PASSWORD:-}
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
networks: [turnstone-net]
depends_on:
server-1: { condition: service_healthy }
redis: { condition: service_healthy }
deploy:
resources:
limits: { memory: 256M, cpus: '0.25' }
restart: unless-stopped
bridge-2:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-2:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-2: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-3:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-3:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-3: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-4:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-4:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-4: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-5:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-5:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-5: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-6:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-6:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-6: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-7:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-7:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-7: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-8:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-8:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-8: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-9:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-9:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-9: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-10:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-10:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-10: { condition: service_healthy }
redis: { condition: service_healthy }
-221
View File
@@ -1,221 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" font-family="ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace" font-size="13">
<style>
@keyframes pulse-green { 0%,100% { opacity:0.5 } 50% { opacity:1 } }
@keyframes pulse-yellow { 0%,100% { opacity:0.4 } 50% { opacity:1 } }
@keyframes pulse-blue { 0%,100% { opacity:0.3 } 50% { opacity:1 } }
@keyframes fadein { from { opacity:0 } to { opacity:1 } }
.pg { animation: pulse-green 2s infinite }
.py { animation: pulse-yellow 1.8s infinite }
.pb { animation: pulse-blue 2.2s infinite }
.f1 { animation: fadein 0.4s 0.2s both }
.f2 { animation: fadein 0.4s 0.4s both }
.f3 { animation: fadein 0.4s 0.6s both }
.f4 { animation: fadein 0.4s 0.8s both }
.f5 { animation: fadein 0.4s 1.0s both }
.f6 { animation: fadein 0.4s 1.3s both }
.f7 { animation: fadein 0.4s 1.5s both }
.f8 { animation: fadein 0.4s 1.7s both }
.f9 { animation: fadein 0.4s 1.9s both }
.f10 { animation: fadein 0.4s 2.1s both }
.f11 { animation: fadein 0.4s 2.3s both }
.f12 { animation: fadein 0.4s 2.5s both }
</style>
<!-- Window chrome -->
<rect rx="10" width="860" height="520" fill="#1a1b26"/>
<rect width="860" height="36" rx="10" fill="#16161e"/>
<rect y="26" width="860" height="10" fill="#16161e"/>
<circle cx="20" cy="18" r="6" fill="#f7768e"/>
<circle cx="40" cy="18" r="6" fill="#e0af68"/>
<circle cx="60" cy="18" r="6" fill="#9ece6a"/>
<text x="430" y="22" text-anchor="middle" fill="#565f89" font-size="12">turnstone — console</text>
<!-- Header -->
<rect y="36" width="860" height="30" fill="#24283b"/>
<rect y="66" width="860" height="1" fill="#3b4261"/>
<text x="16" y="56" fill="#7aa2f7" font-size="14" font-weight="bold">turnstone console</text>
<text x="200" y="56" fill="#565f89" font-size="12">6 nodes · 10 workstreams</text>
<!-- ====== State cards ====== -->
<g transform="translate(16, 78)" class="f1" opacity="0">
<!-- RUN card -->
<rect x="0" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="0" y="0" width="156" height="3" rx="6" fill="#9ece6a"/>
<text x="78" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">3</text>
<text x="78" y="50" text-anchor="middle" fill="#565f89" font-size="10">▸ RUN</text>
<!-- THINK card -->
<rect x="168" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="168" y="0" width="156" height="3" rx="6" fill="#7aa2f7"/>
<text x="246" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">2</text>
<text x="246" y="50" text-anchor="middle" fill="#565f89" font-size="10">◌ THINK</text>
<!-- ATTN card -->
<rect x="336" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="336" y="0" width="156" height="3" rx="6" fill="#e0af68"/>
<text x="414" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">1</text>
<text x="414" y="50" text-anchor="middle" fill="#565f89" font-size="10">◆ ATTN</text>
<!-- ERR card -->
<rect x="504" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="504" y="0" width="156" height="3" rx="6" fill="#f7768e"/>
<text x="582" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">0</text>
<text x="582" y="50" text-anchor="middle" fill="#565f89" font-size="10">✖ ERR</text>
<!-- IDLE card -->
<rect x="672" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="672" y="0" width="156" height="3" rx="6" fill="#565f89"/>
<text x="750" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">4</text>
<text x="750" y="50" text-anchor="middle" fill="#565f89" font-size="10">· IDLE</text>
</g>
<!-- Aggregate bar -->
<text x="16" y="160" fill="#565f89" font-size="11" class="f2" opacity="0">197k tokens · 42 tool calls</text>
<!-- ====== NODES section ====== -->
<text x="16" y="182" fill="#7aa2f7" font-size="12" font-weight="bold" class="f3" opacity="0">NODES</text>
<!-- Node column headers -->
<g transform="translate(0, 190)" class="f4" opacity="0">
<rect width="860" height="20" fill="#24283b"/>
<rect y="20" width="860" height="1" fill="#3b4261"/>
<text y="14" fill="#565f89" font-size="10" letter-spacing="0.5">
<tspan x="36">NODE</tspan>
<tspan x="560">WS</tspan>
<tspan x="610">RUN</tspan>
<tspan x="660">ATTN</tspan>
<tspan x="710">TOKENS</tspan>
<tspan x="790">LOAD</tspan>
</text>
</g>
<!-- Node rows -->
<g transform="translate(0, 214)">
<!-- Node 1: db-west-04 — 3 ws, 1 running, has-running bar -->
<g class="f5" opacity="0">
<rect y="0" width="860" height="38" fill="#1a1b26"/>
<rect y="0" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="19" r="4" fill="#9ece6a"/>
<text x="36" y="23" fill="#a9b1d6" font-size="12" font-weight="bold">db-west-04</text>
<text x="566" y="23" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="23" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="23" fill="#565f89" font-size="11">0</text>
<text x="710" y="23" fill="#565f89" font-size="11">57.6k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="15" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="15" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="23" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 2: api-east-01 — 3 ws, 1 attention, has-attention bar -->
<g class="f6" opacity="0">
<rect y="40" width="860" height="38" fill="#24283b"/>
<rect y="40" width="3" height="38" fill="#e0af68"/>
<circle cx="22" cy="59" r="4" fill="#9ece6a"/>
<text x="36" y="63" fill="#a9b1d6" font-size="12" font-weight="bold">api-east-01</text>
<text x="566" y="63" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="63" fill="#565f89" font-size="11">0</text>
<text x="666" y="63" fill="#a9b1d6" font-size="11">1</text>
<text x="710" y="63" fill="#565f89" font-size="11">109k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="55" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="55" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="63" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 3: sre-node-03 — 2 ws, 1 running, has-running bar -->
<g class="f7" opacity="0">
<rect y="80" width="860" height="38" fill="#1a1b26"/>
<rect y="80" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="99" r="4" fill="#9ece6a"/>
<text x="36" y="103" fill="#a9b1d6" font-size="12" font-weight="bold">sre-node-03</text>
<text x="566" y="103" fill="#a9b1d6" font-size="11">2</text>
<text x="616" y="103" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="103" fill="#565f89" font-size="11">0</text>
<text x="710" y="103" fill="#565f89" font-size="11">64.4k</text>
<!-- Load bar: 2/10 = 20% -->
<rect x="770" y="95" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="95" width="12" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="103" fill="#565f89" font-size="11">20%</text>
</g>
<!-- Node 4: analytics-02 — 1 ws, thinking, has-thinking bar -->
<g class="f8" opacity="0">
<rect y="120" width="860" height="38" fill="#24283b"/>
<rect y="120" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="139" r="4" fill="#9ece6a"/>
<text x="36" y="143" fill="#a9b1d6" font-size="12" font-weight="bold">analytics-02</text>
<text x="566" y="143" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="143" fill="#565f89" font-size="11">0</text>
<text x="666" y="143" fill="#565f89" font-size="11">0</text>
<text x="710" y="143" fill="#565f89" font-size="11">18.3k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="135" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="135" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="143" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 5: data-ops-05 — 1 ws, thinking, has-thinking bar -->
<g class="f9" opacity="0">
<rect y="160" width="860" height="38" fill="#1a1b26"/>
<rect y="160" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="179" r="4" fill="#9ece6a"/>
<text x="36" y="183" fill="#a9b1d6" font-size="12" font-weight="bold">data-ops-05</text>
<text x="566" y="183" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="183" fill="#565f89" font-size="11">0</text>
<text x="666" y="183" fill="#565f89" font-size="11">0</text>
<text x="710" y="183" fill="#565f89" font-size="11">8.7k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="175" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="175" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="183" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 6: ml-gpu-07 — 0 ws, empty, no bar -->
<g class="f10" opacity="0">
<rect y="200" width="860" height="38" fill="#24283b"/>
<rect y="200" width="3" height="38" fill="transparent"/>
<circle cx="22" cy="219" r="4" fill="#9ece6a"/>
<text x="36" y="223" fill="#a9b1d6" font-size="12" font-weight="bold">ml-gpu-07</text>
<text x="566" y="223" fill="#565f89" font-size="11">0</text>
<text x="616" y="223" fill="#565f89" font-size="11">0</text>
<text x="666" y="223" fill="#565f89" font-size="11">0</text>
<text x="710" y="223" fill="#565f89" font-size="11">0</text>
<!-- Load bar: 0/10 = 0% (empty track) -->
<rect x="770" y="215" width="60" height="6" rx="3" fill="#292e42"/>
<text x="842" y="223" fill="#565f89" font-size="11">0%</text>
</g>
</g>
<!-- ====== Footer ====== -->
<g transform="translate(0, 468)" class="f12" opacity="0">
<rect width="860" height="1" fill="#3b4261"/>
<rect y="1" width="860" height="24" fill="#16161e"/>
<circle cx="20" cy="14" r="3" fill="#9ece6a"/>
<text x="28" y="18" fill="#565f89" font-size="10">db-west-04</text>
<circle cx="120" cy="14" r="3" fill="#9ece6a"/>
<text x="128" y="18" fill="#565f89" font-size="10">api-east-01</text>
<circle cx="225" cy="14" r="3" fill="#9ece6a"/>
<text x="233" y="18" fill="#565f89" font-size="10">sre-node-03</text>
<circle cx="335" cy="14" r="3" fill="#9ece6a"/>
<text x="343" y="18" fill="#565f89" font-size="10">analytics-02</text>
<circle cx="450" cy="14" r="3" fill="#9ece6a"/>
<text x="458" y="18" fill="#565f89" font-size="10">data-ops-05</text>
<circle cx="560" cy="14" r="3" fill="#9ece6a"/>
<text x="568" y="18" fill="#565f89" font-size="10">ml-gpu-07</text>
<text x="680" y="18" fill="#3b4261" font-size="10">258k tokens · 42 calls · 12m</text>
</g>
<!-- Bottom edge -->
<rect y="493" width="860" height="27" fill="#16161e"/>
<rect y="510" width="860" height="10" rx="10" fill="#16161e"/>
</svg>

Before

Width:  |  Height:  |  Size: 11 KiB

+11 -14
View File
@@ -515,8 +515,8 @@ Returns a list of all active workstreams.
```json
{
"workstreams": [
{"id": "abc123", "name": "default", "state": "idle", "session_id": "a1b2c3d4e5f6"},
{"id": "def456", "name": "hacker-news", "state": "thinking", "session_id": "c5d6e7f8a9b0"}
{"id": "abc123", "name": "default", "state": "idle"},
{"id": "def456", "name": "hacker-news", "state": "thinking"}
]
}
```
@@ -528,22 +528,21 @@ Each workstream object:
| `id` | string | Unique workstream routing identifier |
| `name` | string | Display name (alias if set, otherwise `ws-xxxx`) |
| `state` | string | Current state (see state values above) |
| `session_id` | string/null | Session ID of the workstream's `ChatSession`, used for deduplication against `/v1/api/sessions` |
---
### `GET /v1/api/sessions`
### `GET /v1/api/workstreams/saved`
Returns a list of saved sessions from the database, ordered by most recently
Returns a list of saved workstreams from the database, ordered by most recently
updated.
**Response:**
```json
{
"sessions": [
"workstreams": [
{
"session_id": "a1b2c3d4e5f6",
"ws_id": "a1b2c3d4e5f6",
"alias": "refactor",
"title": "JWT Authentication Refactor",
"created": "2026-03-01 10:00:00",
@@ -554,18 +553,16 @@ updated.
}
```
Each session object:
Each saved workstream object:
| Field | Type | Description |
|-----------------|-------------|--------------------------------------------|
| `session_id` | string | Unique 32-char hex UUID session identifier |
| `ws_id` | string | Unique workstream identifier |
| `alias` | string/null | User-assigned short name |
| `title` | string/null | LLM-generated title |
| `created` | string | ISO timestamp of session creation |
| `created` | string | ISO timestamp of workstream creation |
| `updated` | string | ISO timestamp of last message |
| `message_count` | int | Number of messages in the session |
| `node_id` | string/null | Server node that created the session |
| `ws_id` | string/null | Workstream the session belongs to |
| `message_count` | int | Number of messages in the workstream |
---
@@ -721,7 +718,7 @@ All fields are optional. The body can be empty or an empty JSON object.
| `name` | string | auto | Workstream display name |
| `model` | string | default | Model alias from the registry (`[models.*]`) |
| `auto_approve` | bool | false | Auto-approve all tool calls for this workstream |
| `resume_session` | string | "" | Session ID to resume atomically during creation (empty = fresh)|
| `resume_ws` | string | "" | Workstream ID to resume atomically during creation (empty = fresh)|
**Response (success):**
+95 -63
View File
@@ -42,7 +42,8 @@ turnstone/
__init__.py create_provider() + create_client() factory functions
workstream.py Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
tools.py Tool schema loader (JSON -> OpenAI function-calling format)
mcp_client.py MCPClientManager — MCP server connections, tool discovery, async-sync bridge
mcp_client.py MCPClientManager — MCP server connections, tool discovery, dynamic refresh, async-sync bridge
tool_search.py Dynamic tool search — BM25 index, session-scoped tool visibility
model_registry.py ModelRegistry — named model configs, lazy client creation, fallback routing
memory.py Persistence facade (delegates to storage backend)
storage/ Pluggable storage: StorageBackend protocol, SQLite + PostgreSQL
@@ -75,6 +76,7 @@ turnstone/
client.py TurnstoneClient library + TurnResult for MQ-based access
console/
collector.py ClusterCollector — aggregates state from all nodes via Redis + HTTP
scheduler.py TaskScheduler — background cron/at scheduler, dispatches via MQ
server.py Cluster dashboard HTTP server + SSE + CLI entry point
static/ Cluster dashboard web UI (page-specific HTML, CSS, JS)
channels/
@@ -93,7 +95,7 @@ turnstone/
style.css Page-specific UI styles (dashboard layout, approval blocks)
app.js Page-specific client-side JavaScript (SSE, workstreams, markdown)
tools/
*.json 14 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 15 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -471,7 +473,7 @@ independently, then returns the final content as the tool result.
- **task**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
to `.plan-<session_id>.md` — unique per `ChatSession` so concurrent workstreams
to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
don't collide. On repeat invocations the prior `plan` tool call and its result
are forwarded from `self.messages` so the agent refines the existing plan rather
than starting over. Planning instructions are injected as a developer message
@@ -496,17 +498,32 @@ bridges this with a background asyncio event loop in a daemon thread.
1. `create_mcp_client()` reads server configs from TOML or JSON
2. `MCPClientManager.start()` launches the background event loop thread
3. `_connect_all()` connects to each server (stdio subprocess or HTTP), runs
`initialize()` + `list_tools()`, converts schemas to OpenAI format
4. `ChatSession.__init__` receives the manager and builds `self._tools` (built-in + MCP)
`initialize()` + `list_tools()`, converts schemas to OpenAI format, detects
`tools.listChanged` capability for push notification support
4. `ChatSession.__init__` receives the manager, builds `self._tools` (built-in + MCP),
and registers a listener callback for tool-change notifications
5. `_prepare_tool()` routes MCP tools to `_prepare_mcp_tool()` / `_exec_mcp_tool()`
6. `_exec_mcp_tool()` calls `call_tool_sync()` which dispatches to the async loop
via `asyncio.run_coroutine_threadsafe()`
**Tool refresh:** Three mechanisms keep tools up-to-date without restart:
- **Push:** Servers declaring `tools.listChanged` send `ToolListChangedNotification`;
the registered `message_handler` triggers immediate single-server refresh.
- **Periodic:** Servers without push support are polled on a staggered interval
(default 4 h, configurable via `[mcp] refresh_interval` or `--mcp-refresh-interval`).
- **Manual:** `/mcp refresh [server]` calls `refresh_sync()` for on-demand refresh
(also attempts reconnection for disconnected servers).
When tools change, `_rebuild_tools()` creates new `_tools`/`_tool_map` objects
(copy-on-write for thread safety) and notifies listener callbacks. Each `ChatSession`
rebuilds its merged tool lists and reconstructs `ToolSearchManager` (preserving
expanded tools).
**Tool naming:** `mcp__{server}__{tool}` — double underscore delimiter, validated
at connection time (server names with `__` are rejected).
**Error isolation:** Per-server connection failures are caught and logged; other
servers still connect. Tool execution errors return error strings to the LLM
**Error isolation:** Per-server connection/refresh failures are caught and logged; other
servers are unaffected. Tool execution errors return error strings to the LLM
rather than crashing the session.
### Provider Adapter Layer
@@ -687,8 +704,11 @@ memories
created TEXT NOT NULL
updated TEXT NOT NULL
sessions
session_id TEXT PRIMARY KEY
workstreams
ws_id TEXT PRIMARY KEY
node_id TEXT NOT NULL
name TEXT NOT NULL
state TEXT NOT NULL DEFAULT 'idle'
alias TEXT UNIQUE -- user-assigned short name (nullable)
title TEXT -- LLM-generated title (nullable)
created TEXT NOT NULL
@@ -696,7 +716,7 @@ sessions
conversations
id INTEGER PRIMARY KEY AUTOINCREMENT
session_id TEXT NOT NULL
ws_id TEXT NOT NULL
timestamp TEXT NOT NULL
role TEXT NOT NULL -- user | assistant | tool_call | tool_result
content TEXT
@@ -705,8 +725,8 @@ conversations
tool_call_id TEXT -- links tool_call ↔ tool_result for resume
provider_data TEXT -- raw provider content (e.g. Anthropic encrypted)
session_config
session_id TEXT NOT NULL -- composite PK with key
workstream_config
ws_id TEXT NOT NULL -- composite PK with key
key TEXT NOT NULL
value TEXT
@@ -721,22 +741,20 @@ and are the single source of truth for both backends and Alembic migrations.
| Method | Purpose |
|--------|---------|
| `register_session(session_id, title, node_id, ws_id)` | Create a sessions row (no-op if exists) |
| `save_message(session_id, role, content, ...)` | Log a message to conversations |
| `load_session_messages(session_id)` | Reconstruct OpenAI message format from DB rows |
| `list_sessions(limit)` | List sessions with >=1 message, ordered by updated DESC |
| `delete_session(session_id)` | Delete session and all its messages |
| `prune_sessions(retention_days)` | Remove empty sessions and old unnamed sessions |
| `resolve_session(alias_or_id)` | Resolve alias, exact id, or id prefix to full session_id |
| `save_session_config(session_id, config)` | Persist session configuration key/value pairs |
| `load_session_config(session_id)` | Retrieve session configuration |
| `set_session_alias(session_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_session_name(session_id)` | Return alias if set, else title, else None |
| `update_session_title(session_id, title)` | Set/update LLM-generated title |
| `register_workstream(ws_id, node_id, name, state)` | Create a workstreams row (no-op if exists) |
| `save_message(ws_id, role, content, ...)` | Log a message to conversations |
| `load_messages(ws_id)` | Reconstruct OpenAI message format from DB rows |
| `list_workstreams_with_history(limit)` | List workstreams with >=1 message, ordered by updated DESC |
| `delete_workstream(ws_id)` | Delete workstream and cascade conversations + config |
| `prune_workstreams(retention_days)` | Remove empty workstreams and old unnamed workstreams |
| `resolve_workstream(alias_or_id)` | Resolve alias, exact id, or id prefix to full ws_id |
| `save_workstream_config(ws_id, config)` | Persist workstream configuration key/value pairs |
| `load_workstream_config(ws_id)` | Retrieve workstream configuration |
| `set_workstream_alias(ws_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_workstream_display_name(ws_id)` | Return alias if set, else title, else None |
| `update_workstream_title(ws_id, title)` | Set/update LLM-generated title |
| `update_workstream_state(ws_id, state)` | Update workstream state and bump timestamp |
| `update_workstream_name(ws_id, name)` | Update workstream display name |
| `delete_workstream(ws_id)` | Delete a workstream row |
| `list_workstreams(node_id, limit)` | List workstreams, optionally by node |
| `kv_get(key)` / `kv_set(key, value)` / `kv_delete(key)` | Generic key-value store (backs memories table) |
| `kv_list()` / `kv_search(query)` | List or search key-value pairs |
@@ -756,59 +774,59 @@ pool_size = 5 # PostgreSQL connection pool size
Environment variables: `TURNSTONE_DB_BACKEND`, `TURNSTONE_DB_URL`, `TURNSTONE_DB_PATH`.
### Session Persistence and Resume
### Persistence and Resume
Each `ChatSession` generates a full 32-char hex UUID `_session_id` on creation
and registers it in the `sessions` table with the server's `node_id` and the
owning `ws_id`. Messages are saved to `conversations` as they happen via
`save_message()`. Workstreams are persisted to the `workstreams` table on
creation, with state changes tracked via `update_workstream_state()`.
`ws_id` is the sole persistent identity for both routing and conversation
history. There is no separate `session_id` — the `workstreams` table holds
alias, title, and state alongside the routing fields (`node_id`, `name`).
Messages are saved to `conversations` (keyed by `ws_id`) as they happen
via `save_message()`. Workstream state changes are tracked via
`update_workstream_state()`.
**Auto-titling:** After the first complete exchange (user message + assistant
response), a background thread calls the LLM with a title-generation prompt
(`reasoning_effort: "low"`, `max_completion_tokens: 200`). The generated
title (3-8 words) is stored in `sessions.title`.
title (3-8 words) is stored in `workstreams.title`.
**Resume flow:** `ChatSession.resume_session(session_id)` calls
`load_session_messages()` which reconstructs the OpenAI message format from
database rows:
**Resume flow:** `ChatSession.resume(ws_id)` calls `load_messages()` which
reconstructs the OpenAI message format from database rows:
- `user` and `assistant` rows map directly
- Consecutive `tool_call` rows are grouped into one assistant message's
`tool_calls` array, paired with subsequent `tool_result` rows via
`tool_call_id` (or positional matching for legacy data)
- **Interrupted session repair:** If the last assistant message has
`tool_calls` but fewer tool results than expected (session was
- **Interrupted conversation repair:** If the last assistant message has
`tool_calls` but fewer tool results than expected (conversation was
interrupted mid-execution), the incomplete turn is stripped so the
LLM can re-generate cleanly
- The session adopts the old `_session_id`, so new messages continue in
the same session
- The `ChatSession` adopts the resumed `_ws_id`, so new messages continue
in the same workstream
**Config persistence:** LLM-affecting parameters (`temperature`,
`reasoning_effort`, `max_tokens`, `instructions`, `creative_mode`) are
persisted to the `session_config` table on creation and whenever changed
via slash commands. `resume_session()` restores these values so resumed
sessions behave identically to the original.
persisted to the `workstream_config` table on creation and whenever changed
via slash commands. `resume()` restores these values so resumed workstreams
behave identically to the original.
**`/clear` vs `/new`:** `/clear` wipes in-memory context but preserves
messages in the database for future resume. `/new` starts a fresh session
(new `_session_id`), leaving the old session resumable.
messages in the database for future resume. `/new` starts a fresh workstream
(new `_ws_id`), leaving the old workstream resumable.
**Resolution:** `resolve_session()` accepts aliases, exact session IDs, or
session ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.
**Resolution:** `resolve_workstream()` accepts aliases, exact workstream IDs,
or ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.
**Session listing:** `list_sessions()` only returns sessions that have at
least one saved message (`WHERE EXISTS` on `conversations`). Sessions
registered but never used (e.g., from process startup) are invisible until
a message is sent.
**Workstream listing:** `list_workstreams_with_history()` only returns
workstreams that have at least one saved message (`WHERE EXISTS` on
`conversations`). Workstreams registered but never used (e.g., from process
startup) are invisible until a message is sent.
**Session pruning:** `prune_sessions(retention_days, log_fn)` runs once at
startup (CLI and server). It removes:
- Sessions with no messages (orphaned registrations)
- Unnamed sessions (`alias IS NULL`) older than `retention_days` days (default 90)
**Workstream pruning:** `prune_workstreams(retention_days, log_fn)` runs once
at startup (CLI and server). It removes:
- Workstreams with no messages (orphaned registrations)
- Unnamed workstreams (`alias IS NULL`) older than `retention_days` days (default 90)
Named (aliased) sessions are never age-pruned. Configure with
`--session-retention-days N` (0 = disable age pruning).
Named (aliased) workstreams are never age-pruned. Configure with
`--retention-days N` (0 = disable age pruning).
---
@@ -946,7 +964,7 @@ Three hierarchical scopes control endpoint access:
| Scope | Grants | Endpoints |
|-------|--------|-----------|
| `read` | SSE streams, workstream listing, sessions | GET endpoints |
| `read` | SSE streams, workstream listing, history | GET endpoints |
| `write` | `read` + send, command, workstream create/close | POST to `/api/send`, `/api/command`, etc. |
| `approve` | `write` + tool approval, admin operations | POST to `/api/approve`, `/api/admin/*` |
@@ -1238,7 +1256,7 @@ typed event dataclasses.
**Two client pairs** (sync + async):
- `TurnstoneServer` / `AsyncTurnstoneServer` — server API (workstreams, chat, streaming, sessions)
- `TurnstoneServer` / `AsyncTurnstoneServer` — server API (workstreams, chat, streaming)
- `TurnstoneConsole` / `AsyncTurnstoneConsole` — console API (cluster overview, nodes, workstreams)
**Design**: async-first with thin sync wrappers. `_BaseClient` provides httpx
@@ -1278,12 +1296,26 @@ The `ChannelRouter` manages bidirectional routing: it maps platform
channel/thread IDs to turnstone workstream IDs, handles workstream
creation and stale-route recovery, and resolves platform users to
turnstone identities via the `channel_users` table. When an evicted
workstream is reactivated, the router uses atomic session resume via the
`resume_session` field on `CreateWorkstreamMessage` — the server resumes
the old session during workstream creation in a single HTTP request,
eliminating ordering fragility. The bridge emits a `SessionResumedEvent`
to confirm success.
workstream is reactivated, the router uses atomic resume via the
`resume_ws` field on `CreateWorkstreamMessage` — the server resumes
the old workstream's conversation during creation in a single HTTP
request, eliminating ordering fragility. The bridge emits a
`WorkstreamResumedEvent` to confirm success.
Discord ships as the first adapter. See [channels.md](channels.md) for
setup instructions, configuration reference, and the adapter development
guide.
### Notification Subsystem
The `notify` tool enables the LLM to send notifications to users or
channels without going through MQ. The server calls the channel gateway
directly over HTTP for lower latency: `_exec_notify()` queries the
`services` database table for healthy channel gateways (heartbeat within
120 seconds), authenticates with a service JWT (`aud: turnstone-channel`),
and POSTs to `POST /v1/api/notify` on the first healthy gateway. The
gateway validates the JWT, resolves the target (username lookup via
`channel_users` or direct `channel_type`+`channel_id`), and delegates to
the appropriate `ChannelAdapter.send()`. Delivery retries up to 3 times
with backoff, re-querying the service registry on each attempt. See
[Notification Flow diagram](diagrams/png/17-notify-flow.png).
+87 -13
View File
@@ -136,11 +136,11 @@ An admin can also force-link or unlink users via the console admin panel
1.5 seconds.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous session via the `resume_session` field on
`CreateWorkstreamMessage`. The server resumes the session during
workstream creation (same HTTP request), and the bridge emits a
`SessionResumedEvent` back to the channel. The thread receives a
*"Session resumed: {name} ({count} messages restored)"* confirmation.
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the bridge emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
### Slash Commands
@@ -198,6 +198,9 @@ Plan review requests are displayed as a blue embed with:
| `--redis-db` | — | `0` | Redis DB number |
| `--model` | — | server default | Default model for new workstreams |
| `--auto-approve` | — | `false` | Auto-approve ALL tool calls (skips approval buttons entirely) |
| `--http-host` | — | `127.0.0.1` | HTTP server bind address for notify endpoint |
| `--http-port` | `TURNSTONE_CHANNEL_PORT` | `8091` | HTTP server port |
| `--auth-token` | `TURNSTONE_CHANNEL_AUTH_TOKEN` | — | Static auth token for `/v1/api/notify` (alternative to JWT) |
| `--log-level` | `TURNSTONE_LOG_LEVEL` | `INFO` | Log level |
| `--log-format` | `TURNSTONE_LOG_FORMAT` | `auto` | Log format (`auto`/`json`/`text`) |
@@ -229,19 +232,90 @@ See [Security: Database Schema](security.md#database-schema) for the
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route (no MQ owner), looks up the old session via
`get_session_id_by_ws()`, and creates a new workstream with
`resume_session` set atomically on the `CreateWorkstreamMessage`. The
server resumes the session during creation (no separate command
needed). The bridge emits a `SessionResumedEvent` to the channel, and
the thread displays *"Session resumed: {name} ({count} messages
restored)"*. If the old session was pruned, the workstream starts
fresh with no error.
route (no MQ owner) and creates a new workstream with the old `ws_id`
as `resume_ws` on the `CreateWorkstreamMessage`. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The bridge emits a `WorkstreamResumedEvent` to the channel, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
5. **Close**`/close` command closes the workstream via MQ, deletes the
route, unsubscribes from events, and archives the Discord thread.
---
## Notifications
> See also: [Notification Flow diagram](diagrams/png/17-notify-flow.png)
The `notify` tool allows the LLM to proactively send notifications to
users or channels on external platforms. This is useful for alerting
people about task completion, errors, or important updates without
waiting for them to check in.
### Targeting
Two modes:
- **Username** — provide a turnstone `username`. The gateway resolves
it via the `channel_users` table and sends to all linked channels
(e.g. Discord + future Slack).
- **Direct** — provide `channel_type` + `channel_id` to target a
specific platform channel or user DM.
### Delivery Flow
Notifications bypass MQ for lower latency. The server calls the channel
gateway directly over HTTP:
1. The LLM calls the `notify` tool with a message and target
2. `_exec_notify()` queries the `services` table for healthy channel
gateways (heartbeat within the last 120 seconds)
3. The server mints a service JWT (`aud: turnstone-channel`) via
`ServiceTokenManager` and POSTs to the first healthy gateway
4. The gateway validates the JWT, resolves the target, and calls
`adapter.send()` on the appropriate platform adapter
5. On failure, the server tries the next gateway. If all fail, it
retries up to 2 more times (delays: 1s, 3s), re-querying the
service registry on each attempt
### Service Registry
The channel gateway registers itself in the `services` database table
on startup and sends a heartbeat every 30 seconds. On shutdown it
deregisters. Services are considered stale after 120 seconds (4 missed
heartbeats) and are excluded from `list_services()` queries.
The `services` table schema:
| Column | Description |
|--------|-------------|
| `service_type` | Service category (e.g. `"channel"`) |
| `service_id` | Unique instance ID (`channel-<hostname>-<random>`) |
| `url` | HTTP base URL for the service |
| `last_heartbeat` | ISO 8601 timestamp of last heartbeat |
| `created` | ISO 8601 timestamp of initial registration |
### Security
- **Authentication** — the gateway's `POST /v1/api/notify` endpoint
requires authentication. Configure either `TURNSTONE_JWT_SECRET`
(the server mints JWTs with `aud: turnstone-channel` automatically)
or a static token via `--auth-token`. If neither is set, the
gateway fails closed and rejects all requests with 401. Server JWTs
(`aud: turnstone-server`) are rejected.
- **Rate limit** — maximum 5 notifications per turn. The counter only
increments on successful delivery, so failures don't consume the
budget.
- **SSRF protection** — only `http://` and `https://` service URLs
are allowed. Other schemes are silently skipped.
- **Mention sanitization** — `discord.utils.escape_mentions()` is
applied before sending, preventing `@everyone` / `@here` abuse.
- **Error redaction** — generic error messages are returned to the
LLM. Internal details (service IDs, URLs, exception messages) are
logged server-side only.
---
## Adding New Adapters
The `ChannelAdapter` protocol defines the interface any platform adapter
+145 -1
View File
@@ -326,7 +326,7 @@ The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/share
### SSE Proxy
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied by creating a per-connection `httpx.AsyncClient(timeout=None)`, streaming the upstream response via `aiter_text()`, parsing SSE framing (`\n\n` delimiters), and re-emitting events through `EventSourceResponse`. Each proxied SSE stream requires its own httpx client since the shared client's 30-second timeout would kill long-lived connections.
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
### Authentication
@@ -423,6 +423,150 @@ to create the initial admin user and receive a JWT in one step. See
---
## Scheduled Tasks
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via the MQ broker. It supports cron-based recurring schedules and one-shot `at` schedules.
### Architecture
The scheduler runs as a daemon thread inside the console process. Every `check_interval` seconds (default 15) it:
1. Acquires a distributed lock via Redis `SET NX EX` (prevents duplicate dispatch in multi-console deployments)
2. Queries the storage backend for tasks whose `next_run <= now` and `enabled = true`
3. Dispatches each due task as one or more `CreateWorkstreamMessage` via MQ
4. Updates `last_run` and computes the next `next_run` (or disables one-shot `at` tasks)
5. Releases the lock via Lua script (safe conditional delete)
Run history is automatically pruned (runs older than 90 days) approximately once per hour.
### Schedule Types
| Type | Field | Behavior |
|------|-------|----------|
| `cron` | `cron_expr` | Recurring schedule using standard 5-field cron syntax. Requires `croniter`. |
| `at` | `at_time` | One-shot: fires once at the given ISO 8601 timestamp (must include timezone), then auto-disables. |
### Target Modes
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Pushes to the shared inbound queue (any bridge picks it up) |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
### Configuration
| Parameter | Default | Description |
|-----------|---------|-------------|
| `check_interval` | `15.0` | Seconds between scheduler ticks |
| `lock_ttl` | `60` | Distributed lock TTL in seconds |
| `max_fan_out` | `20` | Maximum nodes for `all` target mode |
Dependency: `croniter` (installed with turnstone).
### Schedule API
All schedule endpoints require `approve` scope. Maximum 200 schedules.
#### `GET /v1/api/admin/schedules`
List all scheduled tasks.
```json
{
"schedules": [
{
"task_id": "a1b2c3d4",
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"auto_approve_tools": [],
"enabled": true,
"created_by": "u_admin",
"last_run": "2026-03-05T02:00:00Z",
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-01T12:00:00Z",
"updated": "2026-03-05T02:00:01Z"
}
]
}
```
#### `POST /v1/api/admin/schedules`
Create a scheduled task.
Request:
```json
{
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"target_mode": "auto",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"enabled": true
}
```
Required fields: `name`, `schedule_type`, `initial_message`. For `cron` schedules provide `cron_expr`; for `at` schedules provide `at_time` (ISO 8601 with timezone, must be in the future).
Response: `ScheduleInfo` (same shape as list items above). Returns `400` for invalid cron syntax, naive timestamps, or past `at_time`. Returns `409` if the 200-schedule cap is reached.
#### `GET /v1/api/admin/schedules/{task_id}`
Get a single scheduled task. Returns `ScheduleInfo` or `404`.
#### `PUT /v1/api/admin/schedules/{task_id}`
Partial update — only include fields to change. If `schedule_type`, `cron_expr`, or `at_time` change, `next_run` is recomputed automatically.
```json
{
"enabled": false
}
```
Response: updated `ScheduleInfo`. Returns `400` for validation errors, `404` if not found.
#### `DELETE /v1/api/admin/schedules/{task_id}`
Delete a scheduled task and all its run history. Returns `{"status": "ok"}` or `404`.
#### `GET /v1/api/admin/schedules/{task_id}/runs?limit=50`
List execution history for a task (most recent first). `limit` defaults to 50, max 200.
```json
{
"runs": [
{
"run_id": "r_abc123",
"task_id": "a1b2c3d4",
"node_id": "db-west-04",
"ws_id": "ws_xyz",
"correlation_id": "corr_789",
"started": "2026-03-05T02:00:00Z",
"status": "dispatched",
"error": ""
}
]
}
```
Status is `dispatched` on success or `failed` with an `error` message (e.g. no reachable nodes). Failed runs do not advance `next_run`.
---
## CLI Commands
The `/cluster` command in the turnstone CLI queries the console's HTTP API. Requires `--console-url` or `[console] url` in config.toml.
+4 -2
View File
@@ -40,7 +40,8 @@ package "turnstone/core/" <<Rectangle>> {
component [auth.py\nAuthentication] as auth <<core>>
component [healthcheck.py\nBackendHealthMonitor] as healthcheck <<core>>
component [ratelimit.py\nRateLimiter] as ratelimit <<core>>
component [mcp_client.py\nMCPClientManager] as mcp <<core>>
component [mcp_client.py\nMCPClientManager\n(push + periodic refresh)] as mcp <<core>>
component [tool_search.py\nToolSearchManager, BM25] as toolsearch <<core>>
component [model_registry.py\nModelRegistry] as registry <<core>>
}
@@ -95,7 +96,7 @@ package "turnstone/sdk/" <<Rectangle>> {
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\n14 tool schemas] as schemas <<artifact>>
component [*.json\n15 tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
@@ -136,6 +137,7 @@ session --> edit
session --> web
session --> healthcheck
session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
healthcheck --> metrics
+37 -2
View File
@@ -108,6 +108,7 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ thinking_mode: str
+ supports_effort: bool
+ supports_web_search: bool
+ supports_tool_search: bool
}
' ChatSession
@@ -118,8 +119,9 @@ class "ChatSession" as ChatSession {
- ui: SessionUI
- messages: list[dict]
- _msg_tokens: list[int]
- _session_id: str
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
+ model_alias: str | None {property}
- _tools: list[dict]
@@ -130,7 +132,7 @@ class "ChatSession" as ChatSession {
--
+ send(user_input: str)
+ handle_command(command: str)
+ resume_session(session_id: str)
+ resume(ws_id: str)
- _save_config()
- _stream_response(stream) → dict
- _create_stream_with_retry(msgs) → Stream (+ fallback)
@@ -139,6 +141,12 @@ class "ChatSession" as ChatSession {
- _prepare_tool(tc) → item dict
- _prepare_mcp_tool(call_id, name, args) → item dict
- _exec_mcp_tool(item) → (call_id, output)
- _get_active_tools() → list[dict]
- _prepare_tool_search() → None
- _exec_tool_search(item) → (call_id, output)
- _on_mcp_tools_changed()
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
@@ -201,22 +209,48 @@ enum "WorkstreamState" as WsState {
' MCPClientManager
class "MCPClientManager" as MCPMgr {
- _sessions: dict[str, ClientSession]
- _per_server_tools: dict[str, list[dict]]
- _tools: list[dict]
- _tool_map: dict[str, tuple]
- _supports_list_changed: dict[str, bool]
- _listeners: list[Callable]
--
+ start()
+ get_tools() → list[dict]
+ is_mcp_tool(name) → bool
+ call_tool_sync(name, args) → str
+ refresh_sync(server?) → dict
+ add_listener(callback)
+ remove_listener(callback)
+ server_names: list[str] {property}
+ shutdown()
--
Background asyncio event loop
bridges async MCP SDK to
sync ChatSession dispatch.
Push + periodic + manual refresh.
--
core/mcp_client.py
}
' ToolSearchManager
class "ToolSearchManager" as ToolSearchMgr {
- _all_tools: list[dict]
- _always_on: list[dict]
- _deferred: list[dict]
- _expanded: dict[str, None]
- _index: BM25Index
--
+ should_activate() → bool
+ get_visible_tools() → list[dict]
+ get_deferred_tools() → list[dict]
+ get_expanded_names() → list[str]
+ search(query, k) → list[dict]
+ expand_visible(names) → list[dict]
+ get_search_tool_definition() → dict
+ format_search_results(tools) → str
}
' ModelRegistry
class "ModelRegistry" as ModelReg {
- _models: dict[str, ModelConfig]
@@ -317,6 +351,7 @@ LLMProvider <|.. AnthropicProv
ChatSession --> SessionUI : uses
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
+4 -4
View File
@@ -18,7 +18,7 @@ User -> CS : send(user_input)
activate CS
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(session_id, "user", input)
CS -> DB : save_message(ws_id, "user", input)
== LLM Call Loop ==
@@ -65,8 +65,8 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> DB : save_message(session_id, "assistant", content)
CS -> DB : save_message(session_id, "tool_call", ...) ×N
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
@@ -136,7 +136,7 @@ group loop [while tool_calls present]
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(session_id, "tool_result", ...)
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
+25 -21
View File
@@ -24,27 +24,29 @@ partition "Phase 1: Prepare" #E8F5E9 {
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (14 tools):**
┌─────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├─────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✓ Yes │
│ man │ ✗ Auto-approve │
│ web_fetch │ ✓ Yes │
│ web_search │ ✓ Yes │
│ task │ ✓ Yes
plan │ ✓ Yes │
remember │ ✗ Auto-approve
│ recall │ ✗ Auto-approve │
forget │ ✗ Auto-approve │
├─────────────┼──────────────────┤
mcp__*✓ Yes (external)
───────────────────────────────
**Dispatch table (16 tools):**
┌─────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├─────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✓ Yes │
│ man │ ✗ Auto-approve │
│ web_fetch │ ✓ Yes │
│ web_search │ ✓ Yes │
│ tool_search │ ✗ Auto-approve
task │ ✓ Yes │
plan │ ✓ Yes
│ remember │ ✗ Auto-approve │
recall │ ✗ Auto-approve │
│ forget │ ✗ Auto-approve │
notify ✗ Auto-approve
├────────────────────────────────
│ mcp__* │ ✓ Yes (external) │
└──────────────┴──────────────────┘
end note
:Build item dict:
@@ -106,8 +108,10 @@ partition "Phase 3: Execute" #E3F2FD {
├─ _exec_man: man/info subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: Tavily API POST (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_plan: _run_agent(AGENT_TOOLS, read-only)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_remember: SQLite INSERT OR REPLACE
├─ _exec_recall: SQLite FTS5/LIKE search
├─ _exec_forget: SQLite DELETE
+1 -1
View File
@@ -35,7 +35,7 @@ package "turnstone/sdk/ (Python)" {
+ stream_events(ws_id)
+ stream_global_events()
+ send_and_wait()
+ list_sessions()
+ list_saved_workstreams()
+ login() / logout()
+ health()
}
+13 -18
View File
@@ -13,23 +13,19 @@ skinparam class {
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+register_session(session_id, title, node_id, ws_id)
+save_message(session_id, role, content, ...)
+load_session_messages(session_id) → list[dict]
+list_sessions(limit) → list
+delete_session(session_id) → bool
+prune_sessions(retention_days) → (int, int)
+resolve_session(alias_or_id) → str | None
+save_session_config(session_id, config)
+load_session_config(session_id) → dict
+set_session_alias(session_id, alias) → bool
+get_session_name(session_id) → str | None
+update_session_title(session_id, title)
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
@@ -68,9 +64,8 @@ class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+sessions: Table (node_id, ws_id, user_id)
+workstreams: Table (node_id, user_id, state)
+session_config: Table
+workstreams: Table (node_id, alias, title, state)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
@@ -106,14 +101,14 @@ class "_registry.py" as Registry {
' -- Facade --
class "memory.py" as Facade <<facade>> {
+register_session()
+save_message()
+load_session_messages()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all 22 functions)
+... (all delegated functions)
--
Thin delegation to
get_storage()
+43 -4
View File
@@ -44,11 +44,15 @@ class "turnstone-channel" as ChannelService <<service>> {
asyncio event loop
Structured logging (structlog)
--log-level, --log-format
--
POST /v1/api/notify (HTTP)
GET /health
}
class "DiscordBot" as Bot <<service>> {
+on_message(msg)
+on_interaction(interaction)
+send(channel_id, content)
+run(token)
--
discord.py Client
@@ -56,6 +60,7 @@ class "DiscordBot" as Bot <<service>> {
Sends replies + embeds
Creates threads for workstreams
Renders approval buttons
escape_mentions() on send
}
class "ChannelRouter" as Router <<service>> {
@@ -109,6 +114,9 @@ class "turnstone-server" as Server <<server>> {
--
LLM execution + tool use
SSE event stream
--
notify tool: _exec_notify()
ServiceTokenManager (JWT)
}
' -- Storage --
@@ -134,6 +142,18 @@ class "channel_routes" as CR <<storage>> {
to turnstone workstreams
}
class "services" as SVC <<storage>> {
service_type (PK)
service_id (PK)
url
last_heartbeat
created
--
Heartbeat every 30s
Stale after 120s
ON CONFLICT DO UPDATE
}
' -- Relationships --
Discord --> Bot : gateway\nevents
Bot --> Router : on_message\non_interaction
@@ -157,6 +177,11 @@ Teams .[hidden]. Slack
ChannelService --> Bot : creates + runs
ChannelService --> Router : creates
ChannelService --> Broker : creates
ChannelService --> SVC : register / heartbeat /\nderegister
' -- Notification path (direct HTTP, bypasses MQ) --
Server --> ChannelService : POST /v1/api/notify\n(JWT: aud=turnstone-channel)
Server --> SVC : list_services("channel",\nmax_age_seconds=120)
' -- Notes --
note right of Bot
@@ -170,13 +195,13 @@ note right of Bot
5. Broker.push_inbound(SendMessage)
6. Bridge pops from Redis, drives server
**Session Resume (evicted workstreams)**
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no MQ owner)
2. Old session looked up via get_session_id_by_ws()
2. Existing ws_id reused directly from route
3. CreateWorkstreamMessage sent with
resume_session=<old_session_id>
resume_ws=<ws_id>
4. Server resumes atomically during creation
5. Bridge emits SessionResumedEvent → thread
5. Bridge emits WorkstreamResumedEvent → thread
end note
note right of Broker
@@ -208,4 +233,18 @@ note bottom of CU
7. AuthResult scopes applied by server
end note
note bottom of SVC
**Notification Flow** (direct HTTP, bypasses MQ)
1. LLM calls notify tool → _prepare_notify()
2. _exec_notify() checks rate limit (5/turn)
3. Queries services table for healthy gateways
4. Mints JWT (aud: turnstone-channel) via
ServiceTokenManager
5. POSTs to first healthy gateway
6. Gateway validates JWT, resolves target
7. adapter.send() → Discord API
8. On failure: retry up to 3× (1s, 3s backoff)
9. SSRF: only http(s) URLs allowed
end note
@enduml
+116
View File
@@ -0,0 +1,116 @@
@startuml
!theme plain
title Turnstone — Notification Delivery Flow
skinparam participant {
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<service>> #E8EAF6
BackgroundColor<<platform>> #E1BEE7
}
participant "ChatSession\n(turnstone-server)" as Session <<server>>
participant "StorageBackend" as Storage <<storage>>
participant "ServiceTokenManager" as STM <<server>>
participant "Channel Gateway\n(_http.py)" as Gateway <<service>>
participant "ChannelAdapter\n(Discord bot)" as Adapter <<service>>
participant "Discord API" as Discord <<platform>>
== Prepare Phase ==
Session -> Session : _prepare_notify(call_id, args)
note right
Validates:
- message (required, ≤2000 chars)
- target: username OR channel_type+channel_id
- no ambiguous targeting (both set)
- partial targeting errors
end note
== Execute Phase ==
Session -> Session : _exec_notify(item)
Session -> Session : check rate limit\n(≥5 per turn?)
alt rate limit exceeded
Session --> Session : "Error: rate limit exceeded"
end
loop up to 3 attempts (retry delays: 1s, 3s)
Session -> Storage : list_services("channel",\nmax_age_seconds=120)
Storage --> Session : services[] (sorted by\nlast_heartbeat DESC)
alt no healthy services
Session -> Session : log.warning("notify.no_services")
Session -> Session : sleep(delay)
else services available
Session -> STM : bearer_header
note right
Lazy-init ServiceTokenManager
aud: turnstone-channel
scope: write
Auto-rotates 1h JWTs
end note
STM --> Session : Authorization: Bearer <jwt>
loop for each gateway (first-healthy)
Session -> Session : SSRF check:\nurl.startswith("http://"|"https://")
Session -> Gateway : POST /v1/api/notify\n+ Authorization header
Gateway -> Gateway : _check_auth()\nvalidate JWT (aud=turnstone-channel)\nor static token
alt auth failed
Gateway --> Session : 401 Unauthorized
else auth ok
alt username target
Gateway -> Storage : get_user_by_username()
Storage --> Gateway : user
Gateway -> Storage : list_channel_users_by_user()
Storage --> Gateway : linked channels
else direct target
Gateway -> Gateway : use channel_type + channel_id
end
Gateway -> Adapter : send(channel_id, content)
note right
escape_mentions() applied
Chunked for 2000-char limit
end note
Adapter -> Discord : POST message
Discord --> Adapter : message_id
Adapter --> Gateway : message_id
Gateway --> Session : 200 {results: [{status: "sent"}]}
Session -> Session : _notify_count += 1
Session --> Session : "Notification sent successfully"
note right : Return — no further\ngateways tried
end
end
alt all gateways failed
Session -> Session : log.warning(\n"notify.all_gateways_failed")
Session -> Session : sleep(delay)
end
end
end
alt all retries exhausted
Session -> Session : log.warning("notify.delivery_failed")
Session --> Session : "Error: notification delivery failed"
end
== Service Registry (Background) ==
note over Gateway, Storage
**Heartbeat Lifecycle**
1. Gateway startup: register_service("channel", id, url)
2. Every 30s: heartbeat_service("channel", id)
3. Shutdown: deregister_service("channel", id)
4. Stale after 120s (4 missed heartbeats)
end note
@enduml
+286
View File
@@ -0,0 +1,286 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 540" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif">
<defs>
<!-- Arrowhead markers -->
<marker id="arrow" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#484f58"/>
</marker>
<marker id="arrow-blue" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#58a6ff"/>
</marker>
<marker id="arrow-green" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#3fb950"/>
</marker>
<marker id="arrow-orange" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#f0883e"/>
</marker>
<marker id="arrow-coral" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#f47067"/>
</marker>
<marker id="arrow-muted" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#8b949e"/>
</marker>
<!-- Card shadow filter -->
<filter id="shadow" x="-4%" y="-4%" width="108%" height="112%">
<feDropShadow dx="0" dy="1" stdDeviation="2" flood-color="#000" flood-opacity="0.4"/>
</filter>
</defs>
<!-- Background -->
<rect width="1200" height="540" rx="8" fill="#0d1117"/>
<!-- Title -->
<text x="600" y="36" text-anchor="middle" fill="#e6edf3" font-size="15" font-weight="700" letter-spacing="3">TURNSTONE</text>
<text x="600" y="54" text-anchor="middle" fill="#8b949e" font-size="11" letter-spacing="1">SYSTEM ARCHITECTURE</text>
<!-- ==================== COLUMN HEADERS ==================== -->
<text x="90" y="86" text-anchor="middle" fill="#58a6ff" font-size="9" font-weight="600" letter-spacing="2">CLIENTS</text>
<text x="276" y="86" text-anchor="middle" fill="#3fb950" font-size="9" font-weight="600" letter-spacing="2">GATEWAYS</text>
<text x="480" y="86" text-anchor="middle" fill="#f0883e" font-size="9" font-weight="600" letter-spacing="2">MESSAGE QUEUE</text>
<text x="700" y="86" text-anchor="middle" fill="#f47067" font-size="9" font-weight="600" letter-spacing="2">CLUSTER NODES</text>
<text x="940" y="86" text-anchor="middle" fill="#f778ba" font-size="9" font-weight="600" letter-spacing="2">LLM PROVIDERS</text>
<!-- ==================== CLIENT BOXES ==================== -->
<!-- CLI -->
<g filter="url(#shadow)">
<rect x="30" y="108" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="108" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="108" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="111" width="120" height="2" fill="#161b22"/>
<text x="90" y="130" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">CLI</text>
<text x="90" y="145" text-anchor="middle" fill="#8b949e" font-size="9">terminal REPL</text>
</g>
<!-- Browser UI -->
<g filter="url(#shadow)">
<rect x="30" y="174" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="174" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="174" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="177" width="120" height="2" fill="#161b22"/>
<text x="90" y="196" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Browser UI</text>
<text x="90" y="211" text-anchor="middle" fill="#8b949e" font-size="9">HTTP + SSE</text>
</g>
<!-- SDK / API -->
<g filter="url(#shadow)">
<rect x="30" y="244" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="244" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="244" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="247" width="120" height="2" fill="#161b22"/>
<text x="90" y="266" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">SDK / API</text>
<text x="90" y="281" text-anchor="middle" fill="#8b949e" font-size="9">programmatic</text>
</g>
<!-- Discord / Slack -->
<g filter="url(#shadow)">
<rect x="30" y="314" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="314" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="314" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="317" width="120" height="2" fill="#161b22"/>
<text x="90" y="336" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Discord / Slack</text>
<text x="90" y="351" text-anchor="middle" fill="#8b949e" font-size="9">chat platforms</text>
</g>
<!-- ==================== GATEWAY BOXES ==================== -->
<!-- Console -->
<g filter="url(#shadow)">
<rect x="216" y="118" width="120" height="58" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="216" y="118" width="120" height="5" rx="5" fill="#3fb950"/>
<rect x="216" y="118" width="120" height="5" fill="#3fb950"/>
<rect x="216" y="121" width="120" height="2" fill="#161b22"/>
<text x="276" y="142" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<text x="276" y="157" text-anchor="middle" fill="#8b949e" font-size="9">dashboard + proxy</text>
<text x="276" y="169" text-anchor="middle" fill="#8b949e" font-size="9">cluster management</text>
</g>
<!-- Channel Gateway -->
<g filter="url(#shadow)">
<rect x="216" y="292" width="120" height="58" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="216" y="292" width="120" height="5" rx="5" fill="#3fb950"/>
<rect x="216" y="292" width="120" height="5" fill="#3fb950"/>
<rect x="216" y="295" width="120" height="2" fill="#161b22"/>
<text x="276" y="316" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Channel Gateway</text>
<text x="276" y="331" text-anchor="middle" fill="#8b949e" font-size="9">platform adapter</text>
<text x="276" y="343" text-anchor="middle" fill="#8b949e" font-size="9">Discord, Slack, ...</text>
</g>
<!-- ==================== REDIS MQ ==================== -->
<g filter="url(#shadow)">
<rect x="420" y="168" width="120" height="132" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="420" y="168" width="120" height="5" rx="5" fill="#f0883e"/>
<rect x="420" y="168" width="120" height="5" fill="#f0883e"/>
<rect x="420" y="171" width="120" height="2" fill="#161b22"/>
<text x="480" y="198" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Redis MQ</text>
<line x1="438" y1="210" x2="522" y2="210" stroke="#30363d" stroke-width="1"/>
<text x="480" y="228" text-anchor="middle" fill="#8b949e" font-size="9">inbound queues</text>
<text x="480" y="243" text-anchor="middle" fill="#8b949e" font-size="9">event pub/sub</text>
<text x="480" y="258" text-anchor="middle" fill="#8b949e" font-size="9">node heartbeats</text>
<text x="480" y="273" text-anchor="middle" fill="#8b949e" font-size="9">workstream routing</text>
<text x="480" y="288" text-anchor="middle" fill="#8b949e" font-size="9">cluster state</text>
</g>
<!-- ==================== CLUSTER NODES ==================== -->
<!-- Cluster outline -->
<rect x="598" y="100" width="204" height="310" rx="8" fill="none" stroke="#30363d" stroke-width="1" stroke-dasharray="4,3"/>
<text x="700" y="422" text-anchor="middle" fill="#30363d" font-size="9" letter-spacing="1">CLUSTER</text>
<!-- Node A -->
<g filter="url(#shadow)">
<rect x="614" y="118" width="170" height="100" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="118" width="170" height="5" rx="5" fill="#f47067"/>
<rect x="614" y="118" width="170" height="5" fill="#f47067"/>
<rect x="614" y="121" width="170" height="2" fill="#161b22"/>
<text x="699" y="142" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Node A</text>
<line x1="632" y1="152" x2="766" y2="152" stroke="#30363d" stroke-width="1"/>
<!-- Bridge -->
<rect x="626" y="162" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="661" y="180" text-anchor="middle" fill="#8b949e" font-size="9">bridge</text>
<!-- Server -->
<rect x="704" y="162" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="739" y="180" text-anchor="middle" fill="#8b949e" font-size="9">server</text>
<!-- Arrow bridge to server -->
<line x1="696" y1="176" x2="702" y2="176" stroke="#484f58" stroke-width="1" marker-end="url(#arrow)"/>
<!-- Tools label -->
<text x="699" y="206" text-anchor="middle" fill="#484f58" font-size="8">14 tools + MCP</text>
</g>
<!-- Node B -->
<g filter="url(#shadow)">
<rect x="614" y="238" width="170" height="100" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="238" width="170" height="5" rx="5" fill="#f47067"/>
<rect x="614" y="238" width="170" height="5" fill="#f47067"/>
<rect x="614" y="241" width="170" height="2" fill="#161b22"/>
<text x="699" y="262" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Node B</text>
<line x1="632" y1="272" x2="766" y2="272" stroke="#30363d" stroke-width="1"/>
<!-- Bridge -->
<rect x="626" y="282" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="661" y="300" text-anchor="middle" fill="#8b949e" font-size="9">bridge</text>
<!-- Server -->
<rect x="704" y="282" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="739" y="300" text-anchor="middle" fill="#8b949e" font-size="9">server</text>
<!-- Arrow bridge to server -->
<line x1="696" y1="296" x2="702" y2="296" stroke="#484f58" stroke-width="1" marker-end="url(#arrow)"/>
<!-- Tools label -->
<text x="699" y="326" text-anchor="middle" fill="#484f58" font-size="8">14 tools + MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
<!-- OpenAI -->
<g filter="url(#shadow)">
<rect x="870" y="130" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="130" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="130" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="133" width="140" height="2" fill="#161b22"/>
<text x="940" y="153" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">OpenAI</text>
<text x="940" y="167" text-anchor="middle" fill="#8b949e" font-size="9">GPT-5, o-series</text>
</g>
<!-- Anthropic -->
<g filter="url(#shadow)">
<rect x="870" y="196" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="196" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="196" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="199" width="140" height="2" fill="#161b22"/>
<text x="940" y="219" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Anthropic</text>
<text x="940" y="233" text-anchor="middle" fill="#8b949e" font-size="9">Claude 4.5 / 4.6</text>
</g>
<!-- Local / vLLM -->
<g filter="url(#shadow)">
<rect x="870" y="262" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="262" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="262" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="265" width="140" height="2" fill="#161b22"/>
<text x="940" y="285" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Local / vLLM</text>
<text x="940" y="299" text-anchor="middle" fill="#8b949e" font-size="9">llama.cpp, NIM</text>
</g>
<!-- ==================== STORAGE ==================== -->
<g filter="url(#shadow)">
<rect x="614" y="450" width="170" height="52" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="450" width="170" height="5" rx="5" fill="#bc8cff"/>
<rect x="614" y="450" width="170" height="5" fill="#bc8cff"/>
<rect x="614" y="453" width="170" height="2" fill="#161b22"/>
<text x="699" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="699" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
</g>
<text x="699" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
<!-- ==================== CONNECTION LINES ==================== -->
<!-- CLIENT -> GATEWAY connections -->
<!-- Browser -> Console -->
<line x1="150" y1="197" x2="214" y2="155" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- Discord -> Channel -->
<line x1="150" y1="337" x2="214" y2="325" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- CLI -> direct to Node A server (top path, curved) -->
<path d="M 150 131 C 200 131, 200 100, 400 100 L 400 100 C 500 100, 570 140, 612 168" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.5" fill="none" stroke-dasharray="6,3" marker-end="url(#arrow-blue)"/>
<text x="370" y="96" fill="#484f58" font-size="8" text-anchor="middle">direct</text>
<!-- SDK -> Redis (direct push) -->
<line x1="150" y1="267" x2="418" y2="240" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- GATEWAY -> REDIS connections -->
<!-- Console -> Redis -->
<line x1="336" y1="160" x2="418" y2="200" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-green)"/>
<!-- Channel -> Redis -->
<line x1="336" y1="318" x2="418" y2="272" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-green)"/>
<!-- REDIS -> NODE connections -->
<!-- Redis -> Node A bridge -->
<line x1="540" y1="210" x2="624" y2="176" stroke="#f0883e" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-orange)"/>
<!-- Redis -> Node B bridge -->
<line x1="540" y1="260" x2="624" y2="296" stroke="#f0883e" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-orange)"/>
<!-- Console -> Node (proxy, dashed) -->
<path d="M 336 147 C 380 130, 500 108, 612 145" stroke="#3fb950" stroke-width="1" stroke-opacity="0.4" fill="none" stroke-dasharray="4,3" marker-end="url(#arrow-green)"/>
<text x="468" y="120" fill="#484f58" font-size="8" text-anchor="middle">proxy</text>
<!-- NODE -> LLM connections -->
<!-- Node A -> LLM providers -->
<line x1="784" y1="168" x2="868" y2="155" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.5" marker-end="url(#arrow-coral)"/>
<line x1="784" y1="176" x2="868" y2="219" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.3"/>
<line x1="784" y1="180" x2="868" y2="282" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.2"/>
<!-- Node B -> LLM providers -->
<line x1="784" y1="288" x2="868" y2="163" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.2"/>
<line x1="784" y1="296" x2="868" y2="222" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.3"/>
<line x1="784" y1="300" x2="868" y2="288" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.5" marker-end="url(#arrow-coral)"/>
<!-- NODE -> STORAGE connections -->
<line x1="680" y1="338" x2="680" y2="448" stroke="#bc8cff" stroke-width="1.2" stroke-opacity="0.4" stroke-dasharray="4,3" marker-end="url(#arrow-muted)"/>
<line x1="718" y1="218" x2="718" y2="236" stroke="#484f58" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="2,2"/>
<!-- Extensibility hint -->
<text x="699" y="392" text-anchor="middle" fill="#30363d" font-size="10">...</text>
<!-- Event flow: Bridges -> Redis (dashed, bidirectional feel) -->
<line x1="624" y1="186" x2="542" y2="220" stroke="#f0883e" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="3,3"/>
<line x1="624" y1="286" x2="542" y2="250" stroke="#f0883e" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="3,3"/>
<text x="574" y="242" fill="#484f58" font-size="7" text-anchor="middle">events</text>
<!-- ==================== FLOW LABELS ==================== -->
<!-- Interactive flow label -->
<rect x="30" y="395" width="10" height="10" rx="2" fill="none" stroke="#58a6ff" stroke-width="1.5" stroke-dasharray="3,2"/>
<text x="46" y="404" fill="#8b949e" font-size="9">interactive (direct)</text>
<!-- Queue flow label -->
<rect x="160" y="395" width="10" height="10" rx="2" fill="none" stroke="#f0883e" stroke-width="1.5"/>
<text x="176" y="404" fill="#8b949e" font-size="9">queue-driven</text>
<!-- Proxy/event label -->
<rect x="275" y="395" width="10" height="10" rx="2" fill="none" stroke="#3fb950" stroke-width="1.5" stroke-dasharray="3,2"/>
<text x="291" y="404" fill="#8b949e" font-size="9">proxy / events</text>
<!-- ==================== BOTTOM DETAILS ==================== -->
<line x1="30" y1="430" x2="1170" y2="430" stroke="#21262d" stroke-width="1"/>
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#f47067" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">target_node set &#x2192; route to specific node queue</text>
<circle cx="44" cy="494" r="3" fill="#f0883e" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">ws_id set &#x2192; route to owning node</text>
<circle cx="44" cy="514" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="517" fill="#484f58" font-size="9">neither &#x2192; shared queue, any node picks up</text></svg>

After

Width:  |  Height:  |  Size: 18 KiB

+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a1b0361c466327d0011a488847ea3c0365983713537d4a7c27cd7f5538ba33c
size 164829
oid sha256:d8ce6d2a43a991655c3f64a20b6e810fdb2f78eb767acc3d3d1b8d2c9f443181
size 165011
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7d75da92a657525bcbb7a425dc6c8d3cafe074c3ac3bff3cf4b1d44aea607b50
size 330156
oid sha256:0ee0a9391bd19d92e9271bf6bd531e9c2e18baf8c5a11ead49b3c10db4d8939b
size 329625
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:637458e0d78df82752746e519cd7300a830c8ce211f21625694ad0c162ca316d
size 481637
oid sha256:4512be48a51f7cd1136ea8e3344489a8d225c611cb1b961643b869351de76812
size 549668
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dc3b64c9e48153641af62ed43fbc1d89a31d1a8a61e7e71cfc550c805000310d
size 288290
oid sha256:24bdc6a83259e4db6aaa24f581bed59d351c7a83e1c282aac123c52b32d9f80d
size 288250
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b842683d238664a3e35d04358fecfc56cefd013f7dca5f13357b0376f881e1b3
size 245043
oid sha256:027ad99469d69f1d6b2e73ee802b50a75617cd286392375aa69133c13d3683dc
size 256347
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b22d5980fe5cc4b8466ba0797113dc8fa83fab8df24b5dacceaf97e62e2e25b0
size 187649
oid sha256:32a0665cceffcc0517265bde12cfb227688aa8585284b5e946ab23bcc52daee6
size 187650
+1 -1
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:90e4f74be795b530e711faa87bc6eb2b3bf6abb68d8fac8ebff7aaf30c6fbe53
oid sha256:09535722ba975e47cf0557a40b6c481f125ff2022c396f79715c3bba9f715871
size 222032
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d33b9b3affcdb07086b5aebca8a3b9c2b009cdfc6f360950a0e72e65fbcb8f17
size 201602
oid sha256:ed457b10b534b5fc2a5e190b281d7ded4dd1615da2229d67a373cf5dddccd059
size 201601
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:adac93a0bb062d7199b819a600a0983ff011a75d16928fb80322cbb41f9284ea
size 158866
oid sha256:e0a3f48cca1b8408862dc4ba04fd340703346f44d84048c99e9900f48e9c7e22
size 158867
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:69f201cff948cb0a19810b7c4ad26d346f869ee2dd3141eba4f353332efa2e21
size 373649
oid sha256:35cf3a6942f62dabcbbe012ac2f9e6f155332c894692981b076de5a25c1f3330
size 374055
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:97e7210cd8f1ad195f4d5e25e778d82df3c08c5c6e0f09722e84a7a453714867
size 411664
oid sha256:a74b4b8b5dbfb1a51a01100b731477968942b01218bad9451a3d5a9cb3003294
size 411665
+1 -1
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4c3214ef416c1dfe4fa17834c2b6f4071a8093cfdb2b862848ca79938f726a13
oid sha256:84524f4bc900708ac8adf081591d336f862830188eb8505e71a0f071b339d923
size 252599
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9823a41e09611c5c0530d9fc12ad4139cfcc3ae238dc665b2888ec94d7d6781
size 195708
oid sha256:435a58aa09d0e6615e78c0be62e5fd9aa6d7329b1e96619744355c42ade649c9
size 196502
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:733aa17cbfdab60a601cac6adf439c657dd3535e3d6c33c69c2ef93ba8ec5989
size 251042
oid sha256:5faa5335152685cf1c8bf77ed93847d751cde59e1afed651e5991113f2f0f31b
size 242670
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f4b2a2010335f986511c8dabaf49ec046ac02e577f9bc9924897e045f860bb13
size 248808
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:47b106bbcb1041fe4122007065c6cc85605348b42fc7140287194cf25e42e095
size 318036
oid sha256:1380065cbb5f95b5ea7dc6b2a00986c455b82888af60784980dffbd936460dcf
size 431129
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f0f6097840fccdbfe16cd5e4c9f5d063b2c36942460a944df68b8ec947e63ea3
size 221452
+11 -3
View File
@@ -28,6 +28,8 @@ Console dashboard: http://localhost:8090
| `bridge` | — | default | Redis-to-HTTP bridge (multi-node routing) |
| `console` | 8090 | default | Cluster dashboard |
| `channel` | — | production | Channel gateway (Discord, Slack, etc.) |
| `server-1``server-10` | — | cluster | 10-node server fleet (PostgreSQL required) |
| `bridge-1``bridge-10` | — | cluster | Matching bridge fleet |
| `sim` | — | sim | Multi-node cluster simulator |
## Profiles
@@ -44,6 +46,12 @@ docker compose up
docker compose --profile production up
```
**Cluster** — 10-node server/bridge fleet sharing PostgreSQL and Redis. Access all nodes via the console at `:8090`. Requires `POSTGRES_PASSWORD`:
```bash
docker compose --profile cluster up
```
**Sim** — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:
```bash
@@ -135,13 +143,13 @@ The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TO
## Scaling
Scale to multiple server/bridge pairs:
For multi-node testing, use the `cluster` profile which provides 10 dedicated server+bridge pairs with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
docker compose up --scale server=3 --scale bridge=3
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
Each bridge auto-generates a unique node ID from its container hostname. When scaling `server`, remove the host port mapping (or use a reverse proxy) to avoid port conflicts.
The default `server` and `bridge` also run alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
## Volumes
+7 -1
View File
@@ -78,7 +78,7 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
| | `stream_global_events()` | `Iterator[ServerEvent]` |
| **High-level** | `send_and_wait(message, ws_id, *, timeout, on_event)` | `TurnResult` |
| **Sessions** | `list_sessions()` | `ListSessionsResponse` |
| **Saved** | `list_saved_workstreams()` | `ListSavedWorkstreamsResponse` |
| **Auth** | `login(username=..., password=...)` | `AuthLoginResponse` |
| | `login(token="ts_xxx")` | `AuthLoginResponse` |
| | `logout()` | `StatusResponse` |
@@ -96,6 +96,12 @@ Both `TurnstoneConsole` (sync) and `AsyncTurnstoneConsole` (async) expose:
| | `workstreams(*, state, node, search, sort, page, per_page)` | `ClusterWorkstreamsResponse` |
| | `node_detail(node_id)` | `NodeDetailResponse` |
| | `create_workstream(*, node_id, name, model, initial_message)` | `ConsoleCreateWsResponse` |
| **Schedules** | `list_schedules()` | `ListSchedulesResponse` |
| | `create_schedule(*, name, schedule_type, initial_message, ...)` | `ScheduleInfo` |
| | `get_schedule(task_id)` | `ScheduleInfo` |
| | `update_schedule(task_id, *, name=..., enabled=..., ...)` | `ScheduleInfo` |
| | `delete_schedule(task_id)` | `StatusResponse` |
| | `list_schedule_runs(task_id, *, limit=50)` | `ListScheduleRunsResponse` |
| **Streaming** | `stream_cluster_events()` | `Iterator[ClusterEvent]` |
| **Auth** | `login(username=..., password=...)` / `login(token="ts_xxx")` | `AuthLoginResponse` |
| | `logout()` | `StatusResponse` |
+16 -9
View File
@@ -73,7 +73,7 @@ Scopes are hierarchical — higher scopes imply all lower ones.
| Scope | Grants | Implies |
|-------|--------|---------|
| `read` | View workstreams, sessions, history | — |
| `read` | View workstreams, saved workstreams, history | — |
| `write` | Send messages, create/close workstreams | `read` |
| `approve` | Approve tool calls, admin endpoints | `read`, `write` |
@@ -354,15 +354,22 @@ provided, that static token is used instead.
The bridge and console collector use `ServiceTokenManager` for
auto-rotating JWTs when communicating with server nodes:
| Service | Identity | Scope | Purpose |
|---------|----------|-------|---------|
| Bridge | `bridge` | `approve` | Tool approval proxy, message relay |
| Console collector | `console-collector` | `read` | Node health polling |
| Console proxy | `console-proxy` | `write` | Proxied API calls |
| Service | Identity | Scope | Audience | Purpose |
|---------|----------|-------|----------|---------|
| Bridge | `bridge` | `approve` | `turnstone-server` | Tool approval proxy, message relay |
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy | `console-proxy` | `write` | `turnstone-server` | Proxied API calls |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
All service tokens use `aud: turnstone-server` and 1-hour expiry with
automatic refresh. The bridge injects auth headers per-request via httpx
event hooks to ensure rotated tokens are picked up on SSE reconnects.
Service tokens use 1-hour expiry with automatic refresh via
`ServiceTokenManager`. The bridge injects auth headers per-request via
httpx event hooks to ensure rotated tokens are picked up on SSE
reconnects.
Note that the channel gateway uses a distinct JWT audience
(`turnstone-channel`) from the server (`turnstone-server`) and console
(`turnstone-console`). A server-scoped JWT cannot authenticate to the
channel gateway endpoint, and vice versa.
---
+156 -5
View File
@@ -1,6 +1,6 @@
# Tools Reference
turnstone exposes 14 built-in tools plus any number of external MCP tools to the
turnstone exposes 15 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -46,11 +46,12 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 14 tool definitions (sent to the model). |
| `TOOLS` | All 15 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 15 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -68,6 +69,9 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 15
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
@@ -113,6 +117,7 @@ Each item's `execute` callable is invoked:
- `remember` -- writes to persistent memory database (lightweight, always auto-approved)
- `recall` -- reads from persistent memory database
- `forget` -- deletes from persistent memory database (lightweight, always auto-approved)
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
**Requires user confirmation** (write operations, network access, side effects):
- `bash` -- arbitrary command execution
@@ -162,6 +167,7 @@ Every tool defines a `primary_key`. The mapping is:
| `remember` | `key` |
| `recall` | `query` |
| `forget` | `key` |
| `notify` | `message` |
---
@@ -335,7 +341,7 @@ Plan before implementing -- an autonomous agent explores the codebase and writes
|-----------|--------|----------|-------------|
| `prompt` | string | yes | What to plan -- the goal, constraints, and scope. |
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<session_id>.md` (unique per session, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<ws_id>.md` (unique per workstream, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **Auto-approve**: No -- requires user confirmation, plus post-execution review gate.
- **Agent availability**: Not available to sub-agents (top-level only).
@@ -387,6 +393,33 @@ Remove a persistent memory by key.
---
## Notifications
### notify
Send a notification to a user or channel on an external platform.
| Parameter | Type | Required | Description |
|----------------|--------|----------|-------------|
| `message` | string | yes | Notification content (plain text, max 2000 chars). |
| `username` | string | no | Turnstone username — sends to all linked channels. |
| `channel_type` | string | no | Platform for direct targeting (`discord`). |
| `channel_id` | string | no | Platform-specific channel or user ID for direct targeting. |
| `title` | string | no | Optional short title (rendered as bold prefix). |
Provide either `username` for user-based targeting or `channel_type` +
`channel_id` for direct targeting. Do not combine both.
- **What it does**: Sends a notification via the channel gateway's HTTP endpoint (`POST /v1/api/notify`). The server queries the `services` table for healthy channel gateways, authenticates with a service JWT (`aud: turnstone-channel`), and delivers to the first healthy gateway. On failure, retries up to 2 additional times with backoff (1s, 3s). Rate-limited to 5 notifications per turn (counter only increments on success).
- **Auto-approve**: Yes — notifications are time-sensitive and auto-approved so the model can alert users urgently.
- **Agent availability**: `agent` and `task_agent`.
> See [Channel Integrations: Notifications](channels.md#notifications)
> for the full delivery flow, service registry details, and security
> measures.
---
## Summary Table
| Tool | Category | Auto-approve | agent | task_agent | primary_key |
@@ -405,6 +438,78 @@ Remove a persistent memory by key.
| `remember` | Memory | Yes | No | No | `key` |
| `recall` | Memory | Yes | No | No | `query` |
| `forget` | Memory | Yes | No | No | `key` |
| `notify` | Notify | Yes | Yes | Yes | `message` |
| `tool_search`| Search | Yes | No | No | `query` |
---
## Dynamic Tool Search
When many MCP tools are connected, the total tool count can grow large enough to
consume significant context window tokens and reduce model accuracy. Dynamic tool
search addresses this by deferring tools the model is unlikely to need on the
current turn and letting it search for them on demand.
### Three-tier approach
Tool search uses the best available mechanism for each provider:
1. **Anthropic (native)** -- Models that support it receive `defer_loading: true`
on deferred tool definitions plus the `tool_search_tool_bm25_20251119` server-side
search tool. Anthropic's API handles search and expansion transparently.
2. **OpenAI GPT-5.4+ (native)** -- Models with hosted tool search receive
`defer_loading: true` on deferred definitions. The API handles search internally.
3. **vLLM / llama.cpp / NIM (client-side BM25)** -- A synthetic `tool_search`
function tool is injected into the tool list. When the model calls it,
`_exec_tool_search()` runs a pure-Python BM25 index over tool names and
descriptions, then expands the matched tools into the visible set.
### Configuration
Tool search is configured in `config.toml` under the `[tools]` section:
```toml
[tools]
search = "auto" # "auto", "on", or "off"
search_threshold = 20 # minimum total tool count to activate
search_max_results = 5 # max tools returned per search call
```
CLI flags override the config file:
- `--tool-search {auto,on,off}` -- force tool search on or off, or let turnstone
decide based on threshold (default: `auto`).
- `--tool-search-threshold N` -- minimum tool count to activate (default: 20).
- `--tool-search-max-results N` -- max results per search (default: 5).
### How it works
1. **Threshold check**: At session startup, `ToolSearchManager.should_activate()`
counts total tools (built-in + MCP). If the count is below the threshold, tool
search stays off and all tools are sent to the model directly.
2. **Partitioning**: When active, tools are split into two sets:
- **Always-on** -- the 15 built-in tools (members of `BUILTIN_TOOL_NAMES`).
These are always visible to the model.
- **Deferred** -- all MCP tools. These are not sent in the tool list unless
the model searches for them.
3. **Search and expand**: When the model calls `tool_search` (client-side) or the
provider's native search returns results, the matched tools are added to the
visible set via `expand_visible()`. Once expanded, a tool stays visible for
the remainder of the session.
4. **Multi-turn persistence**: Expanded tools are never removed. This avoids
confusing the model when it references a tool it discovered in an earlier turn.
### Agent exemption
Plan and task sub-agents do not use tool search. They operate on scoped tool
sets (`AGENT_TOOLS` for plan agents, `TASK_AGENT_TOOLS` for task agents) with
MCP tools merged in. Tool search is only active for the top-level session,
where the model can interactively search for tools it needs.
---
@@ -421,13 +526,17 @@ MCP-compatible service.
2. **Discovery**: At startup, `MCPClientManager` connects to each configured server
(via stdio subprocess or HTTP), performs the MCP `initialize` handshake, and calls
`tools/list` to discover available tools.
`tools/list` to discover available tools. During the handshake, the manager checks
each server's capabilities for `tools.listChanged` support (push notifications).
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 14 built-in tools via
4. **Merging**: MCP tools are appended after the 15 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
[Dynamic Tool Search](#dynamic-tool-search) above).
5. **Dispatch**: When the LLM calls an MCP tool, `_prepare_mcp_tool()` builds a
generic approval preview and `_exec_mcp_tool()` calls `MCPClientManager.call_tool_sync()`,
@@ -500,3 +609,45 @@ MCP tools (3):
mcp__github__create_issue [MCP: github] Create a GitHub issue
mcp__postgres__query [MCP: postgres] Run a SQL query
```
### Dynamic tool refresh
MCP tool lists stay up-to-date without restart through three mechanisms:
1. **Push notifications** -- MCP servers that declare `tools.listChanged: true` in
their capabilities send `notifications/tools/list_changed` when their tool list
changes. `MCPClientManager` registers a `message_handler` on each `ClientSession`
that triggers an immediate refresh for that server.
2. **Periodic timer** -- Servers that do *not* support push notifications are polled
on a configurable interval (default 4 hours). The timer is staggered using a
launch-time seed (`monotonic_ns ^ pid`) so cluster nodes don't all hit MCP
servers simultaneously. Configure via `[mcp] refresh_interval` in `config.toml`
or `--mcp-refresh-interval SECONDS` on the CLI. Set to `0` to disable.
3. **Manual** -- `/mcp refresh` re-fetches tools from all servers immediately.
`/mcp refresh <server>` targets a single server. If a server has disconnected,
manual refresh attempts reconnection.
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`, `_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```toml
[mcp]
refresh_interval = 14400 # seconds (default 4h), 0 to disable
```
```
/mcp refresh
MCP refresh complete:
github: +1 added
+ mcp__github__create_pr
postgres: no changes
/mcp refresh github
MCP refresh complete:
github: no changes
```
+7 -3
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "0.4.0"
version = "0.5.0"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -43,10 +43,10 @@ Repository = "https://github.com/turnstonelabs/turnstone"
Issues = "https://github.com/turnstonelabs/turnstone/issues"
[project.optional-dependencies]
test = ["pytest>=9.0", "pytest-cov>=6.0"]
test = ["pytest>=9.0", "pytest-cov>=6.0", "croniter>=3.0"]
dev = ["ruff>=0.9", "mypy>=1.14", "types-redis>=4.6"]
mq = ["redis>=7.2"]
console = ["redis>=7.2"]
console = ["redis>=7.2", "croniter>=3.0"]
sim = ["redis>=7.2"]
anthropic = ["anthropic>=0.39"]
postgres = ["psycopg[binary]>=3.2"]
@@ -150,6 +150,10 @@ ignore_missing_imports = true
module = ["discord", "discord.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["croniter", "croniter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["turnstone.channels.discord.*"]
disallow_subclassing_any = false
+245 -47
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "0.3.0",
"version": "0.4.2",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -365,12 +365,12 @@
}
}
},
"/v1/api/sessions": {
"/v1/api/workstreams/saved": {
"get": {
"summary": "List saved sessions",
"operationId": "v1_api_sessions_get",
"summary": "List saved workstreams",
"operationId": "v1_api_workstreams_saved_get",
"tags": [
"Sessions"
"Workstreams"
],
"responses": {
"200": {
@@ -378,7 +378,7 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ListSessionsResponse"
"$ref": "#/components/schemas/ListSavedWorkstreamsResponse"
}
}
}
@@ -427,6 +427,88 @@
}
}
},
"/v1/api/auth/setup": {
"post": {
"summary": "Create first admin user",
"operationId": "v1_api_auth_setup_post",
"tags": [
"Auth"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthSetupRequest"
}
}
}
},
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthSetupResponse"
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "Error 409",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Error 503",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/auth/status": {
"get": {
"summary": "Return auth state",
"operationId": "v1_api_auth_status_get",
"tags": [
"Auth"
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/AuthStatusResponse"
}
}
}
}
}
}
},
"/v1/api/auth/logout": {
"post": {
"summary": "Clear auth cookie",
@@ -503,17 +585,27 @@
"type": "object"
},
"AuthLoginRequest": {
"description": "POST /v1/api/auth/login request body.",
"description": "POST /v1/api/auth/login request body.\n\nEither username+password or token must be provided.",
"properties": {
"username": {
"default": "",
"description": "Login username",
"title": "Username",
"type": "string"
},
"password": {
"default": "",
"description": "Login password",
"title": "Password",
"type": "string"
},
"token": {
"description": "Bearer token to authenticate",
"default": "",
"description": "Legacy: bearer token to authenticate",
"title": "Token",
"type": "string"
}
},
"required": [
"token"
],
"title": "AuthLoginRequest",
"type": "object"
},
@@ -525,14 +617,35 @@
"title": "Status",
"type": "string"
},
"user_id": {
"default": "",
"description": "Authenticated user ID",
"title": "User Id",
"type": "string"
},
"role": {
"description": "Assigned role",
"description": "Legacy role",
"examples": [
"full",
"read"
],
"title": "Role",
"type": "string"
},
"scopes": {
"default": "",
"description": "Comma-separated scopes",
"examples": [
"read,write,approve"
],
"title": "Scopes",
"type": "string"
},
"jwt": {
"default": "",
"description": "JWT session token (if JWT auth is configured)",
"title": "Jwt",
"type": "string"
}
},
"required": [
@@ -541,6 +654,97 @@
"title": "AuthLoginResponse",
"type": "object"
},
"AuthSetupRequest": {
"description": "POST /v1/api/auth/setup request body.",
"properties": {
"username": {
"description": "Login username (1-64 ASCII characters)",
"title": "Username",
"type": "string"
},
"display_name": {
"description": "Display name",
"title": "Display Name",
"type": "string"
},
"password": {
"description": "Password (minimum 8 characters)",
"title": "Password",
"type": "string"
}
},
"required": [
"username",
"display_name",
"password"
],
"title": "AuthSetupRequest",
"type": "object"
},
"AuthSetupResponse": {
"description": "POST /v1/api/auth/setup success response.",
"properties": {
"status": {
"default": "ok",
"title": "Status",
"type": "string"
},
"user_id": {
"title": "User Id",
"type": "string"
},
"username": {
"title": "Username",
"type": "string"
},
"role": {
"default": "full",
"title": "Role",
"type": "string"
},
"scopes": {
"default": "approve,read,write",
"title": "Scopes",
"type": "string"
},
"jwt": {
"default": "",
"description": "JWT session token",
"title": "Jwt",
"type": "string"
}
},
"required": [
"user_id",
"username"
],
"title": "AuthSetupResponse",
"type": "object"
},
"AuthStatusResponse": {
"description": "GET /v1/api/auth/status response.",
"properties": {
"auth_enabled": {
"title": "Auth Enabled",
"type": "boolean"
},
"has_users": {
"title": "Has Users",
"type": "boolean"
},
"setup_required": {
"title": "Setup Required",
"type": "boolean"
}
},
"required": [
"auth_enabled",
"has_users",
"setup_required"
],
"title": "AuthStatusResponse",
"type": "object"
},
"SendRequest": {
"properties": {
"message": {
@@ -677,6 +881,12 @@
"description": "Auto-approve all tool calls",
"title": "Auto Approve",
"type": "boolean"
},
"resume_ws": {
"default": "",
"description": "Workstream ID to resume atomically during creation (empty = fresh start)",
"title": "Resume Ws",
"type": "string"
}
},
"title": "CreateWorkstreamRequest",
@@ -693,6 +903,18 @@
"description": "Assigned workstream name",
"title": "Name",
"type": "string"
},
"resumed": {
"default": false,
"description": "Whether a previous workstream was resumed",
"title": "Resumed",
"type": "boolean"
},
"message_count": {
"default": 0,
"description": "Number of messages in the resumed workstream",
"title": "Message Count",
"type": "integer"
}
},
"required": [
@@ -745,18 +967,6 @@
"state": {
"title": "State",
"type": "string"
},
"session_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Session Id"
}
},
"required": [
@@ -837,18 +1047,6 @@
"title": "State",
"type": "string"
},
"session_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Session Id"
},
"title": {
"default": "",
"title": "Title",
@@ -903,26 +1101,26 @@
"title": "DashboardWorkstream",
"type": "object"
},
"ListSessionsResponse": {
"ListSavedWorkstreamsResponse": {
"properties": {
"sessions": {
"workstreams": {
"items": {
"$ref": "#/components/schemas/SessionInfo"
"$ref": "#/components/schemas/SavedWorkstreamInfo"
},
"title": "Sessions",
"title": "Workstreams",
"type": "array"
}
},
"required": [
"sessions"
"workstreams"
],
"title": "ListSessionsResponse",
"title": "ListSavedWorkstreamsResponse",
"type": "object"
},
"SessionInfo": {
"SavedWorkstreamInfo": {
"properties": {
"session_id": {
"title": "Session Id",
"ws_id": {
"title": "Ws Id",
"type": "string"
},
"alias": {
@@ -963,12 +1161,12 @@
}
},
"required": [
"session_id",
"ws_id",
"created",
"updated",
"message_count"
],
"title": "SessionInfo",
"title": "SavedWorkstreamInfo",
"type": "object"
},
"HealthResponse": {
+41
View File
@@ -10,9 +10,14 @@ import type {
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateScheduleRequest,
ListScheduleRunsResponse,
ListSchedulesResponse,
NodeDetailResponse,
NodesOptions,
ScheduleInfo,
StatusResponse,
UpdateScheduleRequest,
WorkstreamsOptions,
} from "./types.js";
@@ -111,4 +116,40 @@ export class TurnstoneConsole extends BaseClient {
async health(): Promise<ConsoleHealthResponse> {
return this.request("GET", "/health");
}
// -- Schedules ------------------------------------------------------------
async listSchedules(): Promise<ListSchedulesResponse> {
return this.request("GET", "/v1/api/admin/schedules");
}
async createSchedule(opts: CreateScheduleRequest): Promise<ScheduleInfo> {
return this.request("POST", "/v1/api/admin/schedules", { json: opts });
}
async getSchedule(taskId: string): Promise<ScheduleInfo> {
return this.request("GET", `/v1/api/admin/schedules/${taskId}`);
}
async updateSchedule(
taskId: string,
opts: UpdateScheduleRequest,
): Promise<ScheduleInfo> {
return this.request("PUT", `/v1/api/admin/schedules/${taskId}`, {
json: opts,
});
}
async deleteSchedule(taskId: string): Promise<StatusResponse> {
return this.request("DELETE", `/v1/api/admin/schedules/${taskId}`);
}
async listScheduleRuns(
taskId: string,
opts?: { limit?: number },
): Promise<ListScheduleRunsResponse> {
return this.request("GET", `/v1/api/admin/schedules/${taskId}/runs`, {
params: { limit: opts?.limit ?? 50 },
});
}
}
+8 -2
View File
@@ -83,8 +83,8 @@ export type {
DashboardWorkstream,
DashboardAggregate,
DashboardResponse,
SessionInfo,
ListSessionsResponse,
SavedWorkstreamInfo,
ListSavedWorkstreamsResponse,
BackendStatus,
WorkstreamCounts,
HealthResponse,
@@ -103,6 +103,12 @@ export type {
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateScheduleRequest,
UpdateScheduleRequest,
ScheduleInfo,
ScheduleRunInfo,
ListSchedulesResponse,
ListScheduleRunsResponse,
TurnResult,
SendAndWaitOptions,
NodesOptions,
+4 -4
View File
@@ -8,7 +8,7 @@ import type {
CreateWorkstreamResponse,
DashboardResponse,
HealthResponse,
ListSessionsResponse,
ListSavedWorkstreamsResponse,
ListWorkstreamsResponse,
SendAndWaitOptions,
SendResponse,
@@ -178,10 +178,10 @@ export class TurnstoneServer extends BaseClient {
return result;
}
// -- Sessions -------------------------------------------------------------
// -- Saved workstreams ----------------------------------------------------
async listSessions(): Promise<ListSessionsResponse> {
return this.request("GET", "/v1/api/sessions");
async listSavedWorkstreams(): Promise<ListSavedWorkstreamsResponse> {
return this.request("GET", "/v1/api/workstreams/saved");
}
// -- Auth -----------------------------------------------------------------
+77 -9
View File
@@ -71,14 +71,13 @@ export interface CreateWorkstreamRequest {
name?: string;
model?: string;
auto_approve?: boolean;
resume_session?: string;
resume_ws?: string;
}
export interface CreateWorkstreamResponse {
ws_id: string;
name: string;
resumed?: boolean;
session_id?: string;
message_count?: number;
}
@@ -90,7 +89,6 @@ export interface WorkstreamInfo {
id: string;
name: string;
state: string;
session_id?: string | null;
}
export interface ListWorkstreamsResponse {
@@ -101,7 +99,6 @@ export interface DashboardWorkstream {
id: string;
name: string;
state: string;
session_id?: string | null;
title?: string;
tokens?: number;
context_ratio?: number;
@@ -128,11 +125,11 @@ export interface DashboardResponse {
}
// ---------------------------------------------------------------------------
// Server API — Sessions
// Server API — Saved workstreams
// ---------------------------------------------------------------------------
export interface SessionInfo {
session_id: string;
export interface SavedWorkstreamInfo {
ws_id: string;
alias?: string | null;
title?: string | null;
created: string;
@@ -140,8 +137,8 @@ export interface SessionInfo {
message_count: number;
}
export interface ListSessionsResponse {
sessions: SessionInfo[];
export interface ListSavedWorkstreamsResponse {
workstreams: SavedWorkstreamInfo[];
}
// ---------------------------------------------------------------------------
@@ -269,6 +266,77 @@ export interface ConsoleHealthResponse {
versions: string[];
}
// ---------------------------------------------------------------------------
// Console API — Schedules
// ---------------------------------------------------------------------------
export interface CreateScheduleRequest {
name: string;
schedule_type: string;
initial_message: string;
description?: string;
cron_expr?: string;
at_time?: string;
target_mode?: string;
model?: string;
auto_approve?: boolean;
auto_approve_tools?: string[];
enabled?: boolean;
}
export interface UpdateScheduleRequest {
name?: string;
description?: string;
schedule_type?: string;
cron_expr?: string;
at_time?: string;
target_mode?: string;
model?: string;
initial_message?: string;
auto_approve?: boolean;
auto_approve_tools?: string[];
enabled?: boolean;
}
export interface ScheduleInfo {
task_id: string;
name: string;
description: string;
schedule_type: string;
cron_expr: string;
at_time: string;
target_mode: string;
model: string;
initial_message: string;
auto_approve: boolean;
auto_approve_tools: string[];
enabled: boolean;
created_by: string;
last_run: string | null;
next_run: string | null;
created: string;
updated: string;
}
export interface ListSchedulesResponse {
schedules: ScheduleInfo[];
}
export interface ScheduleRunInfo {
run_id: string;
task_id: string;
node_id: string;
ws_id: string;
correlation_id: string;
started: string;
status: string;
error: string;
}
export interface ListScheduleRunsResponse {
runs: ScheduleRunInfo[];
}
// ---------------------------------------------------------------------------
// SDK-specific types
// ---------------------------------------------------------------------------
+2 -2
View File
@@ -719,7 +719,7 @@ class TestServerAuth:
srv_mod._metrics.model = "test-model"
mock_session = MagicMock()
mock_session.session_id = "test-session-id"
mock_session.ws_id = "test-session-id"
mock_ws = MagicMock()
mock_ws.id = "test-ws"
@@ -937,7 +937,7 @@ class TestServerLogin:
srv_mod._metrics.model = "test-model"
mock_session = MagicMock()
mock_session.session_id = "test-session-id"
mock_session.ws_id = "test-session-id"
mock_ws = MagicMock()
mock_ws.id = "test-ws"
+175
View File
@@ -1,5 +1,6 @@
"""Tests for turnstone.console — collector and HTTP server."""
import asyncio
import json
import queue
from unittest.mock import MagicMock
@@ -1299,3 +1300,177 @@ class TestProxySharedStatic:
resp = client.get("/node/unknown/shared/base.css")
assert resp.status_code == 404
client.close()
# ---------------------------------------------------------------------------
# SSE proxy — raw byte passthrough
# ---------------------------------------------------------------------------
class TestSSEProxy:
"""Verify _proxy_sse forwards raw bytes including ping comments."""
def test_proxy_sse_preserves_pings_and_events(self):
"""SSE proxy should forward ping comments and events verbatim."""
from turnstone.console.server import _proxy_sse
# Simulate an upstream SSE response with a ping comment and a real event
sse_payload = b': ping - 2026-03-08T12:00:00Z\n\nevent: message\ndata: {"type": "test"}\n\n'
class FakeResponse:
status_code = 200
headers = {"content-type": "text/event-stream"}
async def aiter_bytes(self):
yield sse_payload
async def aclose(self):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeClient:
def stream(self, method, url, **kwargs):
return FakeResponse()
class FakeRequest:
class url: # noqa: N801
query = "ws_id=test123"
class app: # noqa: N801
class state: # noqa: N801
proxy_sse_client = FakeClient()
proxy_auth_token = ""
headers = {}
async def is_disconnected(self):
return False
async def _run():
response = await _proxy_sse(
FakeRequest(), "http://fake:8080", "events", api_prefix="v1/api"
)
assert response.media_type == "text/event-stream"
# Collect the streamed bytes
chunks: list[bytes] = []
async for chunk in response.body_iterator:
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
body = b"".join(chunks)
# Ping comment must be preserved (not filtered)
assert b": ping" in body
# Real event must be preserved
assert b"event: message" in body
assert b'"type": "test"' in body
asyncio.run(_run())
def test_proxy_sse_upstream_error_status(self):
"""Non-200 upstream status should yield an error event."""
from turnstone.console.server import _proxy_sse
class FakeResponse:
status_code = 502
async def aiter_bytes(self):
return
yield # make it an async generator
async def aclose(self):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeClient:
def stream(self, method, url, **kwargs):
return FakeResponse()
class FakeRequest:
class url: # noqa: N801
query = ""
class app: # noqa: N801
class state: # noqa: N801
proxy_sse_client = FakeClient()
proxy_auth_token = ""
headers = {}
async def is_disconnected(self):
return False
async def _run():
response = await _proxy_sse(FakeRequest(), "http://fake:8080", "events")
chunks: list[bytes] = []
async for chunk in response.body_iterator:
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
body = b"".join(chunks)
assert b"event: error" in body
assert b"502" in body
asyncio.run(_run())
def test_proxy_sse_disconnect_handling(self):
"""Proxy should stop when browser disconnects."""
from turnstone.console.server import _proxy_sse
class FakeResponse:
status_code = 200
async def aiter_bytes(self):
yield b"data: chunk1\n\n"
yield b"data: chunk2\n\n" # should not be reached
yield b"data: chunk3\n\n"
async def aclose(self):
pass
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
class FakeClient:
def stream(self, method, url, **kwargs):
return FakeResponse()
call_count = 0
class FakeRequest:
class url: # noqa: N801
query = ""
class app: # noqa: N801
class state: # noqa: N801
proxy_sse_client = FakeClient()
proxy_auth_token = ""
headers = {}
async def is_disconnected(self):
nonlocal call_count
call_count += 1
return call_count > 1 # disconnect after first chunk
async def _run():
response = await _proxy_sse(FakeRequest(), "http://fake:8080", "events")
chunks: list[bytes] = []
async for chunk in response.body_iterator:
chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
body = b"".join(chunks)
assert b"chunk1" in body
# Should have stopped before chunk3
assert b"chunk3" not in body
asyncio.run(_run())
+299 -1
View File
@@ -2,10 +2,11 @@
from __future__ import annotations
import asyncio
import json
from contextlib import AsyncExitStack
from typing import Any
from unittest.mock import MagicMock, patch
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
@@ -411,3 +412,300 @@ class TestCreateMcpClient:
result = create_mcp_client()
assert result is None
# ---------------------------------------------------------------------------
# Tool refresh — _rebuild_tools, _refresh_server, listeners
# ---------------------------------------------------------------------------
class TestRebuildTools:
def test_rebuild_from_per_server(self):
mgr = MCPClientManager({})
mgr._per_server_tools = {
"github": [_fake_openai_tool("mcp__github__search")],
"slack": [_fake_openai_tool("mcp__slack__send")],
}
mgr._rebuild_tools()
assert len(mgr._tools) == 2
names = {t["function"]["name"] for t in mgr._tools}
assert names == {"mcp__github__search", "mcp__slack__send"}
assert mgr._tool_map["mcp__github__search"] == ("github", "search")
assert mgr._tool_map["mcp__slack__send"] == ("slack", "send")
def test_rebuild_copy_on_write(self):
mgr = MCPClientManager({})
mgr._per_server_tools = {"a": [_fake_openai_tool("mcp__a__x")]}
mgr._rebuild_tools()
old_tools = mgr._tools
old_map = mgr._tool_map
mgr._per_server_tools["b"] = [_fake_openai_tool("mcp__b__y")]
mgr._rebuild_tools()
assert mgr._tools is not old_tools
assert mgr._tool_map is not old_map
def test_rebuild_empty(self):
mgr = MCPClientManager({})
mgr._per_server_tools = {}
mgr._rebuild_tools()
assert mgr._tools == []
assert mgr._tool_map == {}
class TestRefreshServer:
def test_refresh_detects_added_tools(self):
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mock_result = MagicMock()
mock_result.tools = [
_fake_mcp_tool("search"),
_fake_mcp_tool("create"), # new tool
]
mock_session.list_tools = AsyncMock(return_value=mock_result)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
added, removed = await mgr._refresh_server("github")
assert "mcp__github__create" in added
assert removed == []
assert len(mgr._tools) == 2
asyncio.run(_run())
def test_refresh_detects_removed_tools(self):
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mock_result = MagicMock()
mock_result.tools = [] # all tools removed
mock_session.list_tools = AsyncMock(return_value=mock_result)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
added, removed = await mgr._refresh_server("github")
assert added == []
assert "mcp__github__search" in removed
assert mgr._tools == []
asyncio.run(_run())
def test_refresh_no_changes(self):
async def _run() -> None:
mgr = MCPClientManager({})
mock_session = MagicMock()
mock_result = MagicMock()
mock_result.tools = [_fake_mcp_tool("search")]
mock_session.list_tools = AsyncMock(return_value=mock_result)
mgr._sessions["github"] = mock_session
mgr._per_server_tools["github"] = [_fake_openai_tool("mcp__github__search")]
mgr._rebuild_tools()
added, removed = await mgr._refresh_server("github")
assert added == []
assert removed == []
asyncio.run(_run())
def test_refresh_disconnected_raises(self):
async def _run() -> None:
mgr = MCPClientManager({})
with pytest.raises(RuntimeError, match="not connected"):
await mgr._refresh_server("ghost")
asyncio.run(_run())
class TestListeners:
def test_add_and_notify(self):
mgr = MCPClientManager({})
calls: list[int] = []
mgr.add_listener(lambda: calls.append(1))
mgr._per_server_tools = {"a": [_fake_openai_tool("mcp__a__x")]}
mgr._rebuild_tools()
assert len(calls) == 1
def test_remove_listener(self):
mgr = MCPClientManager({})
calls: list[int] = []
cb = lambda: calls.append(1) # noqa: E731
mgr.add_listener(cb)
mgr.remove_listener(cb)
mgr._rebuild_tools()
assert calls == []
def test_remove_nonexistent_listener(self):
mgr = MCPClientManager({})
mgr.remove_listener(lambda: None) # should not raise
def test_listener_error_does_not_propagate(self):
mgr = MCPClientManager({})
mgr.add_listener(lambda: 1 / 0) # will raise ZeroDivisionError
mgr._rebuild_tools() # should not raise
class TestServerNames:
def test_server_names_property(self):
mgr = MCPClientManager({"github": {}, "slack": {}})
assert sorted(mgr.server_names) == ["github", "slack"]
def test_server_names_empty(self):
mgr = MCPClientManager({})
assert mgr.server_names == []
# ---------------------------------------------------------------------------
# Session integration — tool refresh propagation
# ---------------------------------------------------------------------------
class TestSessionRefresh:
@pytest.fixture()
def tmp_db(self, tmp_path):
from turnstone.core.storage import init_storage, reset_storage
reset_storage()
init_storage("sqlite", path=str(tmp_path / "test.db"), run_migrations=False)
yield
reset_storage()
def _make_session(self, mcp_client=None, **kwargs):
from turnstone.core.session import ChatSession
defaults: dict[str, Any] = dict(
client=MagicMock(),
model="test-model",
ui=MagicMock(),
instructions=None,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
mcp_client=mcp_client,
)
defaults.update(kwargs)
return ChatSession(**defaults)
def test_listener_registered_on_init(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = []
session = self._make_session(mcp_client=mock_mcp)
mock_mcp.add_listener.assert_called_once()
assert session._mcp_refresh_cb is not None
def test_no_listener_without_mcp(self, tmp_db):
session = self._make_session(mcp_client=None)
assert session._mcp_refresh_cb is None
def test_close_removes_listener(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = []
session = self._make_session(mcp_client=mock_mcp)
session.close()
mock_mcp.remove_listener.assert_called_once()
assert session._mcp_refresh_cb is None
def test_close_idempotent(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = []
session = self._make_session(mcp_client=mock_mcp)
session.close()
session.close() # should not raise
assert mock_mcp.remove_listener.call_count == 1
def test_on_mcp_tools_changed_rebuilds_tools(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool("mcp__test__a")]
session = self._make_session(mcp_client=mock_mcp)
initial_count = len(session._tools)
# Simulate a tool refresh — MCP now has 2 tools
mock_mcp.get_tools.return_value = [
_fake_openai_tool("mcp__test__a"),
_fake_openai_tool("mcp__test__b"),
]
session._on_mcp_tools_changed()
assert len(session._tools) == initial_count + 1
def test_tool_search_preserved_across_refresh(self, tmp_db):
# Create enough MCP tools to trigger tool search
mcp_tools = [_fake_openai_tool(f"mcp__srv__tool{i}") for i in range(25)]
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = mcp_tools
session = self._make_session(
mcp_client=mock_mcp,
tool_search="auto",
tool_search_threshold=20,
)
assert session._tool_search is not None
# Expand a tool
session._tool_search.expand_visible(["mcp__srv__tool0"])
assert "mcp__srv__tool0" in session._tool_search.get_expanded_names()
# Refresh with same tools
session._on_mcp_tools_changed()
assert session._tool_search is not None
assert "mcp__srv__tool0" in session._tool_search.get_expanded_names()
def test_tool_search_prunes_removed_from_expanded(self, tmp_db):
mcp_tools = [_fake_openai_tool(f"mcp__srv__tool{i}") for i in range(25)]
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = mcp_tools
session = self._make_session(
mcp_client=mock_mcp,
tool_search="auto",
tool_search_threshold=20,
)
session._tool_search.expand_visible(["mcp__srv__tool0"])
# Refresh with tool0 removed
new_tools = [_fake_openai_tool(f"mcp__srv__tool{i}") for i in range(1, 25)]
mock_mcp.get_tools.return_value = new_tools
session._on_mcp_tools_changed()
# tool0 was removed, so it should no longer be in expanded
expanded = session._tool_search.get_expanded_names()
assert "mcp__srv__tool0" not in expanded
def test_mcp_refresh_command(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["test"]
mock_mcp.refresh_sync.return_value = {"test": (["mcp__test__new"], [])}
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh")
mock_mcp.refresh_sync.assert_called_once_with(None)
session.ui.on_info.assert_called()
def test_mcp_refresh_specific_server(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["github", "slack"]
mock_mcp.refresh_sync.return_value = {"github": ([], [])}
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh github")
mock_mcp.refresh_sync.assert_called_once_with("github")
def test_mcp_refresh_unknown_server(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["github"]
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh nonexistent")
session.ui.on_error.assert_called_once()
assert "Unknown MCP server" in session.ui.on_error.call_args[0][0]
def test_mcp_refresh_error_handling(self, tmp_db):
mock_mcp = MagicMock()
mock_mcp.get_tools.return_value = [_fake_openai_tool()]
mock_mcp.server_names = ["test"]
mock_mcp.refresh_sync.side_effect = TimeoutError("timed out")
session = self._make_session(mcp_client=mock_mcp)
session.handle_command("/mcp refresh")
session.ui.on_error.assert_called_once()
assert "MCP refresh failed" in session.ui.on_error.call_args[0][0]
+2 -2
View File
@@ -534,7 +534,7 @@ class TestWorkstreamModelParam:
nonlocal captured_alias
captured_alias = model_alias
mock_session = MagicMock()
mock_session.session_id = "test123"
mock_session.ws_id = "test123"
return mock_session
mgr = WorkstreamManager(factory)
@@ -548,7 +548,7 @@ class TestWorkstreamModelParam:
nonlocal captured_alias
captured_alias = model_alias
mock_session = MagicMock()
mock_session.session_id = "test123"
mock_session.ws_id = "test123"
return mock_session
from turnstone.core.workstream import WorkstreamManager
+338
View File
@@ -0,0 +1,338 @@
"""Tests for the channel gateway HTTP notify endpoint."""
from __future__ import annotations
from unittest.mock import AsyncMock
import pytest
from starlette.testclient import TestClient
from turnstone.channels._http import create_channel_app
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "test.db"))
@pytest.fixture
def mock_adapter():
adapter = AsyncMock()
adapter.channel_type = "discord"
adapter.send = AsyncMock(return_value="msg_001")
return adapter
@pytest.fixture
def no_auth_client(storage, mock_adapter):
"""Client with no auth configured (for fail-closed tests)."""
app = create_channel_app({"discord": mock_adapter}, storage)
return TestClient(app)
@pytest.fixture
def client(storage, mock_adapter):
"""Default client with static auth token configured."""
app = create_channel_app({"discord": mock_adapter}, storage, auth_token="test-secret-token")
return TestClient(app)
@pytest.fixture
def authed_client(storage, mock_adapter):
"""Alias — same as client, for auth-specific test clarity."""
app = create_channel_app({"discord": mock_adapter}, storage, auth_token="test-secret-token")
return TestClient(app)
@pytest.fixture
def jwt_client(storage, mock_adapter):
"""Client with JWT auth configured."""
app = create_channel_app({"discord": mock_adapter}, storage, jwt_secret="a" * 32)
return TestClient(app)
class TestNotifyEndpoint:
def test_health(self, client):
resp = client.get("/health")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
def _headers(self) -> dict[str, str]:
return {"Authorization": "Bearer test-secret-token"}
def test_direct_discord_target(self, client, mock_adapter):
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123456"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert len(results) == 1
assert results[0]["status"] == "sent"
assert results[0]["message_id"] == "msg_001"
mock_adapter.send.assert_called_once_with("123456", "Hello!")
def test_with_title(self, client, mock_adapter):
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123456"},
"message": "Hello!",
"title": "Alert",
},
headers=self._headers(),
)
assert resp.status_code == 200
mock_adapter.send.assert_called_once_with("123456", "**Alert**\nHello!")
def test_username_resolution(self, client, storage, mock_adapter):
# Create a user and link a channel
storage.create_user("u1", "testuser", "Test User", "hash")
storage.create_channel_user("discord", "disc_123", "u1")
resp = client.post(
"/v1/api/notify",
json={
"target": {"username": "testuser"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert len(results) == 1
assert results[0]["status"] == "sent"
mock_adapter.send.assert_called_once_with("disc_123", "Hello!")
def test_unknown_username(self, client):
resp = client.post(
"/v1/api/notify",
json={
"target": {"username": "nobody"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 404
error = resp.json()["error"]
assert "nobody" not in error
assert "not found or has no linked channels" in error
def test_user_no_channels(self, authed_client, storage):
storage.create_user("u1", "testuser", "Test User", "hash")
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"username": "testuser"},
"message": "Hello!",
},
headers={"Authorization": "Bearer test-secret-token"},
)
assert resp.status_code == 404
# Generic message — must not differentiate "not found" vs "no channels"
error = resp.json()["error"]
assert "testuser" not in error
assert "not found or has no linked channels" in error
def test_missing_fields(self, client):
resp = client.post(
"/v1/api/notify",
json={"target": {"username": "x"}},
headers=self._headers(),
)
assert resp.status_code == 400
def test_missing_target(self, client):
resp = client.post(
"/v1/api/notify",
json={"message": "Hello!"},
headers=self._headers(),
)
assert resp.status_code == 400
def test_invalid_target(self, client):
resp = client.post(
"/v1/api/notify",
json={
"target": {"invalid": "field"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 400
def test_no_adapter(self, client, storage):
# App has discord adapter, try email target
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "email", "channel_id": "test@example.com"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert results[0]["status"] == "no_adapter"
def test_adapter_failure(self, client, mock_adapter):
mock_adapter.send.side_effect = RuntimeError("Discord API error")
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123456"},
"message": "Hello!",
},
headers=self._headers(),
)
assert resp.status_code == 200
results = resp.json()["results"]
assert results[0]["status"] == "failed"
def test_invalid_json(self, client):
resp = client.post(
"/v1/api/notify",
content=b"not json",
headers={
"content-type": "application/json",
"Authorization": "Bearer test-secret-token",
},
)
assert resp.status_code == 400
def test_whitespace_only_message(self, client):
"""Whitespace-only messages should be rejected."""
resp = client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": " ",
},
headers=self._headers(),
)
assert resp.status_code == 400
class TestNotifyAuth:
"""Tests for authentication on the /v1/api/notify endpoint."""
def test_reject_when_unconfigured(self, no_auth_client):
"""Requests are rejected (fail closed) when no auth is configured."""
resp = no_auth_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
)
assert resp.status_code == 401
def test_reject_without_token(self, authed_client):
"""Requests without Authorization header are rejected when auth is configured."""
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
)
assert resp.status_code == 401
def test_reject_wrong_token(self, authed_client):
"""Requests with wrong token are rejected."""
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": "Bearer wrong-token"},
)
assert resp.status_code == 401
def test_accept_valid_static_token(self, authed_client, mock_adapter):
"""Requests with correct static token are accepted."""
resp = authed_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": "Bearer test-secret-token"},
)
assert resp.status_code == 200
assert resp.json()["results"][0]["status"] == "sent"
def test_accept_valid_jwt(self, jwt_client, mock_adapter):
"""Requests with a valid JWT for the channel audience are accepted."""
from turnstone.core.auth import JWT_AUD_CHANNEL, create_jwt
token = create_jwt(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret="a" * 32,
audience=JWT_AUD_CHANNEL,
)
resp = jwt_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 200
def test_reject_jwt_wrong_audience(self, jwt_client):
"""JWTs with wrong audience are rejected."""
from turnstone.core.auth import create_jwt
token = create_jwt(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret="a" * 32,
audience="turnstone-server", # wrong audience
)
resp = jwt_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 401
def test_reject_jwt_wrong_secret(self, jwt_client):
"""JWTs signed with wrong secret are rejected."""
from turnstone.core.auth import JWT_AUD_CHANNEL, create_jwt
token = create_jwt(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret="b" * 32, # wrong secret
audience=JWT_AUD_CHANNEL,
)
resp = jwt_client.post(
"/v1/api/notify",
json={
"target": {"channel_type": "discord", "channel_id": "123"},
"message": "Hello!",
},
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 401
def test_health_bypasses_auth(self, authed_client):
"""Health endpoint is always accessible regardless of auth config."""
resp = authed_client.get("/health")
assert resp.status_code == 200
+618
View File
@@ -0,0 +1,618 @@
"""Tests for the notify tool (prepare + execute) in ChatSession."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
if TYPE_CHECKING:
from turnstone.core.session import ChatSession
def _make_session() -> ChatSession:
"""Create a minimal ChatSession with mocked dependencies."""
from unittest.mock import patch
with (
patch("turnstone.core.memory.register_workstream"),
patch("turnstone.core.session.save_message"),
):
from turnstone.core.session import ChatSession
ui = MagicMock()
session = ChatSession(
client=MagicMock(),
model="test-model",
ui=ui,
instructions=None,
temperature=0.7,
max_tokens=1000,
tool_timeout=30,
)
return session
class TestPrepareNotify:
def test_valid_username_target(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"username": "admin",
},
)
assert "execute" in result
assert result["func_name"] == "notify"
assert result["needs_approval"] is False
assert "@admin" in result["header"]
assert result["username"] == "admin"
assert result["message"] == "Hello!"
def test_valid_direct_target(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"channel_type": "discord",
"channel_id": "123456",
},
)
assert "execute" in result
assert result["channel_type"] == "discord"
assert result["channel_id"] == "123456"
assert "discord:123456" in result["header"]
def test_missing_message(self):
session = _make_session()
result = session._prepare_notify("call_1", {"username": "admin"})
assert "error" in result
assert "message" in result["error"].lower()
def test_empty_message(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "",
"username": "admin",
},
)
assert "error" in result
def test_message_too_long(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "x" * 2001,
"username": "admin",
},
)
assert "error" in result
assert "2000" in result["error"]
def test_both_username_and_direct(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"username": "admin",
"channel_type": "discord",
"channel_id": "123",
},
)
assert "error" in result
assert "both" in result["error"].lower() or "ambiguous" in result["error"].lower()
def test_no_target(self):
session = _make_session()
result = session._prepare_notify("call_1", {"message": "Hello!"})
assert "error" in result
def test_channel_type_without_id(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"channel_type": "discord",
},
)
assert "error" in result
assert "channel_id" in result["error"]
def test_channel_id_without_type(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"channel_id": "123456",
},
)
assert "error" in result
assert "channel_type" in result["error"]
def test_preview_truncated(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "a" * 200,
"username": "admin",
},
)
assert result["preview"].endswith("...")
assert len(result["preview"]) <= 123 # 120 chars + "..."
def test_title_passed_through(self):
session = _make_session()
result = session._prepare_notify(
"call_1",
{
"message": "Hello!",
"username": "admin",
"title": "Alert",
},
)
assert result["title"] == "Alert"
class TestExecNotify:
def test_sends_http_to_channel_gateway(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "Alert",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp) as mock_post,
patch.dict("os.environ", {}, clear=False),
):
call_id, msg = session._exec_notify(item)
assert call_id == "call_1"
assert "sent successfully" in msg.lower()
mock_post.assert_called_once()
post_kwargs = mock_post.call_args
assert post_kwargs.kwargs["json"]["target"] == {"username": "admin"}
assert post_kwargs.kwargs["json"]["message"] == "Hello!"
def test_no_services_available(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
# No services registered
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
assert "no channel gateway" in msg.lower()
def test_rate_limit(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp),
patch.dict("os.environ", {}, clear=False),
):
for _i in range(5):
call_id, msg = session._exec_notify(item)
assert "sent successfully" in msg.lower()
# 6th should fail
call_id, msg = session._exec_notify(item)
assert "rate limit" in msg.lower()
def test_rate_limit_not_consumed_on_failure(self, tmp_path):
"""Failed delivery should not consume rate limit slots."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch(
"turnstone.core.session.httpx.post",
side_effect=ConnectionError("refused"),
),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
# All fail — counter should stay at 0
for _i in range(3):
session._exec_notify(item)
assert session._notify_count == 0
def test_counter_on_init(self):
session = _make_session()
assert session._notify_count == 0
def test_http_failure_reported(self, tmp_path):
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "",
"channel_type": "discord",
"channel_id": "999",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch(
"turnstone.core.session.httpx.post",
side_effect=ConnectionError("refused"),
),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
# Error message should be generic (no internal details)
assert "delivery failed" in msg.lower()
assert "refused" not in msg
assert "ch-1" not in msg
def test_first_healthy_only(self, tmp_path):
"""Only the first healthy gateway should receive the request."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
storage.register_service("channel", "ch-2", "http://localhost:8092")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp) as mock_post,
patch.dict("os.environ", {}, clear=False),
):
session._exec_notify(item)
# Should only have been called once (first healthy)
assert mock_post.call_count == 1
def test_ssrf_protection(self, tmp_path):
"""URLs with non-http(s) schemes should be skipped."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
# Register a service with an invalid scheme
storage.register_service("channel", "ch-bad", "ftp://evil.example.com")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post") as mock_post,
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
# httpx.post should never be called for ftp:// URL
mock_post.assert_not_called()
assert "delivery failed" in msg.lower()
def test_retry_on_no_services(self, tmp_path):
"""Retries service lookup when no gateways are initially available."""
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
# First two calls return empty, third returns a service
call_count = 0
def _list_services(stype: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
nonlocal call_count
call_count += 1
if call_count <= 2:
return []
return [
{
"service_type": "channel",
"service_id": "ch-1",
"url": "http://localhost:8091",
"metadata": "{}",
"last_heartbeat": "",
"created": "",
}
]
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
from unittest.mock import patch
mock_storage = MagicMock()
mock_storage.list_services = _list_services
with (
patch("turnstone.core.session.get_storage", return_value=mock_storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep") as mock_sleep,
):
call_id, msg = session._exec_notify(item)
assert "sent successfully" in msg.lower()
# Should have slept twice (retry delays)
assert mock_sleep.call_count == 2
def test_retry_on_all_gateways_failed(self, tmp_path):
"""Retries when all gateways fail on first attempt but succeed on retry."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
call_count = 0
def _post(*args, **kwargs):
nonlocal call_count
call_count += 1
if call_count <= 1:
raise ConnectionError("refused")
resp = MagicMock()
resp.status_code = 200
resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "sent"}]
}
return resp
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", side_effect=_post),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep") as mock_sleep,
):
call_id, msg = session._exec_notify(item)
assert "sent successfully" in msg.lower()
assert mock_sleep.call_count == 1
def test_no_services_logs_warning(self, tmp_path):
"""Server-side warning is logged when no services are available."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.time.sleep"),
patch("turnstone.core.session.log") as mock_log,
):
session._exec_notify(item)
# Should have logged warnings for retries + final exhaustion
warning_calls = [c for c in mock_log.warning.call_args_list]
assert len(warning_calls) >= 3 # 2 retry warnings + 1 exhaustion
events = [c.args[0] for c in warning_calls]
assert "notify.no_services" in events
assert "notify.no_services_exhausted" in events
def test_all_gateways_failed_logs_warning(self, tmp_path):
"""Server-side warning is logged when all gateways fail."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch(
"turnstone.core.session.httpx.post",
side_effect=ConnectionError("refused"),
),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
patch("turnstone.core.session.log") as mock_log,
):
session._exec_notify(item)
warning_calls = [c for c in mock_log.warning.call_args_list]
events = [c.args[0] for c in warning_calls]
# 2 retry warnings + 1 final failure
assert "notify.all_gateways_failed" in events
assert "notify.delivery_failed" in events
def test_gateway_200_but_no_delivery(self, tmp_path):
"""HTTP 200 with all results failed should not count as success."""
from turnstone.core.storage._sqlite import SQLiteBackend
storage = SQLiteBackend(str(tmp_path / "test.db"))
storage.register_service("channel", "ch-1", "http://localhost:8091")
session = _make_session()
item = {
"call_id": "call_1",
"func_name": "notify",
"message": "Hello!",
"username": "admin",
"channel_type": "",
"channel_id": "",
"title": "",
}
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.json.return_value = {
"results": [{"channel_type": "discord", "channel_id": "123", "status": "no_adapter"}]
}
from unittest.mock import patch
with (
patch("turnstone.core.session.get_storage", return_value=storage),
patch("turnstone.core.session.httpx.post", return_value=mock_resp),
patch.dict("os.environ", {}, clear=False),
patch("turnstone.core.session.time.sleep"),
):
call_id, msg = session._exec_notify(item)
assert "delivery failed" in msg.lower()
assert session._notify_count == 0
+1 -1
View File
@@ -27,7 +27,7 @@ class TestServerSpec:
expected = {
"/v1/api/workstreams",
"/v1/api/dashboard",
"/v1/api/sessions",
"/v1/api/workstreams/saved",
"/v1/api/send",
"/v1/api/approve",
"/v1/api/plan",
+153
View File
@@ -1103,6 +1103,43 @@ class TestOpenAIParameterGating:
assert "temperature" not in kwargs
assert "reasoning_effort" not in kwargs
def test_gpt5_pro_unsupported_effort_falls_back(self) -> None:
"""GPT-5 pro only supports 'high'; unsupported values fall back to default."""
caps = self.provider.get_capabilities("gpt-5-pro")
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="medium")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "high" # fell back to default
def test_gpt5_pro_supported_effort_passes_through(self) -> None:
"""GPT-5 pro accepts 'high' directly."""
caps = self.provider.get_capabilities("gpt-5-pro")
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="high")
assert kwargs["reasoning_effort"] == "high"
def test_gpt54_1m_context_and_effort(self) -> None:
"""GPT-5.4: 1M context, temperature when effort=none, xhigh supported."""
caps = self.provider.get_capabilities("gpt-5.4")
assert caps.context_window == 1050000
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="none")
assert kwargs["temperature"] == 0.7
assert "reasoning_effort" not in kwargs
kwargs2: dict[str, Any] = {}
self.provider._apply_model_params(kwargs2, caps, temperature=0.7, reasoning_effort="xhigh")
assert "temperature" not in kwargs2
assert kwargs2["reasoning_effort"] == "xhigh"
def test_gpt54_pro_no_temperature_always_reasoning(self) -> None:
"""GPT-5.4 pro: no temperature, medium/high/xhigh only."""
caps = self.provider.get_capabilities("gpt-5.4-pro")
assert caps.context_window == 1050000
kwargs: dict[str, Any] = {}
self.provider._apply_model_params(kwargs, caps, temperature=0.7, reasoning_effort="low")
assert "temperature" not in kwargs
assert kwargs["reasoning_effort"] == "medium" # fell back from unsupported "low"
class TestAnthropicReasoningNone:
"""Verify 'none' effort disables thinking for manual-thinking models."""
@@ -1896,3 +1933,119 @@ class TestAnthropicProviderBlocks:
assert blocks[1]["input"] == {"query": "test"} # parsed from accumulated JSON
assert blocks[2]["type"] == "web_search_tool_result"
assert blocks[2]["encrypted_content"] == "enc_data"
# ---------------------------------------------------------------------------
# Tool search tests
# ---------------------------------------------------------------------------
class TestAnthropicToolSearch:
"""Test Anthropic provider tool search injection."""
@pytest.fixture()
def provider(self):
from turnstone.core.providers._anthropic import AnthropicProvider
return AnthropicProvider()
def test_tool_search_capability_flag(self, provider):
caps = provider.get_capabilities("claude-opus-4-6-20260101")
assert caps.supports_tool_search is True
def test_tool_search_not_supported_on_haiku(self, provider):
caps = provider.get_capabilities("claude-haiku-4-5-20251001")
assert caps.supports_tool_search is False
def test_inject_tool_search_marks_deferred(self, provider):
caps = provider.get_capabilities("claude-opus-4-6-20260101")
tools = [
{"name": "bash", "description": "Run commands", "input_schema": {}},
{
"name": "mcp__github__create_issue",
"description": "Create issue",
"input_schema": {},
},
]
deferred = frozenset(["mcp__github__create_issue"])
result = provider._inject_tool_search(tools, caps, deferred)
# bash should not be deferred
assert result[0].get("defer_loading") is None or result[0].get("defer_loading") is False
# MCP tool should be deferred
assert result[1]["defer_loading"] is True
# Search tool should be appended
assert result[-1]["type"] == "tool_search_tool_bm25_20251119"
assert result[-1]["name"] == "tool_search"
def test_inject_tool_search_no_op_without_deferred(self, provider):
caps = provider.get_capabilities("claude-opus-4-6-20260101")
tools = [{"name": "bash", "description": "Run commands", "input_schema": {}}]
result = provider._inject_tool_search(tools, caps, None)
assert result == tools
def test_inject_tool_search_no_op_on_unsupported_model(self, provider):
caps = provider.get_capabilities("claude-haiku-4-5-20251001")
tools = [{"name": "bash", "description": "Run commands", "input_schema": {}}]
deferred = frozenset(["some_tool"])
result = provider._inject_tool_search(tools, caps, deferred)
assert result == tools
class TestOpenAIToolSearch:
"""Test OpenAI provider tool search injection."""
@pytest.fixture()
def provider(self):
return OpenAIProvider()
def test_tool_search_capability_on_gpt54(self, provider):
caps = provider.get_capabilities("gpt-5.4")
assert caps.supports_tool_search is True
def test_tool_search_not_supported_on_gpt5(self, provider):
caps = provider.get_capabilities("gpt-5")
assert caps.supports_tool_search is False
def test_apply_tool_search_marks_deferred(self, provider):
caps = provider.get_capabilities("gpt-5.4")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
{
"type": "function",
"function": {"name": "mcp__slack__send", "description": "Send message"},
},
]
deferred = frozenset(["mcp__slack__send"])
result = provider._apply_tool_search(caps, tools, deferred)
assert result is not None
# bash not deferred
assert result[0].get("defer_loading") is None or result[0].get("defer_loading") is False
# slack tool deferred
assert result[1]["defer_loading"] is True
def test_apply_tool_search_no_op_without_deferred(self, provider):
caps = provider.get_capabilities("gpt-5.4")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
]
result = provider._apply_tool_search(caps, tools, None)
assert result == tools
def test_apply_tool_search_no_op_on_unsupported_model(self, provider):
caps = provider.get_capabilities("gpt-5")
tools = [
{"type": "function", "function": {"name": "bash", "description": "Run commands"}},
]
deferred = frozenset(["some_tool"])
result = provider._apply_tool_search(caps, tools, deferred)
assert result == tools
class TestModelCapabilitiesToolSearch:
"""Test supports_tool_search defaults and values."""
def test_default_is_false(self):
from turnstone.core.providers._protocol import ModelCapabilities
caps = ModelCapabilities()
assert caps.supports_tool_search is False
+24 -39
View File
@@ -1,6 +1,6 @@
"""Tests for the atomic workstream resumption flow.
Covers CreateWorkstreamMessage resume_session field, SessionResumedEvent,
Covers CreateWorkstreamMessage resume_ws field, WorkstreamResumedEvent,
WorkstreamCreatedEvent resumed fields, and server endpoint handling.
"""
@@ -10,8 +10,8 @@ import json
from turnstone.mq.protocol import (
CreateWorkstreamMessage,
SessionResumedEvent,
WorkstreamCreatedEvent,
WorkstreamResumedEvent,
)
# ---------------------------------------------------------------------------
@@ -20,93 +20,78 @@ from turnstone.mq.protocol import (
class TestCreateWorkstreamMessageResumeField:
def test_resume_session_defaults_empty(self) -> None:
def test_resume_ws_defaults_empty(self) -> None:
msg = CreateWorkstreamMessage(name="test")
assert msg.resume_session == ""
assert msg.resume_ws == ""
def test_resume_session_set(self) -> None:
msg = CreateWorkstreamMessage(name="test", resume_session="sess-abc")
assert msg.resume_session == "sess-abc"
def test_resume_ws_set(self) -> None:
msg = CreateWorkstreamMessage(name="test", resume_ws="ws-abc")
assert msg.resume_ws == "ws-abc"
def test_resume_session_serializes(self) -> None:
msg = CreateWorkstreamMessage(resume_session="sess-xyz")
def test_resume_ws_serializes(self) -> None:
msg = CreateWorkstreamMessage(resume_ws="ws-xyz")
data = json.loads(msg.to_json())
assert data["resume_session"] == "sess-xyz"
assert data["resume_ws"] == "ws-xyz"
def test_resume_session_deserializes(self) -> None:
msg = CreateWorkstreamMessage(resume_session="sess-123")
def test_resume_ws_deserializes(self) -> None:
msg = CreateWorkstreamMessage(resume_ws="ws-123")
raw = msg.to_json()
from turnstone.mq.protocol import InboundMessage
restored = InboundMessage.from_json(raw)
assert getattr(restored, "resume_session", "") == "sess-123"
assert getattr(restored, "resume_ws", "") == "ws-123"
class TestWorkstreamCreatedEventResumeFields:
def test_default_not_resumed(self) -> None:
event = WorkstreamCreatedEvent(ws_id="ws-1", name="test")
assert event.resumed is False
assert event.session_id == ""
assert event.message_count == 0
def test_resumed_fields(self) -> None:
event = WorkstreamCreatedEvent(
ws_id="ws-1", name="test", resumed=True, session_id="s-1", message_count=42
)
event = WorkstreamCreatedEvent(ws_id="ws-1", name="test", resumed=True, message_count=42)
assert event.resumed is True
assert event.session_id == "s-1"
assert event.message_count == 42
def test_serializes_resumed_fields(self) -> None:
event = WorkstreamCreatedEvent(
ws_id="ws-1", resumed=True, session_id="s-1", message_count=10
)
event = WorkstreamCreatedEvent(ws_id="ws-1", resumed=True, message_count=10)
data = json.loads(event.to_json())
assert data["resumed"] is True
assert data["session_id"] == "s-1"
assert data["message_count"] == 10
def test_deserializes_resumed_fields(self) -> None:
event = WorkstreamCreatedEvent(
ws_id="ws-1", resumed=True, session_id="s-1", message_count=5
)
event = WorkstreamCreatedEvent(ws_id="ws-1", resumed=True, message_count=5)
from turnstone.mq.protocol import OutboundEvent
restored = OutboundEvent.from_json(event.to_json())
assert isinstance(restored, WorkstreamCreatedEvent)
assert restored.resumed is True
assert restored.session_id == "s-1"
assert restored.message_count == 5
class TestSessionResumedEvent:
class TestWorkstreamResumedEvent:
def test_defaults(self) -> None:
event = SessionResumedEvent(ws_id="ws-1")
assert event.type == "session_resumed"
assert event.session_id == ""
event = WorkstreamResumedEvent(ws_id="ws-1")
assert event.type == "ws_resumed"
assert event.message_count == 0
assert event.name == ""
def test_with_values(self) -> None:
event = SessionResumedEvent(
ws_id="ws-1", session_id="s-abc", message_count=25, name="My Chat"
)
assert event.session_id == "s-abc"
event = WorkstreamResumedEvent(ws_id="ws-1", message_count=25, name="My Chat")
assert event.message_count == 25
assert event.name == "My Chat"
def test_round_trip(self) -> None:
event = SessionResumedEvent(ws_id="ws-1", session_id="s-abc", message_count=10, name="Chat")
event = WorkstreamResumedEvent(ws_id="ws-1", message_count=10, name="Chat")
from turnstone.mq.protocol import OutboundEvent
restored = OutboundEvent.from_json(event.to_json())
assert isinstance(restored, SessionResumedEvent)
assert restored.session_id == "s-abc"
assert isinstance(restored, WorkstreamResumedEvent)
assert restored.message_count == 10
assert restored.name == "Chat"
def test_registered_in_outbound_registry(self) -> None:
from turnstone.mq.protocol import _OUTBOUND_REGISTRY
assert "session_resumed" in _OUTBOUND_REGISTRY
assert _OUTBOUND_REGISTRY["session_resumed"] is SessionResumedEvent
assert "ws_resumed" in _OUTBOUND_REGISTRY
assert _OUTBOUND_REGISTRY["ws_resumed"] is WorkstreamResumedEvent
+264
View File
@@ -0,0 +1,264 @@
"""Tests for scheduled task admin API endpoints."""
from __future__ import annotations
import pytest
from starlette.applications import Starlette
from starlette.routing import Mount, Route
from starlette.testclient import TestClient
from turnstone.console.server import (
admin_create_schedule,
admin_delete_schedule,
admin_get_schedule,
admin_list_schedule_runs,
admin_list_schedules,
admin_update_schedule,
)
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
"""Fresh SQLite backend for each test."""
return SQLiteBackend(str(tmp_path / "test.db"))
@pytest.fixture
def client(storage):
"""TestClient with storage and auth bypassed."""
app = Starlette(
routes=[
Mount(
"/v1",
routes=[
Route("/api/admin/schedules", admin_list_schedules),
Route("/api/admin/schedules", admin_create_schedule, methods=["POST"]),
Route("/api/admin/schedules/{task_id}", admin_get_schedule),
Route(
"/api/admin/schedules/{task_id}",
admin_update_schedule,
methods=["PUT"],
),
Route(
"/api/admin/schedules/{task_id}",
admin_delete_schedule,
methods=["DELETE"],
),
Route(
"/api/admin/schedules/{task_id}/runs",
admin_list_schedule_runs,
),
],
),
],
)
app.state.auth_storage = storage
return TestClient(app)
def _cron_payload(**overrides):
"""Build default cron schedule creation payload."""
defaults = {
"name": "Daily report",
"description": "Generate the summary",
"schedule_type": "cron",
"cron_expr": "0 9 * * *",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Generate the daily report",
}
defaults.update(overrides)
return defaults
def _at_payload(**overrides):
"""Build default at-time schedule creation payload."""
defaults = {
"name": "One-shot task",
"description": "Run once",
"schedule_type": "at",
"at_time": "2099-01-01T00:00:00+00:00",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Do the thing",
}
defaults.update(overrides)
return defaults
class TestScheduleAPI:
"""Tests for the 6 admin schedule endpoints."""
def test_list_empty(self, client):
resp = client.get("/v1/api/admin/schedules")
assert resp.status_code == 200
data = resp.json()
assert data["schedules"] == []
def test_create_cron(self, client):
resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
assert resp.status_code == 200
task = resp.json()
assert task["name"] == "Daily report"
assert task["schedule_type"] == "cron"
assert task["cron_expr"] == "0 9 * * *"
assert task["enabled"] is True
assert "task_id" in task
assert "created" in task
assert "next_run" in task
assert task["next_run"] != ""
def test_create_at(self, client):
resp = client.post("/v1/api/admin/schedules", json=_at_payload())
assert resp.status_code == 200
task = resp.json()
assert task["schedule_type"] == "at"
assert task["at_time"] == "2099-01-01T00:00:00+00:00"
assert task["next_run"] == "2099-01-01T00:00:00+00:00"
def test_create_missing_name(self, client):
payload = _cron_payload()
del payload["name"]
resp = client.post("/v1/api/admin/schedules", json=payload)
assert resp.status_code == 400
assert "name" in resp.json()["error"].lower()
def test_create_invalid_cron(self, client):
resp = client.post(
"/v1/api/admin/schedules",
json=_cron_payload(cron_expr="not a cron"),
)
assert resp.status_code == 400
assert "cron" in resp.json()["error"].lower()
def test_create_naive_at_time(self, client):
"""Naive timestamps (no timezone) should be rejected."""
resp = client.post(
"/v1/api/admin/schedules",
json=_at_payload(at_time="2099-01-01T00:00:00"),
)
assert resp.status_code == 400
assert "timezone" in resp.json()["error"].lower()
def test_create_past_at_time(self, client):
resp = client.post(
"/v1/api/admin/schedules",
json=_at_payload(at_time="2000-01-01T00:00:00+00:00"),
)
assert resp.status_code == 400
assert "future" in resp.json()["error"].lower()
def test_get_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.get(f"/v1/api/admin/schedules/{task_id}")
assert resp.status_code == 200
assert resp.json()["task_id"] == task_id
assert resp.json()["name"] == "Daily report"
def test_get_nonexistent(self, client):
resp = client.get("/v1/api/admin/schedules/nonexistent_id")
assert resp.status_code == 404
def test_update_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.put(
f"/v1/api/admin/schedules/{task_id}",
json={"name": "Weekly report"},
)
assert resp.status_code == 200
assert resp.json()["name"] == "Weekly report"
# Verify via GET
get_resp = client.get(f"/v1/api/admin/schedules/{task_id}")
assert get_resp.json()["name"] == "Weekly report"
def test_update_nonexistent(self, client):
resp = client.put(
"/v1/api/admin/schedules/nonexistent_id",
json={"name": "Nope"},
)
assert resp.status_code == 404
def test_delete_schedule(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.delete(f"/v1/api/admin/schedules/{task_id}")
assert resp.status_code == 200
assert resp.json()["status"] == "ok"
# Verify gone
get_resp = client.get(f"/v1/api/admin/schedules/{task_id}")
assert get_resp.status_code == 404
def test_delete_nonexistent(self, client):
resp = client.delete("/v1/api/admin/schedules/nonexistent_id")
assert resp.status_code == 404
def test_list_runs_empty(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
resp = client.get(f"/v1/api/admin/schedules/{task_id}/runs")
assert resp.status_code == 200
assert resp.json()["runs"] == []
def test_list_runs_nonexistent(self, client):
resp = client.get("/v1/api/admin/schedules/nonexistent_id/runs")
assert resp.status_code == 404
def test_create_specific_node_target(self, client):
payload = _cron_payload(target_mode="node-custom-001")
resp = client.post("/v1/api/admin/schedules", json=payload)
assert resp.status_code == 200
data = resp.json()
assert data["target_mode"] == "node-custom-001"
def test_list_runs_with_data(self, client, storage):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
# Record runs directly in storage
storage.record_task_run(
run_id="run_001",
task_id=task_id,
node_id="node-1",
ws_id="ws_abc",
correlation_id="corr_001",
started="2025-06-01T09:00:00",
status="dispatched",
error="",
)
storage.record_task_run(
run_id="run_002",
task_id=task_id,
node_id="node-2",
ws_id="",
correlation_id="corr_002",
started="2025-06-01T09:01:00",
status="failed",
error="No reachable nodes",
)
resp = client.get(f"/v1/api/admin/schedules/{task_id}/runs")
assert resp.status_code == 200
runs = resp.json()["runs"]
assert len(runs) == 2
# Most recent first
assert runs[0]["run_id"] == "run_002"
assert runs[0]["status"] == "failed"
assert runs[1]["run_id"] == "run_001"
def test_list_runs_invalid_limit(self, client):
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
task_id = create_resp.json()["task_id"]
# Invalid limit should not crash — falls back to 50
resp = client.get(f"/v1/api/admin/schedules/{task_id}/runs?limit=abc")
assert resp.status_code == 200
assert resp.json()["runs"] == []
+259
View File
@@ -0,0 +1,259 @@
"""Tests for scheduled_tasks and scheduled_task_runs storage CRUD."""
from __future__ import annotations
import time
import pytest
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def db(tmp_path):
"""Fresh SQLite backend for each test."""
backend = SQLiteBackend(str(tmp_path / "test.db"))
return backend
def _make_task_kwargs(**overrides):
"""Build default kwargs for create_scheduled_task."""
defaults = {
"task_id": "task_001",
"name": "Daily report",
"description": "Generate the daily summary",
"schedule_type": "cron",
"cron_expr": "0 9 * * *",
"at_time": "",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Generate the daily report",
"auto_approve": False,
"auto_approve_tools": [],
"created_by": "u_admin",
"next_run": "2099-01-01T09:00:00",
}
defaults.update(overrides)
return defaults
class TestScheduledTaskCRUD:
"""Tests for scheduled_tasks table operations."""
def test_create_and_get(self, db):
db.create_scheduled_task(**_make_task_kwargs())
result = db.get_scheduled_task("task_001")
assert result is not None
assert result["task_id"] == "task_001"
assert result["name"] == "Daily report"
assert result["description"] == "Generate the daily summary"
assert result["schedule_type"] == "cron"
assert result["cron_expr"] == "0 9 * * *"
assert result["at_time"] == ""
assert result["target_mode"] == "auto"
assert result["model"] == "gpt-5"
assert result["initial_message"] == "Generate the daily report"
assert result["auto_approve"] == 0
assert result["auto_approve_tools"] == ""
assert result["enabled"] == 1
assert result["created_by"] == "u_admin"
assert result["next_run"] == "2099-01-01T09:00:00"
assert "created" in result
assert "updated" in result
def test_get_nonexistent(self, db):
assert db.get_scheduled_task("no_such_task") is None
def test_create_duplicate_noop(self, db):
db.create_scheduled_task(**_make_task_kwargs(name="First"))
db.create_scheduled_task(**_make_task_kwargs(name="Second"))
result = db.get_scheduled_task("task_001")
assert result is not None
assert result["name"] == "First" # first write wins
def test_list_tasks(self, db):
db.create_scheduled_task(**_make_task_kwargs(task_id="task_a", name="Alpha"))
# Ensure different created timestamps (resolution is 1 second)
time.sleep(1.1)
db.create_scheduled_task(**_make_task_kwargs(task_id="task_b", name="Beta"))
tasks = db.list_scheduled_tasks()
assert len(tasks) == 2
# Ordered by created DESC — most recent first
assert tasks[0]["task_id"] == "task_b"
assert tasks[1]["task_id"] == "task_a"
def test_update_task(self, db):
db.create_scheduled_task(**_make_task_kwargs())
original = db.get_scheduled_task("task_001")
assert original is not None
original_updated = original["updated"]
time.sleep(0.05)
result = db.update_scheduled_task("task_001", name="Weekly report")
assert result is True
updated = db.get_scheduled_task("task_001")
assert updated is not None
assert updated["name"] == "Weekly report"
assert updated["updated"] >= original_updated
def test_update_enable_disable(self, db):
db.create_scheduled_task(**_make_task_kwargs())
task = db.get_scheduled_task("task_001")
assert task is not None
assert task["enabled"] == 1
db.update_scheduled_task("task_001", enabled=False)
task = db.get_scheduled_task("task_001")
assert task is not None
assert task["enabled"] == 0
db.update_scheduled_task("task_001", enabled=True)
task = db.get_scheduled_task("task_001")
assert task is not None
assert task["enabled"] == 1
def test_delete_task(self, db):
db.create_scheduled_task(**_make_task_kwargs())
assert db.delete_scheduled_task("task_001") is True
assert db.get_scheduled_task("task_001") is None
# Deleting again returns False
assert db.delete_scheduled_task("task_001") is False
def test_delete_cascades_runs(self, db):
db.create_scheduled_task(**_make_task_kwargs())
db.record_task_run(
run_id="run_001",
task_id="task_001",
node_id="node_1",
ws_id="ws_abc",
correlation_id="corr_001",
started="2025-01-01T09:00:00",
status="dispatched",
error="",
)
assert len(db.list_task_runs("task_001")) == 1
db.delete_scheduled_task("task_001")
assert db.list_task_runs("task_001") == []
def test_list_due_tasks(self, db):
db.create_scheduled_task(
**_make_task_kwargs(task_id="past", next_run="2020-01-01T00:00:00")
)
db.create_scheduled_task(
**_make_task_kwargs(task_id="future", next_run="2099-12-31T23:59:59")
)
now = "2025-06-01T12:00:00"
due = db.list_due_tasks(now)
assert len(due) == 1
assert due[0]["task_id"] == "past"
def test_list_due_tasks_skips_disabled(self, db):
db.create_scheduled_task(
**_make_task_kwargs(task_id="disabled_task", next_run="2020-01-01T00:00:00")
)
db.update_scheduled_task("disabled_task", enabled=False)
due = db.list_due_tasks("2025-06-01T12:00:00")
assert len(due) == 0
def test_list_due_tasks_empty_next_run(self, db):
db.create_scheduled_task(**_make_task_kwargs(task_id="empty_next", next_run=""))
due = db.list_due_tasks("2099-12-31T23:59:59")
assert len(due) == 0
def test_at_task_fields(self, db):
db.create_scheduled_task(
**_make_task_kwargs(
task_id="at_task",
schedule_type="at",
cron_expr="",
at_time="2099-06-15T14:00:00",
next_run="2099-06-15T14:00:00",
)
)
result = db.get_scheduled_task("at_task")
assert result is not None
assert result["schedule_type"] == "at"
assert result["at_time"] == "2099-06-15T14:00:00"
class TestScheduledTaskRuns:
"""Tests for scheduled_task_runs table operations."""
def test_record_and_list(self, db):
db.create_scheduled_task(**_make_task_kwargs())
db.record_task_run(
run_id="run_a",
task_id="task_001",
node_id="node_1",
ws_id="ws_1",
correlation_id="corr_a",
started="2025-01-01T09:00:00",
status="dispatched",
error="",
)
db.record_task_run(
run_id="run_b",
task_id="task_001",
node_id="node_2",
ws_id="ws_2",
correlation_id="corr_b",
started="2025-01-02T09:00:00",
status="dispatched",
error="",
)
runs = db.list_task_runs("task_001")
assert len(runs) == 2
# Ordered by started DESC — most recent first
assert runs[0]["run_id"] == "run_b"
assert runs[1]["run_id"] == "run_a"
def test_list_runs_respects_limit(self, db):
db.create_scheduled_task(**_make_task_kwargs())
for i in range(3):
db.record_task_run(
run_id=f"run_{i}",
task_id="task_001",
node_id="node_1",
ws_id="",
correlation_id=f"corr_{i}",
started=f"2025-01-0{i + 1}T09:00:00",
status="dispatched",
error="",
)
runs = db.list_task_runs("task_001", limit=2)
assert len(runs) == 2
def test_list_runs_empty(self, db):
assert db.list_task_runs("no_such_task") == []
def test_prune_task_runs(self, db):
db.create_scheduled_task(**_make_task_kwargs())
# Old run (should be pruned)
db.record_task_run(
run_id="old_run",
task_id="task_001",
node_id="node_1",
ws_id="",
correlation_id="c_old",
started="2020-01-01T00:00:00",
status="dispatched",
error="",
)
# Recent run (should survive)
db.record_task_run(
run_id="new_run",
task_id="task_001",
node_id="node_1",
ws_id="",
correlation_id="c_new",
started="2099-01-01T00:00:00",
status="dispatched",
error="",
)
pruned = db.prune_task_runs(retention_days=90)
assert pruned == 1
runs = db.list_task_runs("task_001")
assert len(runs) == 1
assert runs[0]["run_id"] == "new_run"
+272
View File
@@ -0,0 +1,272 @@
"""Tests for turnstone.console.scheduler — TaskScheduler tick and dispatch."""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from turnstone.console.scheduler import TaskScheduler
@pytest.fixture
def mocks():
"""Broker, collector, and storage mocks for scheduler tests."""
broker = MagicMock()
broker._redis = MagicMock()
collector = MagicMock()
storage = MagicMock()
return broker, collector, storage
def _make_task(**overrides):
"""Build a minimal task dict matching storage row format."""
defaults = {
"task_id": "task_001",
"name": "Test task",
"description": "",
"schedule_type": "cron",
"cron_expr": "0 9 * * *",
"at_time": "",
"target_mode": "auto",
"model": "gpt-5",
"initial_message": "Run the tests",
"auto_approve": 0,
"auto_approve_tools": "",
"enabled": 1,
"created_by": "u_admin",
"next_run": "2020-01-01T09:00:00",
"last_run": "",
"created": "2020-01-01T00:00:00",
"updated": "2020-01-01T00:00:00",
}
defaults.update(overrides)
return defaults
def _make_node(node_id="node-001", reachable=True, ws_total=2, max_ws=10):
"""Build a minimal node dict matching collector output."""
return {
"node_id": node_id,
"reachable": reachable,
"ws_total": ws_total,
"max_ws": max_ws,
}
class TestSchedulerTick:
"""Tests for _tick() lock acquisition and dispatch logic."""
def test_tick_acquires_lock(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
storage.list_due_tasks.return_value = []
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker._redis.set.assert_called_once()
storage.list_due_tasks.assert_called_once()
# Lock released via Lua eval (conditional delete)
broker._redis.eval.assert_called_once()
def test_tick_skips_when_locked(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = None # lock held by another console
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
storage.list_due_tasks.assert_not_called()
def test_dispatch_auto_mode(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="auto")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
_, kwargs = broker.push_inbound.call_args
assert (
kwargs.get("node_id") == "node-001"
or broker.push_inbound.call_args[1].get("node_id") == "node-001"
)
storage.record_task_run.assert_called_once()
run_kwargs = storage.record_task_run.call_args[1]
assert run_kwargs["node_id"] == "node-001"
assert run_kwargs["status"] == "dispatched"
def test_dispatch_pool_mode(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="pool")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
# Pool dispatch calls push_inbound without node_id kwarg
args, kwargs = broker.push_inbound.call_args
assert kwargs.get("node_id") is None or "node_id" not in kwargs
storage.record_task_run.assert_called_once()
run_kwargs = storage.record_task_run.call_args[1]
assert run_kwargs["node_id"] == "pool"
def test_dispatch_all_mode(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="all")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = (
[_make_node("node-001"), _make_node("node-002")],
2,
)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
assert broker.push_inbound.call_count == 2
assert storage.record_task_run.call_count == 2
def test_dispatch_specific_node(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="node-001")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
_, kwargs = broker.push_inbound.call_args
assert kwargs["node_id"] == "node-001"
def test_at_task_disables_after_dispatch(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(schedule_type="at", cron_expr="", at_time="2099-01-01T00:00:00")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
# At-task should be disabled after dispatch
update_calls = storage.update_scheduled_task.call_args_list
assert len(update_calls) == 1
args, kwargs = update_calls[0]
assert args[0] == "task_001"
assert kwargs["enabled"] is False
assert kwargs["next_run"] == ""
def test_cron_task_updates_next_run(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(schedule_type="cron", cron_expr="0 9 * * *")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([_make_node()], 1)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
update_calls = storage.update_scheduled_task.call_args_list
assert len(update_calls) == 1
_, kwargs = update_calls[0]
assert kwargs["next_run"] != ""
assert "enabled" not in kwargs # cron tasks stay enabled
def test_no_reachable_nodes_records_failure(self, mocks):
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="auto")
storage.list_due_tasks.return_value = [task]
# No reachable nodes
collector.get_nodes.return_value = (
[_make_node("node-001", reachable=False)],
1,
)
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_not_called()
storage.record_task_run.assert_called_once()
run_kwargs = storage.record_task_run.call_args[1]
assert run_kwargs["status"] == "failed"
assert run_kwargs["error"] != ""
def test_failure_does_not_advance_schedule(self, mocks):
"""When dispatch fails, last_run/next_run should not be updated."""
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="auto")
storage.list_due_tasks.return_value = [task]
collector.get_nodes.return_value = ([], 0) # no nodes at all
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
# update_scheduled_task should NOT be called (no last_run/next_run advance)
storage.update_scheduled_task.assert_not_called()
def test_fan_out_capped(self, mocks):
"""Fan-out 'all' mode should respect max_fan_out limit."""
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="all")
storage.list_due_tasks.return_value = [task]
# 10 reachable nodes but max_fan_out=3
nodes = [_make_node(f"node-{i:03d}") for i in range(10)]
collector.get_nodes.return_value = (nodes, 10)
scheduler = TaskScheduler(broker, collector, storage, max_fan_out=3)
scheduler._tick()
assert broker.push_inbound.call_count == 3
assert storage.record_task_run.call_count == 3
def test_specific_node_target(self, mocks):
"""Non-enum target_mode is treated as a specific node_id."""
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="node-custom-123")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
broker.push_inbound.assert_called_once()
call_kwargs = broker.push_inbound.call_args
assert call_kwargs[1]["node_id"] == "node-custom-123"
def test_user_id_in_dispatched_message(self, mocks):
"""Dispatched message should include created_by as user_id."""
import json
broker, collector, storage = mocks
broker._redis.set.return_value = True
task = _make_task(target_mode="pool", created_by="u_scheduler_admin")
storage.list_due_tasks.return_value = [task]
scheduler = TaskScheduler(broker, collector, storage)
scheduler._tick()
msg_json = broker.push_inbound.call_args[0][0]
msg_data = json.loads(msg_json)
assert msg_data["user_id"] == "u_scheduler_admin"
+139
View File
@@ -2,6 +2,8 @@
from __future__ import annotations
import json
import httpx
import pytest
@@ -240,3 +242,140 @@ async def test_query_params_passed():
assert "state=running" in captured_url[0]
assert "page=2" in captured_url[0]
assert "per_page=25" in captured_url[0]
# ---------------------------------------------------------------------------
# Schedules
# ---------------------------------------------------------------------------
_SCHEDULE_FIXTURE = {
"task_id": "t1",
"name": "nightly",
"description": "",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run nightly checks",
"auto_approve": False,
"auto_approve_tools": [],
"enabled": True,
"created_by": "u1",
"last_run": None,
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-05T12:00:00Z",
"updated": "2026-03-05T12:00:00Z",
}
@pytest.mark.anyio
async def test_list_schedules():
transport = _mock_transport(
{"GET /v1/api/admin/schedules": _json_response({"schedules": [_SCHEDULE_FIXTURE]})}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.list_schedules()
assert len(resp.schedules) == 1
assert resp.schedules[0].task_id == "t1"
assert resp.schedules[0].name == "nightly"
@pytest.mark.anyio
async def test_create_schedule():
captured_body: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
captured_body.append(json.loads(request.content))
return _json_response(_SCHEDULE_FIXTURE)
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.create_schedule(
name="nightly",
schedule_type="cron",
initial_message="Run nightly checks",
cron_expr="0 2 * * *",
)
assert resp.task_id == "t1"
body = captured_body[0]
assert body["name"] == "nightly"
assert body["schedule_type"] == "cron"
assert body["cron_expr"] == "0 2 * * *"
assert body["initial_message"] == "Run nightly checks"
# Optional fields with defaults should not appear when not set
assert "description" not in body
assert "model" not in body
@pytest.mark.anyio
async def test_get_schedule():
transport = _mock_transport(
{"GET /v1/api/admin/schedules/t1": _json_response(_SCHEDULE_FIXTURE)}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.get_schedule("t1")
assert resp.task_id == "t1"
assert resp.schedule_type == "cron"
@pytest.mark.anyio
async def test_update_schedule_partial():
"""Only explicitly-passed fields should appear in the request body."""
captured_body: list[dict] = []
def handler(request: httpx.Request) -> httpx.Response:
captured_body.append(json.loads(request.content))
return _json_response({**_SCHEDULE_FIXTURE, "enabled": False})
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.update_schedule("t1", enabled=False)
assert resp.enabled is False
body = captured_body[0]
assert body == {"enabled": False}
@pytest.mark.anyio
async def test_delete_schedule():
transport = _mock_transport(
{"DELETE /v1/api/admin/schedules/t1": _json_response({"status": "ok"})}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.delete_schedule("t1")
assert resp.status == "ok"
@pytest.mark.anyio
async def test_list_schedule_runs():
transport = _mock_transport(
{
"GET /v1/api/admin/schedules/t1/runs": _json_response(
{
"runs": [
{
"run_id": "r1",
"task_id": "t1",
"node_id": "n1",
"ws_id": "ws1",
"correlation_id": "c1",
"started": "2026-03-05T02:00:00Z",
"status": "dispatched",
"error": "",
}
]
}
)
}
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneConsole(httpx_client=hc)
resp = await client.list_schedule_runs("t1", limit=10)
assert len(resp.runs) == 1
assert resp.runs[0].run_id == "r1"
assert resp.runs[0].status == "dispatched"
+7 -7
View File
@@ -148,19 +148,19 @@ async def test_command():
# ---------------------------------------------------------------------------
# Sessions
# History
# ---------------------------------------------------------------------------
@pytest.mark.anyio
async def test_list_sessions():
async def test_list_saved_workstreams():
transport = _mock_transport(
{
"GET /v1/api/sessions": _json_response(
"GET /v1/api/workstreams/saved": _json_response(
{
"sessions": [
"workstreams": [
{
"session_id": "s1",
"ws_id": "s1",
"title": "test",
"created": "2024-01-01",
"updated": "2024-01-02",
@@ -173,8 +173,8 @@ async def test_list_sessions():
)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneServer(httpx_client=hc)
resp = await client.list_sessions()
assert len(resp.sessions) == 1
resp = await client.list_saved_workstreams()
assert len(resp.workstreams) == 1
# ---------------------------------------------------------------------------
+2 -2
View File
@@ -609,7 +609,7 @@ class TestServerHealthMetrics:
mock_ui._ws_context_ratio = 0.0
mock_session = MagicMock()
mock_session.session_id = "test-session-id"
mock_session.ws_id = "test-session-id"
mock_ws = MagicMock()
mock_ws.id = "test-ws"
@@ -785,7 +785,7 @@ class TestServerRateLimiting:
mock_ui._ws_context_ratio = 0.0
mock_session = MagicMock()
mock_session.session_id = "test-session-id"
mock_session.ws_id = "test-session-id"
mock_ws = MagicMock()
mock_ws.id = "test-ws"
+86
View File
@@ -0,0 +1,86 @@
"""Tests for the services registry storage methods."""
from __future__ import annotations
import pytest
from turnstone.core.storage._sqlite import SQLiteBackend
@pytest.fixture
def storage(tmp_path):
return SQLiteBackend(str(tmp_path / "test.db"))
class TestServiceRegistry:
def test_register_and_list(self, storage):
storage.register_service("channel", "ch-1", "http://localhost:8091")
services = storage.list_services("channel", max_age_seconds=120)
assert len(services) == 1
assert services[0]["service_type"] == "channel"
assert services[0]["service_id"] == "ch-1"
assert services[0]["url"] == "http://localhost:8091"
def test_register_upsert(self, storage):
storage.register_service("channel", "ch-1", "http://old:8091")
storage.register_service("channel", "ch-1", "http://new:8091")
services = storage.list_services("channel", max_age_seconds=120)
assert len(services) == 1
assert services[0]["url"] == "http://new:8091"
def test_heartbeat(self, storage):
storage.register_service("channel", "ch-1", "http://localhost:8091")
result = storage.heartbeat_service("channel", "ch-1")
assert result is True
def test_heartbeat_nonexistent(self, storage):
result = storage.heartbeat_service("channel", "nonexistent")
assert result is False
def test_list_filters_stale(self, storage):
storage.register_service("channel", "ch-1", "http://localhost:8091")
# Manually set heartbeat to the past so it's stale
from datetime import UTC, datetime, timedelta
import sqlalchemy as sa
from turnstone.core.storage._schema import services
old_time = (datetime.now(UTC) - timedelta(seconds=300)).strftime("%Y-%m-%dT%H:%M:%S")
with storage._engine.connect() as conn:
conn.execute(sa.update(services).values(last_heartbeat=old_time))
conn.commit()
# Should be excluded with 120s max age
result = storage.list_services("channel", max_age_seconds=120)
assert len(result) == 0
def test_list_empty(self, storage):
services = storage.list_services("channel", max_age_seconds=120)
assert services == []
def test_list_filters_by_type(self, storage):
storage.register_service("channel", "ch-1", "http://localhost:8091")
storage.register_service("bridge", "br-1", "http://localhost:8080")
channels = storage.list_services("channel", max_age_seconds=120)
bridges = storage.list_services("bridge", max_age_seconds=120)
assert len(channels) == 1
assert len(bridges) == 1
def test_deregister(self, storage):
storage.register_service("channel", "ch-1", "http://localhost:8091")
result = storage.deregister_service("channel", "ch-1")
assert result is True
services = storage.list_services("channel", max_age_seconds=120)
assert services == []
def test_deregister_nonexistent(self, storage):
result = storage.deregister_service("channel", "nonexistent")
assert result is False
def test_metadata(self, storage):
storage.register_service(
"channel", "ch-1", "http://localhost:8091", metadata='{"adapter": "discord"}'
)
services = storage.list_services("channel", max_age_seconds=120)
assert services[0]["metadata"] == '{"adapter": "discord"}'
+5 -5
View File
@@ -161,12 +161,12 @@ class TestPlanExec:
return call_id, content, captured.get("messages", [])
def test_plan_file_uses_session_id(self, tmp_db, tmp_path, monkeypatch):
"""Plan file is named .plan-<session_id>.md, not .plan.md."""
def test_plan_file_uses_ws_id(self, tmp_db, tmp_path, monkeypatch):
"""Plan file is named .plan-<ws_id>.md, not .plan.md."""
monkeypatch.chdir(tmp_path)
session = _make_session()
self._run_plan(session, "add feature")
expected = tmp_path / f".plan-{session._session_id}.md"
expected = tmp_path / f".plan-{session._ws_id}.md"
assert expected.exists(), f"Expected {expected} to be created"
assert not (tmp_path / ".plan.md").exists()
@@ -176,7 +176,7 @@ class TestPlanExec:
session = _make_session()
plan_content = "## Goal\n\nAdd a new endpoint."
self._run_plan(session, "add endpoint", agent_return=plan_content)
plan_file = tmp_path / f".plan-{session._session_id}.md"
plan_file = tmp_path / f".plan-{session._ws_id}.md"
assert plan_file.read_text() == plan_content
def test_two_sessions_produce_different_files(self, tmp_db, tmp_path, monkeypatch):
@@ -184,7 +184,7 @@ class TestPlanExec:
monkeypatch.chdir(tmp_path)
s1 = _make_session()
s2 = _make_session()
assert s1._session_id != s2._session_id
assert s1._ws_id != s2._ws_id
self._run_plan(s1, "feature A")
self._run_plan(s2, "feature B")
files = list(tmp_path.glob(".plan-*.md"))
+178 -179
View File
@@ -1,155 +1,155 @@
"""Tests for session persistence and resume functionality."""
"""Tests for workstream persistence and resume functionality."""
from unittest.mock import MagicMock
import sqlalchemy as sa
from turnstone.core.memory import (
delete_session,
list_sessions,
load_session_config,
load_session_messages,
prune_sessions,
register_session,
resolve_session,
delete_workstream,
list_workstreams_with_history,
load_messages,
load_workstream_config,
prune_workstreams,
register_workstream,
resolve_workstream,
save_message,
save_session_config,
set_session_alias,
update_session_title,
save_workstream_config,
set_workstream_alias,
update_workstream_title,
)
from turnstone.core.session import ChatSession
from turnstone.core.storage import get_storage
# ── Session registration ──────────────────────────────────────────────
# ── Workstream registration ───────────────────────────────────────────
class TestRegisterSession:
class TestRegisterWorkstream:
def test_register_creates_row(self, tmp_db):
register_session("abc123")
# Session exists in DB (resolve works) even without messages
assert resolve_session("abc123") == "abc123"
register_workstream("abc123")
# Workstream exists in DB (resolve works) even without messages
assert resolve_workstream("abc123") == "abc123"
def test_register_with_title(self, tmp_db):
register_session("abc123", title="My Session")
register_workstream("abc123", name="My Workstream")
save_message("abc123", "user", "hello")
rows = list_sessions()
assert rows[0][2] == "My Session" # title
rows = list_workstreams_with_history()
assert rows[0][2] is None # title column (name is separate)
def test_register_idempotent(self, tmp_db):
register_session("abc123", title="First")
register_session("abc123", title="Second") # should be ignored
register_workstream("abc123")
update_workstream_title("abc123", "First")
register_workstream("abc123") # should be ignored
update_workstream_title("abc123", "First") # title is set via update
save_message("abc123", "user", "hello")
rows = list_sessions()
rows = list_workstreams_with_history()
assert len(rows) == 1
assert rows[0][2] == "First" # original title preserved
assert rows[0][2] == "First" # title preserved
def test_update_title(self, tmp_db):
register_session("abc123")
update_session_title("abc123", "New Title")
register_workstream("abc123")
update_workstream_title("abc123", "New Title")
save_message("abc123", "user", "hello")
rows = list_sessions()
rows = list_workstreams_with_history()
assert rows[0][2] == "New Title"
# ── Session alias ─────────────────────────────────────────────────────
# ── Workstream alias ──────────────────────────────────────────────────
class TestSessionAlias:
class TestWorkstreamAlias:
def test_set_alias(self, tmp_db):
register_session("abc123")
assert set_session_alias("abc123", "my-session") is True
register_workstream("abc123")
assert set_workstream_alias("abc123", "my-session") is True
save_message("abc123", "user", "hello")
rows = list_sessions()
rows = list_workstreams_with_history()
assert rows[0][1] == "my-session" # alias
def test_alias_conflict(self, tmp_db):
register_session("abc123")
register_session("def456")
set_session_alias("abc123", "taken")
assert set_session_alias("def456", "taken") is False
register_workstream("abc123")
register_workstream("def456")
set_workstream_alias("abc123", "taken")
assert set_workstream_alias("def456", "taken") is False
def test_alias_same_session_ok(self, tmp_db):
register_session("abc123")
set_session_alias("abc123", "mine")
assert set_session_alias("abc123", "mine") is True # no-op, same session
def test_alias_same_workstream_ok(self, tmp_db):
register_workstream("abc123")
set_workstream_alias("abc123", "mine")
assert set_workstream_alias("abc123", "mine") is True # no-op, same workstream
# ── Session resolution ────────────────────────────────────────────────
# ── Workstream resolution ─────────────────────────────────────────────
class TestResolveSession:
class TestResolveWorkstream:
def test_resolve_by_alias(self, tmp_db):
register_session("abc123")
set_session_alias("abc123", "my-alias")
assert resolve_session("my-alias") == "abc123"
register_workstream("abc123")
set_workstream_alias("abc123", "my-alias")
assert resolve_workstream("my-alias") == "abc123"
def test_resolve_by_exact_id(self, tmp_db):
register_session("abc123def456")
assert resolve_session("abc123def456") == "abc123def456"
register_workstream("abc123def456")
assert resolve_workstream("abc123def456") == "abc123def456"
def test_resolve_by_prefix(self, tmp_db):
register_session("abc123def456")
assert resolve_session("abc123") == "abc123def456"
register_workstream("abc123def456")
assert resolve_workstream("abc123") == "abc123def456"
def test_resolve_prefix_ambiguous(self, tmp_db):
register_session("abc123aaaaaa")
register_session("abc123bbbbbb")
register_workstream("abc123aaaaaa")
register_workstream("abc123bbbbbb")
# Ambiguous prefix should return None
assert resolve_session("abc123") is None
assert resolve_workstream("abc123") is None
def test_resolve_not_found(self, tmp_db):
assert resolve_session("nonexistent") is None
def test_resolve_legacy_session(self, tmp_db):
"""Sessions that exist only in conversations (pre-migration) should auto-register."""
save_message("legacy123456", "user", "old message")
result = resolve_session("legacy123456")
assert result == "legacy123456"
# Should now appear in sessions list
rows = list_sessions()
assert any(r[0] == "legacy123456" for r in rows)
assert resolve_workstream("nonexistent") is None
# ── List sessions ─────────────────────────────────────────────────────
# ── List workstreams with history ──────────────────────────────────────
class TestListSessions:
class TestListWorkstreamsWithHistory:
def test_empty(self, tmp_db):
assert list_sessions() == []
assert list_workstreams_with_history() == []
def test_ordered_by_updated(self, tmp_db):
register_session("first")
register_workstream("first")
save_message("first", "user", "hello")
register_session("second")
# Force an older timestamp so ordering is deterministic
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
conn.execute(
sa.text("UPDATE workstreams SET updated = '2020-01-01' WHERE ws_id = 'first'")
)
conn.commit()
register_workstream("second")
save_message("second", "user", "hello")
# second is more recent
rows = list_sessions()
rows = list_workstreams_with_history()
assert rows[0][0] == "second"
assert rows[1][0] == "first"
def test_includes_message_count(self, tmp_db):
register_session("sess1")
register_workstream("sess1")
save_message("sess1", "user", "hello")
save_message("sess1", "assistant", "hi")
rows = list_sessions()
rows = list_workstreams_with_history()
assert rows[0][5] == 2 # msg_count
def test_respects_limit(self, tmp_db):
for i in range(5):
register_session(f"sess{i}")
register_workstream(f"sess{i}")
save_message(f"sess{i}", "user", "hello")
rows = list_sessions(limit=3)
rows = list_workstreams_with_history(limit=3)
assert len(rows) == 3
# ── Load session messages ─────────────────────────────────────────────
# ── Load messages ─────────────────────────────────────────────────────
class TestLoadSessionMessages:
class TestLoadMessages:
def test_simple_user_assistant(self, tmp_db):
save_message("s1", "user", "hello")
save_message("s1", "assistant", "hi there")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 2
assert msgs[0] == {"role": "user", "content": "hello"}
assert msgs[1] == {"role": "assistant", "content": "hi there"}
@@ -159,7 +159,7 @@ class TestLoadSessionMessages:
save_message("s1", "assistant", "Let me check.")
save_message("s1", "tool_call", None, "bash", '{"command":"ls"}', tool_call_id="call_abc")
save_message("s1", "tool_result", "file1.txt\nfile2.txt", "bash", tool_call_id="call_abc")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 3 # user, assistant+tool_calls, tool
# Assistant should have content merged with tool_calls
assert msgs[1]["role"] == "assistant"
@@ -177,7 +177,7 @@ class TestLoadSessionMessages:
save_message("s1", "user", "do stuff")
save_message("s1", "tool_call", None, "bash", '{"command":"ls"}')
save_message("s1", "tool_result", "output", "bash")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 3
# Synthetic IDs should match
tc_id = msgs[1]["tool_calls"][0]["id"]
@@ -189,36 +189,36 @@ class TestLoadSessionMessages:
save_message("s1", "tool_call", None, "search", '{"query":"b"}', tool_call_id="call_2")
save_message("s1", "tool_result", "result a", "search", tool_call_id="call_1")
save_message("s1", "tool_result", "result b", "search", tool_call_id="call_2")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 4 # user, assistant+2 tool_calls, 2 tool results
assert len(msgs[1]["tool_calls"]) == 2
assert msgs[2]["tool_call_id"] == "call_1"
assert msgs[3]["tool_call_id"] == "call_2"
def test_empty_session(self, tmp_db):
assert load_session_messages("nonexistent") == []
def test_empty_workstream(self, tmp_db):
assert load_messages("nonexistent") == []
def test_orphaned_tool_result_skipped(self, tmp_db):
save_message("s1", "user", "hello")
save_message("s1", "tool_result", "orphan", "bash")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 1 # only the user message
# ── Delete session ────────────────────────────────────────────────────
# ── Delete workstream ─────────────────────────────────────────────────
class TestDeleteSession:
def test_delete_removes_session_and_messages(self, tmp_db):
register_session("abc123")
class TestDeleteWorkstream:
def test_delete_removes_workstream_and_messages(self, tmp_db):
register_workstream("abc123")
save_message("abc123", "user", "hello")
save_message("abc123", "assistant", "hi")
assert delete_session("abc123") is True
assert list_sessions() == []
assert load_session_messages("abc123") == []
assert delete_workstream("abc123") is True
assert list_workstreams_with_history() == []
assert load_messages("abc123") == []
def test_delete_nonexistent(self, tmp_db):
assert delete_session("nonexistent") is True # no-op, still returns True
assert delete_workstream("nonexistent") is False
# ── save_message with tool_call_id ────────────────────────────────────
@@ -230,7 +230,7 @@ class TestSaveMessageToolCallId:
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
row = conn.execute(
sa.text("SELECT tool_call_id FROM conversations WHERE session_id = 's1'")
sa.text("SELECT tool_call_id FROM conversations WHERE ws_id = 's1'")
).fetchone()
assert row[0] == "call_xyz"
@@ -239,20 +239,20 @@ class TestSaveMessageToolCallId:
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
row = conn.execute(
sa.text("SELECT tool_call_id FROM conversations WHERE session_id = 's1'")
sa.text("SELECT tool_call_id FROM conversations WHERE ws_id = 's1'")
).fetchone()
assert row[0] is None
# ── Sessions table creation ───────────────────────────────────────────
# ── Workstreams table creation ────────────────────────────────────────
class TestSessionsTable:
def test_sessions_table_exists(self, tmp_db):
class TestWorkstreamsTable:
def test_workstreams_table_exists(self, tmp_db):
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
rows = conn.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name='sessions'")
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name='workstreams'")
).fetchall()
assert len(rows) == 1
@@ -263,15 +263,15 @@ class TestSessionsTable:
conn.execute(sa.text("SELECT tool_call_id FROM conversations LIMIT 0"))
# ── ChatSession.resume_session ────────────────────────────────────────
# ── ChatSession.resume ────────────────────────────────────────────────
class TestResumeSession:
class TestResumeWorkstream:
def test_resume_loads_messages(self, tmp_db, mock_openai_client):
# Set up a session with messages in DB
register_session("old_sess_123")
save_message("old_sess_123", "user", "hello world")
save_message("old_sess_123", "assistant", "hi there")
# Set up a workstream with messages in DB
register_workstream("old_ws_123")
save_message("old_ws_123", "user", "hello world")
save_message("old_ws_123", "assistant", "hi there")
# Create a new session and resume
session = ChatSession(
@@ -283,12 +283,12 @@ class TestResumeSession:
max_tokens=1000,
tool_timeout=10,
)
original_id = session._session_id
assert original_id != "old_sess_123"
original_id = session._ws_id
assert original_id != "old_ws_123"
result = session.resume_session("old_sess_123")
result = session.resume("old_ws_123")
assert result is True
assert session._session_id == "old_sess_123"
assert session._ws_id == "old_ws_123"
assert len(session.messages) == 2
assert session.messages[0]["content"] == "hello world"
assert session._title_generated is True
@@ -303,9 +303,9 @@ class TestResumeSession:
max_tokens=1000,
tool_timeout=10,
)
assert session.resume_session("nonexistent") is False
assert session.resume("nonexistent") is False
def test_session_registered_on_init(self, tmp_db, mock_openai_client):
def test_workstream_not_registered_until_message(self, tmp_db, mock_openai_client):
session = ChatSession(
client=mock_openai_client,
model="test-model",
@@ -315,20 +315,19 @@ class TestResumeSession:
max_tokens=1000,
tool_timeout=10,
)
# Session is registered in DB (resolvable) even before any messages
assert resolve_session(session._session_id) == session._session_id
# But does not appear in list_sessions until a message is saved
assert not any(r[0] == session._session_id for r in list_sessions())
# Workstream is not auto-registered on init — only on /new or server creation
assert resolve_workstream(session._ws_id) is None
assert not any(r[0] == session._ws_id for r in list_workstreams_with_history())
# ── save_message updates sessions.updated ─────────────────────────────
# ── save_message updates workstreams.updated ──────────────────────────
class TestSaveMessageUpdatesSession:
class TestSaveMessageUpdatesWorkstream:
def test_updated_timestamp_bumped(self, tmp_db):
register_session("s1")
register_workstream("s1")
save_message("s1", "user", "first")
rows = list_sessions()
rows = list_workstreams_with_history()
_original_updated = rows[0][4]
import time
@@ -336,18 +335,18 @@ class TestSaveMessageUpdatesSession:
time.sleep(0.01) # ensure different timestamp
save_message("s1", "user", "hello")
rows = list_sessions()
rows = list_workstreams_with_history()
new_updated = rows[0][4]
# updated should be same or later (sqlite datetime resolution is seconds,
# so they may be equal in fast tests — just verify no error)
assert new_updated is not None
# ── Interrupted session repair ───────────────────────────────────────
# ── Interrupted workstream repair ─────────────────────────────────────
class TestInterruptedSessionRepair:
"""load_session_messages() should strip trailing incomplete tool call turns."""
class TestInterruptedWorkstreamRepair:
"""load_messages() should strip trailing incomplete tool call turns."""
def test_complete_tool_turn_preserved(self, tmp_db):
"""2 tool_calls + 2 tool_results = complete, no stripping."""
@@ -356,7 +355,7 @@ class TestInterruptedSessionRepair:
save_message("s1", "tool_call", None, "bash", '{"command":"pwd"}', "call_2")
save_message("s1", "tool_result", "file.txt", tool_call_id="call_1")
save_message("s1", "tool_result", "/home", tool_call_id="call_2")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 4 # user + assistant(2 calls) + 2 tool results
def test_partial_tool_results_stripped(self, tmp_db):
@@ -365,7 +364,7 @@ class TestInterruptedSessionRepair:
save_message("s1", "tool_call", None, "bash", '{"command":"ls"}', "call_1")
save_message("s1", "tool_call", None, "bash", '{"command":"pwd"}', "call_2")
save_message("s1", "tool_result", "file.txt", tool_call_id="call_1")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 1 # only user message remains
assert msgs[0]["role"] == "user"
@@ -375,7 +374,7 @@ class TestInterruptedSessionRepair:
save_message("s1", "assistant", "Let me check")
save_message("s1", "tool_call", None, "bash", '{"command":"ls"}', "call_1")
save_message("s1", "tool_call", None, "bash", '{"command":"pwd"}', "call_2")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
# assistant with content was merged into tool_call assistant, so stripped
assert len(msgs) == 1
assert msgs[0]["role"] == "user"
@@ -386,42 +385,42 @@ class TestInterruptedSessionRepair:
save_message("s1", "assistant", "response")
save_message("s1", "user", "second")
save_message("s1", "tool_call", None, "bash", '{"command":"ls"}', "call_1")
msgs = load_session_messages("s1")
msgs = load_messages("s1")
assert len(msgs) == 3 # user + assistant + user (incomplete turn stripped)
assert msgs[0]["role"] == "user"
assert msgs[1]["role"] == "assistant"
assert msgs[2]["role"] == "user"
# ── Session config persistence ───────────────────────────────────────
# ── Workstream config persistence ─────────────────────────────────────
class TestSessionConfig:
class TestWorkstreamConfig:
def test_save_load_roundtrip(self, tmp_db):
config = {"temperature": "0.3", "reasoning_effort": "high", "creative_mode": "False"}
save_session_config("s1", config)
loaded = load_session_config("s1")
save_workstream_config("s1", config)
loaded = load_workstream_config("s1")
assert loaded == config
def test_update_existing_key(self, tmp_db):
save_session_config("s1", {"temperature": "0.3"})
save_session_config("s1", {"temperature": "0.7"})
loaded = load_session_config("s1")
save_workstream_config("s1", {"temperature": "0.3"})
save_workstream_config("s1", {"temperature": "0.7"})
loaded = load_workstream_config("s1")
assert loaded["temperature"] == "0.7"
def test_missing_session_returns_empty(self, tmp_db):
loaded = load_session_config("nonexistent")
def test_missing_workstream_returns_empty(self, tmp_db):
loaded = load_workstream_config("nonexistent")
assert loaded == {}
def test_delete_session_removes_config(self, tmp_db):
register_session("s1")
def test_delete_workstream_removes_config(self, tmp_db):
register_workstream("s1")
save_message("s1", "user", "hi")
save_session_config("s1", {"temperature": "0.5"})
delete_session("s1")
assert load_session_config("s1") == {}
save_workstream_config("s1", {"temperature": "0.5"})
delete_workstream("s1")
assert load_workstream_config("s1") == {}
def test_resume_restores_config(self, tmp_db):
"""ChatSession.resume_session() should restore persisted config."""
"""ChatSession.resume() should restore persisted config."""
client = MagicMock()
client.models.list.return_value.data = [MagicMock(id="test-model")]
ui = MagicMock()
@@ -430,11 +429,11 @@ class TestSessionConfig:
ui.on_state_change = MagicMock()
ui.on_rename = MagicMock()
# Create a session with specific config
register_session("orig")
# Create a workstream with specific config
register_workstream("orig")
save_message("orig", "user", "hello")
save_message("orig", "assistant", "hi there")
save_session_config(
save_workstream_config(
"orig",
{
"temperature": "0.3",
@@ -456,7 +455,7 @@ class TestSessionConfig:
tool_timeout=30,
)
assert session.temperature == 0.7 # default
result = session.resume_session("orig")
result = session.resume("orig")
assert result is True
assert session.temperature == 0.3
assert session.reasoning_effort == "high"
@@ -465,84 +464,84 @@ class TestSessionConfig:
assert session.creative_mode is True
# ── Prune sessions ───────────────────────────────────────────────────
# ── Prune workstreams ─────────────────────────────────────────────────
class TestPruneSessions:
class TestPruneWorkstreams:
def test_orphan_removed(self, tmp_db):
"""Session registered with no messages should be pruned."""
register_session("orphan")
orphans, stale = prune_sessions()
"""Workstream registered with no messages should be pruned."""
register_workstream("orphan")
orphans, stale = prune_workstreams()
assert orphans == 1
assert list_sessions() == []
assert list_workstreams_with_history() == []
def test_session_with_messages_kept(self, tmp_db):
"""Session with messages should not be pruned."""
register_session("active")
def test_workstream_with_messages_kept(self, tmp_db):
"""Workstream with messages should not be pruned."""
register_workstream("active")
save_message("active", "user", "hello")
orphans, _stale = prune_sessions()
orphans, _stale = prune_workstreams()
assert orphans == 0
assert len(list_sessions()) == 1
assert len(list_workstreams_with_history()) == 1
def test_stale_unnamed_removed(self, tmp_db):
"""Old unnamed session should be pruned by retention policy."""
register_session("old1")
"""Old unnamed workstream should be pruned by retention policy."""
register_workstream("old1")
save_message("old1", "user", "ancient message")
# Force the updated timestamp to the past so it looks stale
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
conn.execute(
sa.text("UPDATE sessions SET updated = '2020-01-01' WHERE session_id = 'old1'")
sa.text("UPDATE workstreams SET updated = '2020-01-01' WHERE ws_id = 'old1'")
)
conn.commit()
_orphans, stale = prune_sessions(retention_days=30)
_orphans, stale = prune_workstreams(retention_days=30)
assert stale == 1
def test_named_session_preserved(self, tmp_db):
"""Session with alias should be kept regardless of age."""
register_session("old2")
set_session_alias("old2", "important")
def test_named_workstream_preserved(self, tmp_db):
"""Workstream with alias should be kept regardless of age."""
register_workstream("old2")
set_workstream_alias("old2", "important")
save_message("old2", "user", "old but named")
# Force old timestamp
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
conn.execute(
sa.text("UPDATE sessions SET updated = '2020-01-01' WHERE session_id = 'old2'")
sa.text("UPDATE workstreams SET updated = '2020-01-01' WHERE ws_id = 'old2'")
)
conn.commit()
_orphans, stale = prune_sessions(retention_days=30)
_orphans, stale = prune_workstreams(retention_days=30)
assert stale == 0
assert len(list_sessions()) == 1
assert len(list_workstreams_with_history()) == 1
def test_fresh_unnamed_preserved(self, tmp_db):
"""Recent unnamed session should not be pruned."""
register_session("fresh")
"""Recent unnamed workstream should not be pruned."""
register_workstream("fresh")
save_message("fresh", "user", "just now")
_orphans, stale = prune_sessions(retention_days=30)
_orphans, stale = prune_workstreams(retention_days=30)
assert stale == 0
assert len(list_sessions()) == 1
assert len(list_workstreams_with_history()) == 1
def test_prune_removes_session_config(self, tmp_db):
"""Pruning orphan/stale sessions should also remove their config rows."""
register_session("orphan_cfg")
save_session_config("orphan_cfg", {"temperature": "0.5"})
def test_prune_removes_workstream_config(self, tmp_db):
"""Pruning orphan/stale workstreams should also remove their config rows."""
register_workstream("orphan_cfg")
save_workstream_config("orphan_cfg", {"temperature": "0.5"})
register_session("stale_cfg")
register_workstream("stale_cfg")
save_message("stale_cfg", "user", "old")
save_session_config("stale_cfg", {"temperature": "0.9"})
save_workstream_config("stale_cfg", {"temperature": "0.9"})
engine = get_storage()._engine # noqa: SLF001
with engine.connect() as conn:
conn.execute(
sa.text("UPDATE sessions SET updated = '2020-01-01' WHERE session_id = 'stale_cfg'")
sa.text("UPDATE workstreams SET updated = '2020-01-01' WHERE ws_id = 'stale_cfg'")
)
conn.commit()
# Both should have config before prune
assert load_session_config("orphan_cfg") == {"temperature": "0.5"}
assert load_session_config("stale_cfg") == {"temperature": "0.9"}
assert load_workstream_config("orphan_cfg") == {"temperature": "0.5"}
assert load_workstream_config("stale_cfg") == {"temperature": "0.9"}
prune_sessions(retention_days=30)
prune_workstreams(retention_days=30)
# Config rows should be cleaned up
assert load_session_config("orphan_cfg") == {}
assert load_session_config("stale_cfg") == {}
assert load_workstream_config("orphan_cfg") == {}
assert load_workstream_config("stale_cfg") == {}
+76 -77
View File
@@ -14,28 +14,28 @@ def backend(tmp_path):
reset_storage()
# -- Session operations --------------------------------------------------------
# -- Workstream registration ---------------------------------------------------
class TestRegisterSession:
def test_register_creates_session(self, backend):
backend.register_session("s1", title="Test")
name = backend.get_session_name("s1")
class TestRegisterWorkstream:
def test_register_creates_workstream(self, backend):
backend.register_workstream("s1", title="Test")
name = backend.get_workstream_display_name("s1")
assert name == "Test"
def test_register_idempotent(self, backend):
backend.register_session("s1", title="First")
backend.register_session("s1", title="Second")
name = backend.get_session_name("s1")
backend.register_workstream("s1", title="First")
backend.register_workstream("s1", title="Second")
name = backend.get_workstream_display_name("s1")
assert name == "First" # INSERT OR IGNORE preserves first
class TestSaveAndLoadMessages:
def test_roundtrip(self, backend):
backend.register_session("s1")
backend.register_workstream("s1")
backend.save_message("s1", "user", "hello")
backend.save_message("s1", "assistant", "world")
msgs = backend.load_session_messages("s1")
msgs = backend.load_messages("s1")
assert len(msgs) == 2
assert msgs[0]["role"] == "user"
assert msgs[0]["content"] == "hello"
@@ -43,12 +43,12 @@ class TestSaveAndLoadMessages:
assert msgs[1]["content"] == "world"
def test_tool_call_grouping(self, backend):
backend.register_session("s1")
backend.register_workstream("s1")
backend.save_message("s1", "user", "do something")
backend.save_message("s1", "tool_call", None, "bash", '{"cmd":"ls"}', tool_call_id="c1")
backend.save_message("s1", "tool_result", "file.txt", tool_call_id="c1")
backend.save_message("s1", "assistant", "done")
msgs = backend.load_session_messages("s1")
msgs = backend.load_messages("s1")
assert len(msgs) == 4
assert msgs[1]["role"] == "assistant"
assert len(msgs[1]["tool_calls"]) == 1
@@ -57,136 +57,136 @@ class TestSaveAndLoadMessages:
assert msgs[2]["content"] == "file.txt"
def test_incomplete_turn_repair(self, backend):
backend.register_session("s1")
backend.register_workstream("s1")
backend.save_message("s1", "user", "do something")
backend.save_message("s1", "tool_call", None, "bash", '{"cmd":"ls"}', tool_call_id="c1")
backend.save_message("s1", "tool_call", None, "read", '{"path":"a"}', tool_call_id="c2")
# Only 1 result for 2 calls — incomplete turn
backend.save_message("s1", "tool_result", "ok", tool_call_id="c1")
msgs = backend.load_session_messages("s1")
msgs = backend.load_messages("s1")
# Incomplete turn should be stripped
assert len(msgs) == 1 # only the user message remains
def test_provider_data_preserved(self, backend):
import json
backend.register_session("s1")
backend.register_workstream("s1")
pd = json.dumps({"encrypted": True})
backend.save_message("s1", "assistant", "hi", provider_data=pd)
msgs = backend.load_session_messages("s1")
msgs = backend.load_messages("s1")
assert msgs[0].get("_provider_content") == {"encrypted": True}
def test_empty_session_returns_empty(self, backend):
assert backend.load_session_messages("nonexistent") == []
def test_empty_workstream_returns_empty(self, backend):
assert backend.load_messages("nonexistent") == []
class TestListSessions:
def test_lists_sessions_with_messages(self, backend):
backend.register_session("s1")
class TestListWorkstreamsWithHistory:
def test_lists_workstreams_with_messages(self, backend):
backend.register_workstream("s1")
backend.save_message("s1", "user", "hi")
backend.register_session("s2") # no messages
rows = backend.list_sessions()
backend.register_workstream("s2") # no messages
rows = backend.list_workstreams_with_history()
assert len(rows) == 1
assert rows[0][0] == "s1"
def test_respects_limit(self, backend):
for i in range(5):
sid = f"s{i}"
backend.register_session(sid)
backend.register_workstream(sid)
backend.save_message(sid, "user", f"msg {i}")
rows = backend.list_sessions(limit=3)
rows = backend.list_workstreams_with_history(limit=3)
assert len(rows) == 3
class TestDeleteSession:
class TestDeleteWorkstream:
def test_deletes_all_data(self, backend):
backend.register_session("s1")
backend.register_workstream("s1")
backend.save_message("s1", "user", "hi")
backend.save_session_config("s1", {"temp": "0.5"})
assert backend.delete_session("s1")
assert backend.load_session_messages("s1") == []
assert backend.load_session_config("s1") == {}
assert backend.get_session_name("s1") is None
backend.save_workstream_config("s1", {"temp": "0.5"})
assert backend.delete_workstream("s1")
assert backend.load_messages("s1") == []
assert backend.load_workstream_config("s1") == {}
assert backend.get_workstream_display_name("s1") is None
class TestPruneSessions:
class TestPruneWorkstreams:
def test_orphan_removed(self, backend):
backend.register_session("orphan")
orphans, stale = backend.prune_sessions()
backend.register_workstream("orphan")
orphans, stale = backend.prune_workstreams()
assert orphans == 1
def test_stale_removed(self, backend):
import sqlalchemy as sa
backend.register_session("old")
backend.register_workstream("old")
backend.save_message("old", "user", "hi")
# Force old timestamp
with backend._engine.connect() as conn:
conn.execute(
sa.text("UPDATE sessions SET updated = '2020-01-01' WHERE session_id = 'old'")
sa.text("UPDATE workstreams SET updated = '2020-01-01' WHERE ws_id = 'old'")
)
conn.commit()
_, stale = backend.prune_sessions(retention_days=30)
_, stale = backend.prune_workstreams(retention_days=30)
assert stale == 1
class TestResolveSession:
class TestResolveWorkstream:
def test_exact_alias(self, backend):
backend.register_session("s1")
backend.set_session_alias("s1", "myalias")
assert backend.resolve_session("myalias") == "s1"
backend.register_workstream("s1")
backend.set_workstream_alias("s1", "myalias")
assert backend.resolve_workstream("myalias") == "s1"
def test_exact_id(self, backend):
backend.register_session("abc-123-def")
assert backend.resolve_session("abc-123-def") == "abc-123-def"
backend.register_workstream("abc-123-def")
assert backend.resolve_workstream("abc-123-def") == "abc-123-def"
def test_prefix_match(self, backend):
backend.register_session("abc-123-def")
assert backend.resolve_session("abc") == "abc-123-def"
backend.register_workstream("abc-123-def")
assert backend.resolve_workstream("abc") == "abc-123-def"
def test_not_found(self, backend):
assert backend.resolve_session("nonexistent") is None
assert backend.resolve_workstream("nonexistent") is None
# -- Session config ------------------------------------------------------------
# -- Workstream config ---------------------------------------------------------
class TestSessionConfig:
class TestWorkstreamConfig:
def test_roundtrip(self, backend):
backend.register_session("s1")
backend.save_session_config("s1", {"temperature": "0.7", "effort": "high"})
cfg = backend.load_session_config("s1")
backend.register_workstream("s1")
backend.save_workstream_config("s1", {"temperature": "0.7", "effort": "high"})
cfg = backend.load_workstream_config("s1")
assert cfg == {"temperature": "0.7", "effort": "high"}
def test_empty_config(self, backend):
assert backend.load_session_config("nonexistent") == {}
assert backend.load_workstream_config("nonexistent") == {}
# -- Session metadata ----------------------------------------------------------
# -- Workstream metadata ------------------------------------------------------
class TestSessionMetadata:
class TestWorkstreamMetadata:
def test_alias(self, backend):
backend.register_session("s1")
assert backend.set_session_alias("s1", "my-session")
assert backend.get_session_name("s1") == "my-session"
backend.register_workstream("s1")
assert backend.set_workstream_alias("s1", "my-session")
assert backend.get_workstream_display_name("s1") == "my-session"
def test_alias_conflict(self, backend):
backend.register_session("s1")
backend.register_session("s2")
backend.set_session_alias("s1", "taken")
assert not backend.set_session_alias("s2", "taken")
backend.register_workstream("s1")
backend.register_workstream("s2")
backend.set_workstream_alias("s1", "taken")
assert not backend.set_workstream_alias("s2", "taken")
def test_title(self, backend):
backend.register_session("s1")
backend.update_session_title("s1", "My Title")
assert backend.get_session_name("s1") == "My Title"
backend.register_workstream("s1")
backend.update_workstream_title("s1", "My Title")
assert backend.get_workstream_display_name("s1") == "My Title"
def test_alias_preferred_over_title(self, backend):
backend.register_session("s1")
backend.update_session_title("s1", "Title")
backend.set_session_alias("s1", "Alias")
assert backend.get_session_name("s1") == "Alias"
backend.register_workstream("s1")
backend.update_workstream_title("s1", "Title")
backend.set_workstream_alias("s1", "Alias")
assert backend.get_workstream_display_name("s1") == "Alias"
# -- Key-value store -----------------------------------------------------------
@@ -234,7 +234,7 @@ class TestKVStore:
class TestSearch:
def test_search_history(self, backend):
backend.register_session("s1")
backend.register_workstream("s1")
backend.save_message("s1", "user", "hello world")
backend.save_message("s1", "user", "goodbye world")
results = backend.search_history("hello")
@@ -242,7 +242,7 @@ class TestSearch:
assert any("hello" in str(r[3]) for r in results)
def test_search_recent(self, backend):
backend.register_session("s1")
backend.register_workstream("s1")
backend.save_message("s1", "user", "msg1")
backend.save_message("s1", "user", "msg2")
results = backend.search_history_recent(limit=1)
@@ -293,15 +293,14 @@ class TestWorkstreams:
assert len(rows) == 1
assert rows[0][0] == "ws1"
def test_session_with_ws_id(self, backend):
def test_workstream_with_messages_in_history(self, backend):
backend.register_workstream("ws1", node_id="node-a")
backend.register_session("s1", node_id="node-a", ws_id="ws1")
backend.save_message("s1", "user", "hello")
rows = backend.list_sessions()
backend.save_message("ws1", "user", "hello")
rows = backend.list_workstreams_with_history()
assert len(rows) == 1
# Columns: sid, alias, title, created, updated, count, node_id, ws_id
# Columns: ws_id, alias, title, created, updated, count, node_id
assert rows[0][0] == "ws1"
assert rows[0][6] == "node-a"
assert rows[0][7] == "ws1"
# -- Lifecycle -----------------------------------------------------------------
+254
View File
@@ -0,0 +1,254 @@
"""Tests for turnstone.core.tool_search — BM25 index and tool search manager."""
from __future__ import annotations
import pytest
from turnstone.core.tool_search import (
BM25Index,
ToolSearchManager,
_mcp_server_summary,
_tokenize,
_tool_name,
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_tool(name: str, description: str = "") -> dict:
"""Create a minimal OpenAI-format tool dict for testing."""
return {
"type": "function",
"function": {
"name": name,
"description": description or f"Tool {name}",
"parameters": {"type": "object", "properties": {}},
},
}
# ---------------------------------------------------------------------------
# BM25Index tests
# ---------------------------------------------------------------------------
class TestTokenize:
def test_basic_split(self):
assert _tokenize("hello world") == ["hello", "world"]
def test_underscore_split(self):
assert _tokenize("create_issue") == ["create", "issue"]
def test_mixed_delimiters(self):
assert _tokenize("mcp__github__create-issue") == ["mcp", "github", "create", "issue"]
def test_empty_string(self):
assert _tokenize("") == []
def test_lowercased(self):
assert _tokenize("GitHub Create") == ["github", "create"]
class TestBM25Index:
def test_empty_corpus(self):
idx = BM25Index([])
assert idx.search("test") == []
def test_empty_query(self):
idx = BM25Index(["hello world", "foo bar"])
assert idx.search("") == []
def test_single_document(self):
idx = BM25Index(["create github issue"])
assert idx.search("github") == [0]
def test_ranking_order(self):
docs = [
"list_repos List all repositories",
"create_issue Create a new GitHub issue",
"get_issue Get details of a GitHub issue",
]
idx = BM25Index(docs)
results = idx.search("github issue")
# Both issue-related docs should rank above list_repos
assert 1 in results[:2]
assert 2 in results[:2]
def test_top_k_limit(self):
docs = [f"tool_{i} description {i}" for i in range(20)]
idx = BM25Index(docs)
results = idx.search("tool description", k=3)
assert len(results) <= 3
def test_no_match(self):
idx = BM25Index(["alpha beta gamma"])
assert idx.search("zzzzz") == []
def test_exact_name_match_ranks_high(self):
docs = [
"send_email Send an email message",
"send_slack Send a Slack message",
"read_email Read email inbox",
]
idx = BM25Index(docs)
results = idx.search("send email")
assert results[0] == 0 # send_email should rank first
# ---------------------------------------------------------------------------
# ToolSearchManager tests
# ---------------------------------------------------------------------------
class TestToolSearchManager:
@pytest.fixture()
def builtin_tools(self):
return [
_make_tool("bash", "Execute shell commands"),
_make_tool("read_file", "Read a file"),
_make_tool("edit_file", "Edit a file"),
]
@pytest.fixture()
def mcp_tools(self):
return [
_make_tool("mcp__github__create_issue", "Create a new GitHub issue"),
_make_tool("mcp__github__list_issues", "List GitHub issues"),
_make_tool("mcp__github__get_repo", "Get repository details"),
_make_tool("mcp__slack__send_message", "Send a Slack message"),
_make_tool("mcp__slack__list_channels", "List Slack channels"),
_make_tool("mcp__jira__create_ticket", "Create a Jira ticket"),
]
@pytest.fixture()
def manager(self, builtin_tools, mcp_tools):
all_tools = builtin_tools + mcp_tools
return ToolSearchManager(
all_tools,
always_on_names={"bash", "read_file", "edit_file"},
threshold=5,
max_results=3,
)
def test_should_activate_above_threshold(self, manager):
assert manager.should_activate()
def test_should_not_activate_below_threshold(self, builtin_tools):
mgr = ToolSearchManager(builtin_tools, always_on_names={"bash", "read_file", "edit_file"})
assert not mgr.should_activate()
def test_visible_tools_initially_builtin_only(self, manager):
visible = manager.get_visible_tools()
names = {_tool_name(t) for t in visible}
assert names == {"bash", "read_file", "edit_file"}
def test_deferred_tools_excludes_builtin(self, manager):
deferred = manager.get_deferred_tools()
names = {_tool_name(t) for t in deferred}
assert "bash" not in names
assert "mcp__github__create_issue" in names
def test_search_returns_relevant_tools(self, manager):
results = manager.search("github issue")
names = {_tool_name(t) for t in results}
assert "mcp__github__create_issue" in names or "mcp__github__list_issues" in names
def test_search_respects_max_results(self, manager):
results = manager.search("tool")
assert len(results) <= 3
def test_search_excludes_already_expanded(self, manager):
# Expand a github tool, then search for github — expanded tool should not appear
manager.expand_visible(["mcp__github__create_issue"])
results = manager.search("github issue")
names = {_tool_name(t) for t in results}
assert "mcp__github__create_issue" not in names
def test_expand_visible_adds_tools(self, manager):
manager.expand_visible(["mcp__github__create_issue"])
visible = manager.get_visible_tools()
names = {_tool_name(t) for t in visible}
assert "mcp__github__create_issue" in names
def test_expand_visible_returns_newly_added(self, manager):
added = manager.expand_visible(["mcp__github__create_issue", "mcp__slack__send_message"])
assert len(added) == 2
names = {_tool_name(t) for t in added}
assert names == {"mcp__github__create_issue", "mcp__slack__send_message"}
def test_expand_visible_idempotent(self, manager):
manager.expand_visible(["mcp__github__create_issue"])
added = manager.expand_visible(["mcp__github__create_issue"])
assert added == []
def test_expand_visible_ignores_unknown(self, manager):
added = manager.expand_visible(["nonexistent_tool"])
assert added == []
def test_get_expanded_names_empty(self, manager):
assert manager.get_expanded_names() == []
def test_get_expanded_names_after_expand(self, manager):
manager.expand_visible(["mcp__github__create_issue", "mcp__slack__send_message"])
names = manager.get_expanded_names()
assert names == ["mcp__github__create_issue", "mcp__slack__send_message"]
def test_deferred_excludes_expanded(self, manager):
manager.expand_visible(["mcp__github__create_issue"])
deferred = manager.get_deferred_tools()
names = {_tool_name(t) for t in deferred}
assert "mcp__github__create_issue" not in names
def test_get_all_tools_returns_everything(self, manager, builtin_tools, mcp_tools):
assert len(manager.get_all_tools()) == len(builtin_tools) + len(mcp_tools)
def test_search_tool_definition_format(self, manager):
defn = manager.get_search_tool_definition()
assert defn["type"] == "function"
fn = defn["function"]
assert fn["name"] == "tool_search"
assert "query" in fn["parameters"]["properties"]
assert "query" in fn["parameters"]["required"]
def test_search_tool_description_has_server_hint(self, manager):
defn = manager.get_search_tool_definition()
desc = defn["function"]["description"]
assert "github" in desc
assert "slack" in desc
assert "jira" in desc
def test_format_search_results_empty(self, manager):
text = manager.format_search_results([])
assert "No matching tools found" in text
def test_format_search_results_with_tools(self, manager, mcp_tools):
text = manager.format_search_results(mcp_tools[:2])
assert "Found 2" in text
assert "mcp__github__create_issue" in text
# ---------------------------------------------------------------------------
# Helper function tests
# ---------------------------------------------------------------------------
class TestMCPServerSummary:
def test_groups_by_server(self):
tools = [
_make_tool("mcp__github__a"),
_make_tool("mcp__github__b"),
_make_tool("mcp__slack__c"),
]
summary = _mcp_server_summary(tools)
assert "github (2 tools)" in summary
assert "slack (1 tool)" in summary
def test_non_mcp_tools_counted_as_other(self):
tools = [_make_tool("custom_tool")]
summary = _mcp_server_summary(tools)
assert "other (1 tool)" in summary
def test_empty_list(self):
assert _mcp_server_summary([]) == ""
+5 -4
View File
@@ -72,16 +72,16 @@ class TestToolsMetadata:
"""Validate the metadata extracted from JSON files."""
def test_tool_count(self):
assert len(TOOLS) == 14
assert len(TOOLS) == 15
def test_agent_tools_count(self):
assert len(AGENT_TOOLS) == 6
assert len(AGENT_TOOLS) == 7
def test_task_agent_tools_count(self):
assert len(TASK_AGENT_TOOLS) == 9
assert len(TASK_AGENT_TOOLS) == 10
def test_auto_approve_sets_match(self):
expected = {"read_file", "search", "math", "man", "web_fetch", "web_search"}
expected = {"read_file", "search", "math", "man", "web_fetch", "web_search", "notify"}
assert expected == AGENT_AUTO_TOOLS
assert expected == TASK_AUTO_TOOLS
@@ -101,6 +101,7 @@ class TestToolsMetadata:
"remember": "key",
"recall": "query",
"forget": "key",
"notify": "message",
}
assert expected == PRIMARY_KEY_MAP
+5 -19
View File
@@ -127,21 +127,7 @@ class TestApiTokenCRUD:
assert "expires" not in tok
class TestSessionWorkstreamUserId:
def test_register_session_with_user_id(self, db):
db.register_session("s1", user_id="u1")
# Verify via raw SQL that user_id is stored
import sqlalchemy as sa
from turnstone.core.storage._schema import sessions
with db._engine.connect() as conn:
row = conn.execute(
sa.select(sessions.c.user_id).where(sessions.c.session_id == "s1")
).fetchone()
assert row is not None
assert row[0] == "u1"
class TestWorkstreamUserId:
def test_register_workstream_with_user_id(self, db):
db.register_workstream("ws1", user_id="u1")
import sqlalchemy as sa
@@ -155,15 +141,15 @@ class TestSessionWorkstreamUserId:
assert row is not None
assert row[0] == "u1"
def test_register_session_without_user_id(self, db):
db.register_session("s1")
def test_register_workstream_without_user_id(self, db):
db.register_workstream("ws1")
import sqlalchemy as sa
from turnstone.core.storage._schema import sessions
from turnstone.core.storage._schema import workstreams
with db._engine.connect() as conn:
row = conn.execute(
sa.select(sessions.c.user_id).where(sessions.c.session_id == "s1")
sa.select(workstreams.c.user_id).where(workstreams.c.ws_id == "ws1")
).fetchone()
assert row is not None
assert row[0] is None
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "0.4.0"
__version__ = "0.5.0"
+65
View File
@@ -23,13 +23,18 @@ from turnstone.api.schemas import (
AuthSetupRequest,
AuthSetupResponse,
AuthStatusResponse,
CreateScheduleRequest,
CreateTokenRequest,
CreateTokenResponse,
CreateUserRequest,
ErrorResponse,
ListScheduleRunsResponse,
ListSchedulesResponse,
ListTokensResponse,
ListUsersResponse,
ScheduleInfo,
StatusResponse,
UpdateScheduleRequest,
UserInfo,
)
@@ -182,6 +187,61 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
error_codes=[404],
tags=["Admin"],
),
# --- Schedules ---
EndpointSpec(
"/v1/api/admin/schedules",
"GET",
"List all scheduled tasks",
response_model=ListSchedulesResponse,
tags=["Schedules"],
),
EndpointSpec(
"/v1/api/admin/schedules",
"POST",
"Create a scheduled task",
request_model=CreateScheduleRequest,
response_model=ScheduleInfo,
error_codes=[400],
tags=["Schedules"],
),
EndpointSpec(
"/v1/api/admin/schedules/{task_id}",
"GET",
"Get a scheduled task",
response_model=ScheduleInfo,
error_codes=[404],
tags=["Schedules"],
),
EndpointSpec(
"/v1/api/admin/schedules/{task_id}",
"PUT",
"Update a scheduled task",
request_model=UpdateScheduleRequest,
response_model=ScheduleInfo,
error_codes=[400, 404],
tags=["Schedules"],
),
EndpointSpec(
"/v1/api/admin/schedules/{task_id}",
"DELETE",
"Delete a scheduled task",
response_model=StatusResponse,
error_codes=[404],
tags=["Schedules"],
),
EndpointSpec(
"/v1/api/admin/schedules/{task_id}/runs",
"GET",
"List run history for a scheduled task",
response_model=ListScheduleRunsResponse,
query_params=[
QueryParam(
"limit", "Max results (default 50, max 200)", schema_type="integer", default=50
),
],
error_codes=[404],
tags=["Schedules"],
),
# --- Observability ---
EndpointSpec(
"/health",
@@ -213,6 +273,11 @@ _ALL_MODELS: list[type[BaseModel]] = [
ConsoleCreateWsRequest,
ConsoleCreateWsResponse,
ConsoleHealthResponse,
CreateScheduleRequest,
UpdateScheduleRequest,
ScheduleInfo,
ListSchedulesResponse,
ListScheduleRunsResponse,
]
+84
View File
@@ -155,3 +155,87 @@ class AuthStatusResponse(BaseModel):
auth_enabled: bool
has_users: bool
setup_required: bool
# ---------------------------------------------------------------------------
# Schedules
# ---------------------------------------------------------------------------
class CreateScheduleRequest(BaseModel):
"""POST /v1/api/admin/schedules request body."""
name: str = Field(description="Human-readable schedule name")
description: str = Field(default="", description="Optional description")
schedule_type: str = Field(description="'cron' or 'at'")
cron_expr: str = Field(default="", description="Cron expression (when schedule_type='cron')")
at_time: str = Field(default="", description="ISO8601 timestamp (when schedule_type='at')")
target_mode: str = Field(default="auto", description="auto, pool, all, or specific node_id")
model: str = Field(default="", description="Model alias for the workstream")
initial_message: str = Field(description="Message sent to the new workstream")
auto_approve: bool = Field(default=False)
auto_approve_tools: list[str] = Field(default_factory=list)
enabled: bool = Field(default=True)
class UpdateScheduleRequest(BaseModel):
"""PUT /v1/api/admin/schedules/{task_id} request body (partial update)."""
name: str | None = None
description: str | None = None
schedule_type: str | None = None
cron_expr: str | None = None
at_time: str | None = None
target_mode: str | None = None
model: str | None = None
initial_message: str | None = None
auto_approve: bool | None = None
auto_approve_tools: list[str] | None = None
enabled: bool | None = None
class ScheduleInfo(BaseModel):
"""Scheduled task details."""
task_id: str
name: str
description: str = ""
schedule_type: str
cron_expr: str = ""
at_time: str = ""
target_mode: str = "auto"
model: str = ""
initial_message: str
auto_approve: bool = False
auto_approve_tools: list[str] = Field(default_factory=list)
enabled: bool = True
created_by: str = ""
last_run: str | None = None
next_run: str | None = None
created: str = ""
updated: str = ""
class ListSchedulesResponse(BaseModel):
"""GET /v1/api/admin/schedules response."""
schedules: list[ScheduleInfo]
class ScheduleRunInfo(BaseModel):
"""Single execution record for a scheduled task."""
run_id: str
task_id: str
node_id: str = ""
ws_id: str = ""
correlation_id: str = ""
started: str
status: str = "dispatched"
error: str = ""
class ListScheduleRunsResponse(BaseModel):
"""GET /v1/api/admin/schedules/{task_id}/runs response."""
runs: list[ScheduleRunInfo]
+11 -12
View File
@@ -39,18 +39,19 @@ class CreateWorkstreamRequest(BaseModel):
name: str = Field(default="", description="Workstream display name (auto-generated if empty)")
model: str = Field(default="", description="Model alias from registry")
auto_approve: bool = Field(default=False, description="Auto-approve all tool calls")
resume_session: str = Field(
resume_ws: str = Field(
default="",
description="Session ID to resume atomically during creation (empty = fresh start)",
description="Workstream ID to resume atomically during creation (empty = fresh start)",
)
class CreateWorkstreamResponse(BaseModel):
ws_id: str = Field(description="Unique ID of the new workstream")
name: str = Field(description="Assigned workstream name")
resumed: bool = Field(default=False, description="Whether a previous session was resumed")
session_id: str = Field(default="", description="Resolved session ID (set when resumed)")
message_count: int = Field(default=0, description="Number of messages in the resumed session")
resumed: bool = Field(default=False, description="Whether a previous workstream was resumed")
message_count: int = Field(
default=0, description="Number of messages in the resumed workstream"
)
class CloseWorkstreamRequest(BaseModel):
@@ -66,7 +67,6 @@ class WorkstreamInfo(BaseModel):
id: str
name: str
state: str
session_id: str | None = None
class ListWorkstreamsResponse(BaseModel):
@@ -77,7 +77,6 @@ class DashboardWorkstream(BaseModel):
id: str
name: str
state: str
session_id: str | None = None
title: str = ""
tokens: int = 0
context_ratio: float = 0.0
@@ -104,12 +103,12 @@ class DashboardResponse(BaseModel):
# ---------------------------------------------------------------------------
# Sessions
# Saved workstreams
# ---------------------------------------------------------------------------
class SessionInfo(BaseModel):
session_id: str
class SavedWorkstreamInfo(BaseModel):
ws_id: str
alias: str | None = None
title: str | None = None
created: str
@@ -117,8 +116,8 @@ class SessionInfo(BaseModel):
message_count: int
class ListSessionsResponse(BaseModel):
sessions: list[SessionInfo]
class ListSavedWorkstreamsResponse(BaseModel):
workstreams: list[SavedWorkstreamInfo]
# ---------------------------------------------------------------------------
+7 -7
View File
@@ -25,7 +25,7 @@ from turnstone.api.server_schemas import (
CreateWorkstreamResponse,
DashboardResponse,
HealthResponse,
ListSessionsResponse,
ListSavedWorkstreamsResponse,
ListWorkstreamsResponse,
PlanFeedbackRequest,
SendRequest,
@@ -122,13 +122,13 @@ SERVER_ENDPOINTS: list[EndpointSpec] = [
"across all workstreams. Returns text/event-stream.",
tags=["Streaming"],
),
# --- Sessions ---
# --- Saved workstreams ---
EndpointSpec(
"/v1/api/sessions",
"/v1/api/workstreams/saved",
"GET",
"List saved sessions",
response_model=ListSessionsResponse,
tags=["Sessions"],
"List saved workstreams",
response_model=ListSavedWorkstreamsResponse,
tags=["Workstreams"],
),
# --- Auth ---
EndpointSpec(
@@ -191,7 +191,7 @@ _ALL_MODELS: list[type[BaseModel]] = [
CloseWorkstreamRequest,
ListWorkstreamsResponse,
DashboardResponse,
ListSessionsResponse,
ListSavedWorkstreamsResponse,
HealthResponse,
]
+195
View File
@@ -0,0 +1,195 @@
"""Lightweight HTTP server for the channel gateway.
Runs alongside the channel adapters (Discord, etc.) to receive notification
requests from the bridge. Exposes ``POST /v1/api/notify`` and ``GET /health``.
"""
from __future__ import annotations
import asyncio
import json
import socket
import uuid
from typing import TYPE_CHECKING, Any
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Mount, Route
from turnstone.core.log import get_logger
if TYPE_CHECKING:
from starlette.requests import Request
from turnstone.channels._protocol import ChannelAdapter
from turnstone.core.storage._protocol import StorageBackend
log = get_logger(__name__)
async def _handle_health(request: Request) -> JSONResponse:
return JSONResponse({"status": "ok", "service": "channel"})
def _check_auth(request: Request) -> JSONResponse | None:
"""Validate the request's Authorization header. Returns an error response or None."""
auth_token: str = getattr(request.app.state, "auth_token", "")
jwt_secret: str = getattr(request.app.state, "jwt_secret", "")
if not auth_token and not jwt_secret:
log.warning("notify.auth_not_configured")
return JSONResponse({"error": "authentication not configured"}, status_code=401)
header = request.headers.get("Authorization", "")
if not header.startswith("Bearer "):
return JSONResponse({"error": "Unauthorized"}, status_code=401)
token = header[7:]
# Static token check
if auth_token:
import hmac
if hmac.compare_digest(token, auth_token):
return None
# JWT check
if jwt_secret and "." in token:
from turnstone.core.auth import JWT_AUD_CHANNEL, validate_jwt
result = validate_jwt(token, jwt_secret, audience=JWT_AUD_CHANNEL)
if result is not None:
return None
return JSONResponse({"error": "Unauthorized"}, status_code=401)
async def _handle_notify(request: Request) -> JSONResponse:
"""Deliver a notification to one or more channel adapters."""
auth_err = _check_auth(request)
if auth_err is not None:
return auth_err
adapters: dict[str, ChannelAdapter] = request.app.state.adapters
storage: StorageBackend = request.app.state.storage
try:
body: dict[str, Any] = await request.json()
except (json.JSONDecodeError, ValueError):
return JSONResponse({"error": "invalid JSON"}, status_code=400)
target = body.get("target")
message = body.get("message", "").strip() if isinstance(body.get("message"), str) else ""
title = body.get("title", "").strip() if isinstance(body.get("title"), str) else ""
if not target or not message:
return JSONResponse({"error": "target and message are required"}, status_code=400)
content = f"**{title}**\n{message}" if title else message
# Resolve targets
targets: list[tuple[str, str]] = []
if "username" in target:
user = await asyncio.to_thread(storage.get_user_by_username, target["username"])
if user is None:
log.warning("notify.user_not_found", username=target["username"])
return JSONResponse(
{"error": "target not found or has no linked channels"},
status_code=404,
)
links = await asyncio.to_thread(storage.list_channel_users_by_user, user["user_id"])
for link in links:
targets.append((link["channel_type"], link["channel_user_id"]))
if not targets:
log.warning("notify.user_no_linked_channels", username=target["username"])
return JSONResponse(
{"error": "target not found or has no linked channels"},
status_code=404,
)
elif "channel_type" in target and "channel_id" in target:
targets.append((target["channel_type"], target["channel_id"]))
else:
return JSONResponse(
{"error": "target must have username or channel_type+channel_id"},
status_code=400,
)
results: list[dict[str, str]] = []
for channel_type, channel_id in targets:
adapter = adapters.get(channel_type)
if adapter is None:
results.append(
{
"channel_type": channel_type,
"channel_id": channel_id,
"status": "no_adapter",
}
)
log.warning(
"notify.no_adapter",
channel_type=channel_type,
channel_id=channel_id,
)
continue
try:
msg_id = await adapter.send(channel_id, content)
results.append(
{
"channel_type": channel_type,
"channel_id": channel_id,
"status": "sent",
"message_id": msg_id,
}
)
log.info(
"notify.delivered",
channel_type=channel_type,
channel_id=channel_id,
message_id=msg_id,
)
except Exception:
log.exception(
"notify.delivery_failed",
channel_type=channel_type,
channel_id=channel_id,
)
results.append(
{
"channel_type": channel_type,
"channel_id": channel_id,
"status": "failed",
}
)
return JSONResponse({"results": results})
def create_channel_app(
adapters: dict[str, ChannelAdapter],
storage: StorageBackend,
*,
auth_token: str = "",
jwt_secret: str = "",
) -> Starlette:
"""Create the channel gateway HTTP application."""
app = Starlette(
routes=[
Route("/health", _handle_health),
Mount(
"/v1",
routes=[
Route("/api/notify", _handle_notify, methods=["POST"]),
],
),
],
)
app.state.adapters = adapters
app.state.storage = storage
app.state.auth_token = auth_token
app.state.jwt_secret = jwt_secret
return app
def _get_service_id() -> str:
"""Generate a unique service ID from hostname + random suffix."""
return f"channel-{socket.gethostname()}-{uuid.uuid4().hex[:8]}"
+6 -13
View File
@@ -150,7 +150,7 @@ class ChannelRouter:
owner = await self._broker.get_ws_owner(route["ws_id"])
if owner:
return route["ws_id"], False
# Workstream was evicted/closed — capture old ws_id for session
# Workstream was evicted/closed — capture old ws_id for
# resume, then remove the stale route.
old_ws_id = route["ws_id"]
await asyncio.to_thread(
@@ -163,20 +163,13 @@ class ChannelRouter:
channel_id=channel_id,
)
# 2. Look up old session for atomic resume (if stale route).
resume_session = ""
if old_ws_id:
old_sid: str | None = await asyncio.to_thread(
self._storage.get_session_id_by_ws, old_ws_id
)
resume_session = old_sid or ""
# 3. Create via MQ with atomic resume.
# 2. Create via MQ with atomic resume (reuse old ws_id directly).
resume_ws = old_ws_id or ""
msg = CreateWorkstreamMessage(
name=name,
model=model,
initial_message="" if resume_session else initial_message,
resume_session=resume_session,
initial_message="" if resume_ws else initial_message,
resume_ws=resume_ws,
auto_approve=self._auto_approve,
auto_approve_tools=list(self._auto_approve_tools),
)
@@ -190,7 +183,7 @@ class ChannelRouter:
correlation_id=cid,
channel_type=channel_type,
channel_id=channel_id,
resume_session=resume_session or None,
resume_ws=resume_ws or None,
)
try:
+106 -5
View File
@@ -1,8 +1,8 @@
"""Unified channel gateway entry point.
Launches one or more channel adapters (Discord, Slack, etc.) connected to
the turnstone cluster via Redis MQ. Currently supports Discord; future
adapters will be added as additional ``--*-token`` flags.
the turnstone cluster via Redis MQ. An HTTP server runs alongside for
inbound notification delivery from the server.
Run as: ``turnstone-channel --discord-token $TURNSTONE_DISCORD_TOKEN``
"""
@@ -10,6 +10,7 @@ Run as: ``turnstone-channel --discord-token $TURNSTONE_DISCORD_TOKEN``
from __future__ import annotations
import os
import socket
import sys
@@ -44,6 +45,26 @@ def main() -> None:
help="Comma-separated list of allowed Discord channel IDs (default: all)",
)
# -- HTTP server ---------------------------------------------------------
parser.add_argument(
"--http-host",
default="127.0.0.1",
help="HTTP server bind address (default: %(default)s)",
)
parser.add_argument(
"--http-port",
type=int,
default=int(os.environ.get("TURNSTONE_CHANNEL_PORT", "8091")),
help="HTTP server port (default: $TURNSTONE_CHANNEL_PORT or 8091)",
)
# -- Auth ----------------------------------------------------------------
parser.add_argument(
"--auth-token",
default=os.environ.get("TURNSTONE_CHANNEL_AUTH_TOKEN", ""),
help="Static auth token for /v1/api/notify (default: $TURNSTONE_CHANNEL_AUTH_TOKEN)",
)
# -- Workstream defaults -------------------------------------------------
parser.add_argument(
"--model",
@@ -85,6 +106,10 @@ def main() -> None:
path=db_path,
)
# -- Auth config ---------------------------------------------------------
auth_token = args.auth_token
jwt_secret = os.environ.get("TURNSTONE_JWT_SECRET", "").strip()
# -- Broker --------------------------------------------------------------
from turnstone.mq.broker import async_broker_from_args
@@ -106,6 +131,9 @@ def main() -> None:
# -- Run -----------------------------------------------------------------
if args.discord_token:
import asyncio
from turnstone.channels._http import _get_service_id, create_channel_app
from turnstone.channels.discord.bot import TurnstoneBot
from turnstone.channels.discord.config import DiscordConfig
from turnstone.core.storage._registry import get_storage
@@ -128,9 +156,82 @@ def main() -> None:
allowed_channels=allowed_channels,
)
bot = TurnstoneBot(config, broker, get_storage())
log.info("channel.starting", adapter="discord", guild_id=config.guild_id)
bot.run()
storage = get_storage()
bot = TurnstoneBot(config, broker, storage)
adapters = {"discord": bot}
# Create HTTP app for notification delivery
channel_app = create_channel_app(
adapters, # type: ignore[arg-type]
storage,
auth_token=auth_token,
jwt_secret=jwt_secret,
)
log.info(
"channel.starting",
adapter="discord",
guild_id=config.guild_id,
http_port=args.http_port,
)
async def _run_all() -> None:
"""Run Discord bot + HTTP server + service heartbeat concurrently."""
import uvicorn
service_id = _get_service_id()
# Resolve advertise URL — env override for Docker/K8s,
# otherwise derive from bind address.
advertise_url = os.environ.get("TURNSTONE_CHANNEL_ADVERTISE_URL", "").strip()
if not advertise_url:
if args.http_host in ("0.0.0.0", "::"):
advertise_host = socket.gethostname()
else:
advertise_host = args.http_host
advertise_url = f"http://{advertise_host}:{args.http_port}"
service_url = advertise_url
# Register in service registry
storage.register_service("channel", service_id, service_url)
log.info(
"channel.service_registered",
service_id=service_id,
url=service_url,
)
async def _heartbeat_loop() -> None:
"""Periodically update service heartbeat."""
while True:
await asyncio.sleep(30)
try:
await asyncio.to_thread(storage.heartbeat_service, "channel", service_id)
except Exception:
log.exception("channel.heartbeat_failed")
uv_config = uvicorn.Config(
channel_app,
host=args.http_host,
port=args.http_port,
log_level="warning",
)
server = uvicorn.Server(uv_config)
heartbeat_task = asyncio.create_task(_heartbeat_loop())
try:
await asyncio.gather(
bot.start(),
server.serve(),
)
finally:
heartbeat_task.cancel()
await asyncio.to_thread(storage.deregister_service, "channel", service_id)
log.info("channel.service_deregistered", service_id=service_id)
import contextlib
with contextlib.suppress(KeyboardInterrupt):
asyncio.run(_run_all())
if __name__ == "__main__":
+30 -4
View File
@@ -21,8 +21,8 @@ from turnstone.mq.protocol import (
ErrorEvent,
OutboundEvent,
PlanReviewEvent,
SessionResumedEvent,
TurnCompleteEvent,
WorkstreamResumedEvent,
)
if TYPE_CHECKING:
@@ -307,10 +307,10 @@ class TurnstoneBot:
if sm is not None:
await sm.finalize()
elif isinstance(event, SessionResumedEvent):
name = event.name or "previous session"
elif isinstance(event, WorkstreamResumedEvent):
name = event.name or "previous workstream"
count = event.message_count
await thread.send(f"*Session resumed: {name} ({count} messages restored)*")
await thread.send(f"*Resumed: {name} ({count} messages restored)*")
elif isinstance(event, ErrorEvent):
safe_msg = event.message[:500] if event.message else "An error occurred"
@@ -348,6 +348,32 @@ class TurnstoneBot:
"""Start the bot (async). Use this for multi-adapter ``asyncio.gather``."""
await self._bot.start(self.config.bot_token, reconnect=True)
async def send(self, channel_id: str, content: str) -> str:
"""Send a message to a Discord channel or user DM.
Implements the :class:`ChannelAdapter` protocol. Tries the ID as a
channel first; if not found, attempts a user DM. Long messages are
chunked via :func:`chunk_message`.
"""
import discord
int_id = int(channel_id)
target: discord.abc.Messageable | None = self._bot.get_channel(int_id) # type: ignore[assignment]
if target is None:
try:
user = await self._bot.fetch_user(int_id)
target = await user.create_dm()
except discord.NotFound as exc:
raise ValueError(f"Discord channel/user {channel_id} not found") from exc
content = discord.utils.escape_mentions(content)
chunks = chunk_message(content, self.config.max_message_length)
msg: discord.Message | None = None
for chunk in chunks:
msg = await target.send(chunk) # type: ignore[union-attr]
return str(msg.id) if msg else ""
async def stop(self) -> None:
"""Disconnect the bot and clean up subscriptions."""
for ws_id in list(self._subscribed_ws):
+53 -16
View File
@@ -41,7 +41,7 @@ SLASH_COMMANDS = [
"/instructions",
"/clear",
"/new",
"/sessions",
"/workstreams",
"/resume",
"/name",
"/delete",
@@ -784,11 +784,29 @@ def main() -> None:
default=0,
help="Tool output truncation limit in chars, 0 for auto (50%% of context window) (default: 0)",
)
parser.add_argument(
"--tool-search",
choices=["auto", "on", "off"],
default="auto",
help="Dynamic tool search: auto (enable when tool count exceeds threshold), on, off (default: auto)",
)
parser.add_argument(
"--tool-search-threshold",
type=int,
default=20,
help="Min tools before tool search activates (default: 20)",
)
parser.add_argument(
"--tool-search-max-results",
type=int,
default=5,
help="Max tools returned per tool search query (default: 5)",
)
parser.add_argument(
"--resume",
default=None,
metavar="SESSION",
help="Resume a previous session by alias or session_id",
metavar="WS",
help="Resume a previous workstream by alias or ws_id",
)
parser.add_argument(
"--skip-permissions",
@@ -801,11 +819,11 @@ def main() -> None:
help="API key (default: $OPENAI_API_KEY, or 'dummy' for local servers)",
)
parser.add_argument(
"--session-retention-days",
"--retention-days",
type=int,
default=90,
metavar="DAYS",
help="Delete unnamed sessions older than DAYS days on startup, 0 to disable (default: 90)",
help="Delete unnamed workstreams older than DAYS days on startup, 0 to disable (default: 90)",
)
parser.add_argument(
"--console-url",
@@ -823,6 +841,16 @@ def main() -> None:
metavar="PATH",
help="Path to MCP server config file (standard mcpServers JSON format)",
)
from turnstone.core.config import nonneg_float
parser.add_argument(
"--mcp-refresh-interval",
type=nonneg_float,
default=14400,
metavar="SECONDS",
help="Periodic MCP tool refresh interval for servers without push notifications (default: 14400 = 4h, 0 to disable)",
)
from turnstone.core.config import apply_config
apply_config(parser, ["api", "model", "session", "tools", "console", "auth", "mcp", "database"])
@@ -845,10 +873,10 @@ def main() -> None:
)
init_storage(db_backend, path=db_path, url=db_url, pool_size=db_pool_size)
# Prune stale / empty sessions on startup
from turnstone.core.memory import prune_sessions
# Prune stale / empty workstreams on startup
from turnstone.core.memory import prune_workstreams
prune_sessions(retention_days=args.session_retention_days, log_fn=print)
prune_workstreams(retention_days=args.retention_days, log_fn=print)
# Set up readline
setup_readline()
@@ -892,9 +920,12 @@ def main() -> None:
# Initialize MCP client (connects to configured MCP servers, if any)
from turnstone.core.mcp_client import create_mcp_client
mcp_client = create_mcp_client(getattr(args, "mcp_config", None))
mcp_client = create_mcp_client(
getattr(args, "mcp_config", None),
refresh_interval=getattr(args, "mcp_refresh_interval", 14400),
)
# Session factory — captures shared config for creating workstream sessions
# ChatSession factory — captures shared config for creating workstreams
def session_factory(
ui: SessionUI | None, model_alias: str | None = None, ws_id: str | None = None
) -> ChatSession:
@@ -917,6 +948,9 @@ def main() -> None:
mcp_client=mcp_client,
registry=registry,
model_alias=model_alias or registry.default,
tool_search=args.tool_search,
tool_search_threshold=args.tool_search_threshold,
tool_search_max_results=args.tool_search_max_results,
)
# Create workstream manager and initial workstream
@@ -929,19 +963,19 @@ def main() -> None:
# Handle --resume
if args.resume:
from turnstone.core.memory import resolve_session
from turnstone.core.memory import resolve_workstream
target_id = resolve_session(args.resume)
target_id = resolve_workstream(args.resume)
if not target_id:
print(red(f"Session not found: {args.resume}"))
print(red(f"Workstream not found: {args.resume}"))
sys.exit(1)
if ws.session is None:
print(red("No session available."))
sys.exit(1)
if not ws.session.resume_session(target_id):
print(red(f"Session '{args.resume}' has no messages."))
if not ws.session.resume(target_id):
print(red(f"Workstream '{args.resume}' has no messages."))
sys.exit(1)
print(f"Resumed session {bold(target_id)} ({len(ws.session.messages)} messages)")
print(f"Resumed workstream {bold(target_id)} ({len(ws.session.messages)} messages)")
# Background attention notification — write to stderr while user types
def _bg_attention_notify(ws_id: str, state: WorkstreamState) -> None:
@@ -1020,6 +1054,9 @@ def main() -> None:
except Exception as e:
print(f"\n{red(f'Error: {e}')}")
# Close active session (removes MCP listener) before shutting down MCP
if active and active.session:
active.session.close()
if mcp_client:
mcp_client.shutdown()
registry.shutdown()
+259
View File
@@ -0,0 +1,259 @@
"""Background task scheduler for timed workstream dispatch.
Runs as a daemon thread inside the console process. Checks for due tasks
every ``check_interval`` seconds and dispatches them as
``CreateWorkstreamMessage`` via the MQ broker.
Uses Redis ``SET NX EX`` for distributed locking in multi-console deployments.
"""
from __future__ import annotations
import threading
import uuid
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
import structlog
if TYPE_CHECKING:
from turnstone.console.collector import ClusterCollector
from turnstone.core.storage._protocol import StorageBackend
from turnstone.mq.broker import RedisBroker
log = structlog.get_logger(__name__)
def _pick_best_node(collector: ClusterCollector) -> str:
"""Select the reachable node with the most available capacity."""
nodes, _ = collector.get_nodes(sort_by="activity", limit=1000, offset=0)
best_id = ""
best_headroom = -1
for n in nodes:
if not n.get("reachable", False):
continue
headroom = n.get("max_ws", 10) - n.get("ws_total", 0)
if headroom > best_headroom:
best_headroom = headroom
best_id = n["node_id"]
return best_id
class TaskScheduler:
"""Background scheduler for dispatching timed workstreams."""
def __init__(
self,
broker: RedisBroker,
collector: ClusterCollector,
storage: StorageBackend,
prefix: str = "turnstone",
check_interval: float = 15.0,
lock_ttl: int = 60,
max_fan_out: int = 20,
) -> None:
self._broker = broker
self._collector = collector
self._storage = storage
self._prefix = prefix
self._check_interval = check_interval
self._lock_ttl = lock_ttl
self._max_fan_out = max_fan_out
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
self._tick_count = 0
self._prune_every = 240 # ~1 hour at 15s intervals
def start(self) -> None:
"""Start the scheduler daemon thread."""
self._stop_event.clear()
self._thread = threading.Thread(target=self._loop, daemon=True, name="scheduler")
self._thread.start()
log.info("scheduler.started", check_interval=self._check_interval)
def stop(self) -> None:
"""Stop the scheduler and wait for the thread to finish."""
self._stop_event.set()
if self._thread is not None:
self._thread.join(timeout=5)
log.info("scheduler.stopped")
def _loop(self) -> None:
"""Main scheduler loop — tick then sleep."""
while not self._stop_event.is_set():
try:
self._tick()
except Exception:
log.exception("scheduler.tick_error")
self._stop_event.wait(self._check_interval)
# Lua script for safe lock release — only delete if we still own the lock
_UNLOCK_SCRIPT = "if redis.call('get',KEYS[1])==ARGV[1] then return redis.call('del',KEYS[1]) else return 0 end"
def _tick(self) -> None:
"""Single scheduler iteration: acquire lock, query due tasks, dispatch."""
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
# Distributed lock with unique owner — prevents releasing another instance's lock
lock_key = f"{self._prefix}:scheduler:lock"
lock_value = uuid.uuid4().hex
acquired = self._broker._redis.set(lock_key, lock_value, nx=True, ex=self._lock_ttl)
if not acquired:
return
try:
due_tasks = self._storage.list_due_tasks(now)
for task in due_tasks:
self._dispatch_task(task, now)
# Periodic run history pruning (~once per hour)
self._tick_count += 1
if self._tick_count % self._prune_every == 0:
pruned = self._storage.prune_task_runs(retention_days=90)
if pruned:
log.info("scheduler.pruned_runs", count=pruned)
finally:
# Only release our own lock (safe even if TTL expired and another took it)
self._broker._redis.eval( # type: ignore[no-untyped-call]
self._UNLOCK_SCRIPT, 1, lock_key, lock_value
)
def _dispatch_task(self, task: dict[str, Any], now: str) -> None:
"""Dispatch a single task as one or more CreateWorkstreamMessages."""
target_mode = task["target_mode"]
task_id = task["task_id"]
dispatched = False
if target_mode == "all":
nodes, _ = self._collector.get_nodes(sort_by="activity", limit=1000, offset=0)
fan_count = 0
for n in nodes:
if n.get("reachable", False):
if fan_count >= self._max_fan_out:
log.warning(
"scheduler.fan_out_capped",
task_id=task_id,
max_fan_out=self._max_fan_out,
)
break
self._dispatch_to_node(task, n["node_id"], now)
fan_count += 1
dispatched = True
if not dispatched:
self._record_failure(task, now, "No reachable nodes for fan-out")
elif target_mode == "pool":
self._dispatch_to_pool(task, now)
dispatched = True
elif target_mode == "auto":
node_id = _pick_best_node(self._collector)
if node_id:
self._dispatch_to_node(task, node_id, now)
dispatched = True
else:
self._record_failure(task, now, "No reachable nodes")
else:
# Specific node_id
self._dispatch_to_node(task, target_mode, now)
dispatched = True
if not dispatched:
return # Don't advance schedule on failure
# Update last_run and compute next_run
next_run = self._compute_next_run(task)
if task["schedule_type"] == "at":
self._storage.update_scheduled_task(task_id, last_run=now, next_run="", enabled=False)
else:
self._storage.update_scheduled_task(task_id, last_run=now, next_run=next_run)
log_kw: dict[str, Any] = {
"task_id": task_id,
"target_mode": target_mode,
"schedule_type": task["schedule_type"],
"created_by": task.get("created_by", ""),
}
if task.get("auto_approve", 0):
log_kw["auto_approve"] = True
log_kw["auto_approve_tools"] = task.get("auto_approve_tools", "")
log.warning("scheduler.task_dispatched_auto_approve", **log_kw)
else:
log.info("scheduler.task_dispatched", **log_kw)
@staticmethod
def _parse_tools(task: dict[str, Any]) -> list[str]:
raw = task.get("auto_approve_tools", "")
return [t.strip() for t in raw.split(",") if t.strip()]
def _dispatch_to_node(self, task: dict[str, Any], node_id: str, now: str) -> None:
"""Send a CreateWorkstreamMessage to a specific node."""
from turnstone.mq.protocol import CreateWorkstreamMessage
msg = CreateWorkstreamMessage(
name=task["name"],
model=task.get("model", ""),
target_node=node_id,
initial_message=task["initial_message"],
auto_approve=bool(task.get("auto_approve", 0)),
auto_approve_tools=self._parse_tools(task),
user_id=task.get("created_by", ""),
)
self._broker.push_inbound(msg.to_json(), node_id=node_id)
self._storage.record_task_run(
run_id=uuid.uuid4().hex,
task_id=task["task_id"],
node_id=node_id,
ws_id="",
correlation_id=msg.correlation_id,
started=now,
status="dispatched",
error="",
)
def _dispatch_to_pool(self, task: dict[str, Any], now: str) -> None:
"""Send a CreateWorkstreamMessage to the shared pool queue."""
from turnstone.mq.protocol import CreateWorkstreamMessage
msg = CreateWorkstreamMessage(
name=task["name"],
model=task.get("model", ""),
initial_message=task["initial_message"],
auto_approve=bool(task.get("auto_approve", 0)),
auto_approve_tools=self._parse_tools(task),
user_id=task.get("created_by", ""),
)
self._broker.push_inbound(msg.to_json())
self._storage.record_task_run(
run_id=uuid.uuid4().hex,
task_id=task["task_id"],
node_id="pool",
ws_id="",
correlation_id=msg.correlation_id,
started=now,
status="dispatched",
error="",
)
def _record_failure(self, task: dict[str, Any], now: str, error: str) -> None:
"""Record a failed dispatch attempt."""
self._storage.record_task_run(
run_id=uuid.uuid4().hex,
task_id=task["task_id"],
node_id="",
ws_id="",
correlation_id="",
started=now,
status="failed",
error=error,
)
log.warning("scheduler.dispatch_failed", task_id=task["task_id"], error=error)
@staticmethod
def _compute_next_run(task: dict[str, Any]) -> str:
"""Compute the next run time. Returns empty string for one-shot tasks."""
from turnstone.console.server import _compute_next_run
return _compute_next_run(
task["schedule_type"], task.get("cron_expr", ""), task.get("at_time", "")
)
+325 -16
View File
@@ -30,7 +30,7 @@ import httpx
from sse_starlette import EventSourceResponse
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.responses import HTMLResponse, JSONResponse, Response
from starlette.responses import HTMLResponse, JSONResponse, Response, StreamingResponse
from starlette.routing import Mount, Route
from starlette.staticfiles import StaticFiles
@@ -568,7 +568,11 @@ async def _proxy_post(
async def _proxy_sse(
request: Request, server_url: str, path: str, *, api_prefix: str = "api"
) -> Response:
"""Proxy an SSE stream from the target server to the browser."""
"""Proxy an SSE stream from the target server to the browser.
Relays raw bytes verbatim so server-side ping comments, event framing,
and keepalives all pass through unchanged.
"""
target = f"{server_url}/{api_prefix}/{path}"
if request.url.query:
target += f"?{request.url.query}"
@@ -576,30 +580,34 @@ async def _proxy_sse(
sse_client: httpx.AsyncClient = request.app.state.proxy_sse_client
sse_auth = _proxy_auth_headers(request)
async def sse_generator() -> AsyncGenerator[dict[str, str], None]:
from httpx_sse import aconnect_sse
async def raw_stream() -> AsyncGenerator[bytes, None]:
try:
async with aconnect_sse(sse_client, "GET", target, headers=sse_auth) as source:
if source.response.status_code != 200:
async with sse_client.stream(
"GET",
target,
headers={**sse_auth, "Accept": "text/event-stream", "Cache-Control": "no-store"},
timeout=httpx.Timeout(connect=10, read=None, write=5, pool=None),
) as response:
if response.status_code != 200:
log.debug(
"SSE proxy received status %s from %s",
source.response.status_code,
response.status_code,
target,
)
yield {
"event": "error",
"data": f"Upstream returned status {source.response.status_code}",
}
yield f"event: error\ndata: Upstream returned status {response.status_code}\n\n".encode()
return
async for sse in source.aiter_sse():
async for chunk in response.aiter_bytes():
if await request.is_disconnected():
return
yield {"event": sse.event, "data": sse.data}
yield chunk
except httpx.HTTPError:
log.debug("SSE proxy stream ended for %s", target)
return EventSourceResponse(sse_generator(), ping=5)
return StreamingResponse(
raw_stream(),
media_type="text/event-stream",
headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"},
)
# ---------------------------------------------------------------------------
@@ -620,8 +628,14 @@ async def _lifespan(app: Starlette) -> AsyncGenerator[None, None]:
timeout=httpx.Timeout(connect=5, read=30, write=5, pool=5),
headers=headers,
)
# Start scheduler if configured
scheduler = getattr(app.state, "scheduler", None)
if scheduler is not None:
scheduler.start()
yield
# Shutdown
if scheduler is not None:
scheduler.stop()
await app.state.proxy_sse_client.aclose()
await app.state.proxy_client.aclose()
app.state.collector.stop()
@@ -878,6 +892,277 @@ async def admin_delete_channel(request: Request) -> JSONResponse:
return JSONResponse({"error": "Channel link not found"}, status_code=404)
# ---------------------------------------------------------------------------
# Admin API endpoints — scheduled tasks
# ---------------------------------------------------------------------------
def _normalize_task_dict(task: dict[str, Any]) -> dict[str, Any]:
"""Convert DB row ints/csv to JSON-friendly bools/lists."""
tools_str = task.get("auto_approve_tools", "")
task["auto_approve_tools"] = [s.strip() for s in tools_str.split(",") if s.strip()]
task["auto_approve"] = bool(task.get("auto_approve", 0))
task["enabled"] = bool(task.get("enabled", 1))
return task
def _compute_next_run(schedule_type: str, cron_expr: str, at_time: str) -> str:
"""Compute the next run time for a schedule. Empty string if invalid."""
if schedule_type == "at":
return at_time
if schedule_type == "cron" and cron_expr:
from datetime import UTC, datetime
from croniter import croniter
cron = croniter(cron_expr, datetime.now(UTC))
next_dt = cron.get_next(datetime)
return str(next_dt.strftime("%Y-%m-%dT%H:%M:%S"))
return ""
def _validate_schedule_fields(schedule_type: str, cron_expr: str, at_time: str) -> str | None:
"""Validate schedule type/expression. Returns error string or None."""
if schedule_type not in ("cron", "at"):
return "schedule_type must be 'cron' or 'at'"
if schedule_type == "cron":
if not cron_expr:
return "cron_expr is required when schedule_type is 'cron'"
from croniter import croniter
if not croniter.is_valid(cron_expr):
return f"Invalid cron expression: {cron_expr}"
if schedule_type == "at":
if not at_time:
return "at_time is required when schedule_type is 'at'"
from datetime import UTC, datetime
try:
dt = datetime.fromisoformat(at_time)
if dt.tzinfo is None:
return (
"at_time must include a timezone offset (e.g. 2024-01-01T12:00:00Z or +00:00)"
)
if dt <= datetime.now(UTC):
return "at_time must be in the future"
except ValueError:
return "at_time must be a valid ISO8601 timestamp with timezone"
return None
async def admin_list_schedules(request: Request) -> JSONResponse:
"""GET /v1/api/admin/schedules — list all scheduled tasks."""
from turnstone.core.web_helpers import require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
tasks = storage.list_scheduled_tasks()
for t in tasks:
_normalize_task_dict(t)
return JSONResponse({"schedules": tasks})
async def admin_create_schedule(request: Request) -> JSONResponse:
"""POST /v1/api/admin/schedules — create a scheduled task."""
import uuid
from turnstone.core.web_helpers import read_json_or_400, require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
body = await read_json_or_400(request)
if isinstance(body, JSONResponse):
return body
name = str(body.get("name", "")).strip()[:256]
description = str(body.get("description", "")).strip()[:1024]
schedule_type = str(body.get("schedule_type", "")).strip()
cron_expr = str(body.get("cron_expr", "")).strip()[:256]
at_time = str(body.get("at_time", "")).strip()[:64]
target_mode = str(body.get("target_mode", "auto")).strip()[:256]
model = str(body.get("model", "")).strip()[:128]
initial_message = str(body.get("initial_message", "")).strip()[:4096]
auto_approve = bool(body.get("auto_approve", False))
raw_tools = body.get("auto_approve_tools", [])
auto_approve_tools = raw_tools if isinstance(raw_tools, list) else []
enabled = bool(body.get("enabled", True))
if not name:
return JSONResponse({"error": "name is required"}, status_code=400)
if not initial_message:
return JSONResponse({"error": "initial_message is required"}, status_code=400)
validation_err = _validate_schedule_fields(schedule_type, cron_expr, at_time)
if validation_err:
return JSONResponse({"error": validation_err}, status_code=400)
if not target_mode:
return JSONResponse({"error": "target_mode is required"}, status_code=400)
# Cap total schedule count to prevent unbounded growth
max_schedules = 200
existing = storage.list_scheduled_tasks()
if len(existing) >= max_schedules:
return JSONResponse(
{"error": f"Maximum of {max_schedules} schedules reached"}, status_code=409
)
next_run = _compute_next_run(schedule_type, cron_expr, at_time)
task_id = uuid.uuid4().hex
created_by = getattr(getattr(request, "state", None), "user_id", "")
storage.create_scheduled_task(
task_id=task_id,
name=name,
description=description,
schedule_type=schedule_type,
cron_expr=cron_expr,
at_time=at_time,
target_mode=target_mode,
model=model,
initial_message=initial_message,
auto_approve=auto_approve,
auto_approve_tools=auto_approve_tools,
created_by=created_by,
next_run=next_run if enabled else "",
)
if not enabled:
# Storage backends default enabled=1 on create; persist user's choice
storage.update_scheduled_task(task_id, enabled=False)
task = storage.get_scheduled_task(task_id)
if task:
_normalize_task_dict(task)
return JSONResponse(task)
async def admin_get_schedule(request: Request) -> JSONResponse:
"""GET /v1/api/admin/schedules/{task_id} — get single task."""
from turnstone.core.web_helpers import require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
task_id = request.path_params["task_id"]
task = storage.get_scheduled_task(task_id)
if task is None:
return JSONResponse({"error": "Schedule not found"}, status_code=404)
_normalize_task_dict(task)
return JSONResponse(task)
async def admin_update_schedule(request: Request) -> JSONResponse:
"""PUT /v1/api/admin/schedules/{task_id} — partial update."""
from turnstone.core.web_helpers import read_json_or_400, require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
task_id = request.path_params["task_id"]
existing = storage.get_scheduled_task(task_id)
if existing is None:
return JSONResponse({"error": "Schedule not found"}, status_code=404)
body = await read_json_or_400(request)
if isinstance(body, JSONResponse):
return body
updates: dict[str, Any] = {}
if "name" in body:
updates["name"] = str(body["name"]).strip()[:256]
if "description" in body:
updates["description"] = str(body["description"]).strip()[:1024]
if "schedule_type" in body:
updates["schedule_type"] = str(body["schedule_type"]).strip()
if "cron_expr" in body:
updates["cron_expr"] = str(body["cron_expr"]).strip()[:256]
if "at_time" in body:
updates["at_time"] = str(body["at_time"]).strip()[:64]
if "target_mode" in body:
updates["target_mode"] = str(body["target_mode"]).strip()[:256]
if "model" in body:
updates["model"] = str(body["model"]).strip()[:128]
if "initial_message" in body:
updates["initial_message"] = str(body["initial_message"]).strip()[:4096]
if "auto_approve" in body:
updates["auto_approve"] = bool(body["auto_approve"])
if "auto_approve_tools" in body:
raw = body["auto_approve_tools"]
updates["auto_approve_tools"] = raw if isinstance(raw, list) else []
if "enabled" in body:
updates["enabled"] = bool(body["enabled"])
# Validate schedule fields if changed
stype = updates.get("schedule_type", existing["schedule_type"])
cexpr = updates.get("cron_expr", existing["cron_expr"])
atime = updates.get("at_time", existing["at_time"])
schedule_fields_changed = (
"schedule_type" in updates or "cron_expr" in updates or "at_time" in updates
)
if schedule_fields_changed:
validation_err = _validate_schedule_fields(stype, cexpr, atime)
if validation_err:
return JSONResponse({"error": validation_err}, status_code=400)
# Recompute next_run if schedule changed or enabled toggled
if schedule_fields_changed or "enabled" in updates:
enabled = updates.get("enabled", bool(existing.get("enabled", 1)))
if enabled:
# Re-validate at_time when re-enabling a one-shot task
if stype == "at" and not schedule_fields_changed:
validation_err = _validate_schedule_fields(stype, cexpr, atime)
if validation_err:
return JSONResponse({"error": validation_err}, status_code=400)
updates["next_run"] = _compute_next_run(stype, cexpr, atime)
else:
updates["next_run"] = ""
storage.update_scheduled_task(task_id, **updates)
task = storage.get_scheduled_task(task_id)
if task:
_normalize_task_dict(task)
return JSONResponse(task)
async def admin_delete_schedule(request: Request) -> JSONResponse:
"""DELETE /v1/api/admin/schedules/{task_id} — delete task + runs."""
from turnstone.core.web_helpers import require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
task_id = request.path_params["task_id"]
if storage.delete_scheduled_task(task_id):
return JSONResponse({"status": "ok"})
return JSONResponse({"error": "Schedule not found"}, status_code=404)
async def admin_list_schedule_runs(request: Request) -> JSONResponse:
"""GET /v1/api/admin/schedules/{task_id}/runs — run history."""
from turnstone.core.web_helpers import require_storage_or_503
storage, err = require_storage_or_503(request)
if err:
return err
task_id = request.path_params["task_id"]
# Verify task exists
if storage.get_scheduled_task(task_id) is None:
return JSONResponse({"error": "Schedule not found"}, status_code=404)
try:
limit = min(int(request.query_params.get("limit", "50")), 200)
except (ValueError, TypeError):
limit = 50
runs = storage.list_task_runs(task_id, limit=limit)
return JSONResponse({"runs": runs})
# ---------------------------------------------------------------------------
# App factory
# ---------------------------------------------------------------------------
@@ -937,6 +1222,16 @@ def create_app(
admin_delete_channel,
methods=["DELETE"],
),
Route("/api/admin/schedules", admin_list_schedules),
Route("/api/admin/schedules", admin_create_schedule, methods=["POST"]),
Route("/api/admin/schedules/{task_id}", admin_get_schedule),
Route("/api/admin/schedules/{task_id}", admin_update_schedule, methods=["PUT"]),
Route(
"/api/admin/schedules/{task_id}",
admin_delete_schedule,
methods=["DELETE"],
),
Route("/api/admin/schedules/{task_id}/runs", admin_list_schedule_runs),
],
),
Route("/health", health),
@@ -966,6 +1261,20 @@ def create_app(
from turnstone.core.auth import LoginRateLimiter
app.state.login_limiter = LoginRateLimiter()
# Scheduler — start background thread if storage is available
if auth_storage is not None:
from turnstone.console.scheduler import TaskScheduler
scheduler = TaskScheduler(
broker=broker,
collector=collector,
storage=auth_storage,
)
app.state.scheduler = scheduler
else:
app.state.scheduler = None
return app
@@ -1100,7 +1409,7 @@ def main() -> None:
proxy_token_mgr = ServiceTokenManager(
user_id="console-proxy",
scopes=frozenset({"write"}),
scopes=frozenset({"read", "write", "approve"}),
source="console",
secret=jwt_secret,
audience=JWT_AUD_SERVER,
+538 -2
View File
@@ -46,10 +46,13 @@ function switchAdminTab(tab) {
tab === "tokens" ? "" : "none";
document.getElementById("admin-channels").style.display =
tab === "channels" ? "" : "none";
document.getElementById("admin-schedules").style.display =
tab === "schedules" ? "" : "none";
if (tab === "users") loadAdminUsers();
if (tab === "tokens") _populateTokenUserSelect();
if (tab === "channels") _populateChannelUserSelect();
if (tab === "schedules") loadAdminSchedules();
}
// ---------------------------------------------------------------------------
@@ -373,6 +376,517 @@ function confirmUnlinkChannel(channelType, channelUserId) {
);
}
// ---------------------------------------------------------------------------
// Schedules
// ---------------------------------------------------------------------------
var _csTrapHandler = null;
var _esTrapHandler = null;
var _srTrapHandler = null;
var _editScheduleTriggerEl = null;
var _runsScheduleTriggerEl = null;
function loadAdminSchedules() {
authFetch("/v1/api/admin/schedules")
.then(function (r) {
if (!r.ok) throw new Error("Failed to load schedules");
return r.json();
})
.then(function (data) {
_renderSchedules(data.schedules || []);
})
.catch(function () {
document.getElementById("admin-schedules-table").innerHTML =
'<div class="dashboard-empty">Failed to load schedules</div>';
});
}
function _renderSchedules(schedules) {
var container = document.getElementById("admin-schedules-table");
if (!schedules.length) {
container.innerHTML =
'<div class="dashboard-empty">No scheduled tasks. Create one to get started.</div>';
return;
}
var html = "";
for (var i = 0; i < schedules.length; i++) {
var s = schedules[i];
var typeLabel = s.schedule_type === "cron" ? "cron" : "at";
var typeCls = s.schedule_type === "cron" ? "scope-write" : "scope-approve";
var schedule =
s.schedule_type === "cron"
? s.cron_expr
: (s.at_time || "").slice(0, 16).replace("T", " ");
var target = s.target_mode;
var nextRun = s.next_run
? escapeHtml(s.next_run).slice(0, 16).replace("T", " ")
: "\u2014";
var enabled = s.enabled;
var statusCls = enabled ? "sched-active" : "sched-disabled";
var statusLabel = enabled ? "active" : "disabled";
var statusDot = enabled ? "\u25cf " : "\u25cb ";
if (s.schedule_type === "at" && !enabled && s.last_run) {
statusCls = "sched-expired";
statusLabel = "completed";
statusDot = "\u25c9 ";
}
html +=
'<div class="admin-row" role="listitem">' +
'<span class="admin-col admin-col-sname">' +
escapeHtml(s.name) +
"</span>" +
'<span class="admin-col admin-col-stype"><span class="scope-badge ' +
typeCls +
'">' +
typeLabel +
"</span></span>" +
'<span class="admin-col admin-col-sschedule"><code>' +
escapeHtml(schedule) +
"</code></span>" +
'<span class="admin-col admin-col-starget">' +
escapeHtml(target) +
"</span>" +
'<span class="admin-col admin-col-snext">' +
nextRun +
"</span>" +
'<span class="admin-col admin-col-sstatus"><span class="' +
statusCls +
'">' +
statusDot +
statusLabel +
"</span></span>" +
'<span class="admin-col admin-col-actions">' +
'<button class="admin-btn-action" data-edit-sched="' +
escapeHtml(s.task_id) +
'" title="Edit">edit</button>' +
'<button class="admin-btn-action" data-runs-sched="' +
escapeHtml(s.task_id) +
'" title="Run history">runs</button>' +
'<button class="admin-btn-action" data-toggle-sched="' +
escapeHtml(s.task_id) +
'" data-enabled="' +
(enabled ? "1" : "0") +
'" title="' +
(enabled ? "Disable" : "Enable") +
'">' +
(enabled ? "disable" : "enable") +
"</button>" +
'<button class="admin-btn-danger" data-delete-sched="' +
escapeHtml(s.task_id) +
'" data-sname="' +
escapeHtml(s.name) +
'" title="Delete">delete</button>' +
"</span></div>";
}
container.innerHTML = html;
// Bind buttons
var editBtns = container.querySelectorAll("[data-edit-sched]");
for (var j = 0; j < editBtns.length; j++) {
editBtns[j].addEventListener("click", function () {
showEditScheduleModal(this.getAttribute("data-edit-sched"));
});
}
var runsBtns = container.querySelectorAll("[data-runs-sched]");
for (var k = 0; k < runsBtns.length; k++) {
runsBtns[k].addEventListener("click", function () {
showScheduleRuns(this.getAttribute("data-runs-sched"));
});
}
var toggleBtns = container.querySelectorAll("[data-toggle-sched]");
for (var m = 0; m < toggleBtns.length; m++) {
toggleBtns[m].addEventListener("click", function () {
toggleSchedule(
this.getAttribute("data-toggle-sched"),
this.getAttribute("data-enabled") === "1",
);
});
}
var delBtns = container.querySelectorAll("[data-delete-sched]");
for (var n = 0; n < delBtns.length; n++) {
delBtns[n].addEventListener("click", function () {
confirmDeleteSchedule(
this.getAttribute("data-delete-sched"),
this.getAttribute("data-sname"),
);
});
}
}
function toggleSchedule(taskId, currentlyEnabled) {
authFetch("/v1/api/admin/schedules/" + encodeURIComponent(taskId), {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ enabled: !currentlyEnabled }),
})
.then(function (r) {
if (!r.ok) throw new Error("Toggle failed");
showToast(currentlyEnabled ? "Schedule disabled" : "Schedule enabled");
loadAdminSchedules();
})
.catch(function () {
showToast("Failed to toggle schedule");
});
}
function confirmDeleteSchedule(taskId, name) {
showConfirmModal(
"Delete Schedule",
"Delete schedule \u2018" +
name +
"\u2019 and its run history? This cannot be undone.",
"Delete",
function () {
authFetch("/v1/api/admin/schedules/" + encodeURIComponent(taskId), {
method: "DELETE",
})
.then(function (r) {
if (!r.ok) throw new Error("Delete failed");
showToast("Schedule deleted");
loadAdminSchedules();
})
.catch(function () {
showToast("Failed to delete schedule");
});
},
);
}
// --- Create Schedule Modal ---
function toggleScheduleTypeFields() {
var t = document.getElementById("cs-type").value;
document.getElementById("cs-cron-group").style.display =
t === "cron" ? "" : "none";
document.getElementById("cs-at-group").style.display =
t === "at" ? "" : "none";
if (t === "cron") document.getElementById("cs-cron").focus();
else document.getElementById("cs-at").focus();
}
function toggleScheduleNodeField() {
var v = document.getElementById("cs-target").value;
document.getElementById("cs-node-group").style.display =
v === "node" ? "" : "none";
if (v === "node") document.getElementById("cs-node").focus();
}
function showCreateScheduleModal() {
var overlay = document.getElementById("create-schedule-overlay");
overlay.style.display = "flex";
document.getElementById("create-schedule-error").style.display = "none";
document.getElementById("cs-name").value = "";
document.getElementById("cs-desc").value = "";
document.getElementById("cs-type").value = "cron";
document.getElementById("cs-cron").value = "";
document.getElementById("cs-at").value = "";
document.getElementById("cs-target").value = "auto";
document.getElementById("cs-node").value = "";
document.getElementById("cs-model").value = "";
document.getElementById("cs-message").value = "";
document.getElementById("cs-autoapprove").checked = false;
toggleScheduleTypeFields();
toggleScheduleNodeField();
document.getElementById("cs-submit").disabled = false;
document.getElementById("cs-submit").textContent = "Create";
_csTrapHandler = _installTrap(
"create-schedule-overlay",
"create-schedule-box",
);
setTimeout(function () {
document.getElementById("cs-name").focus();
}, 50);
}
function hideCreateScheduleModal() {
document.getElementById("create-schedule-overlay").style.display = "none";
_csTrapHandler = _removeTrap(_csTrapHandler);
var trigger = document.querySelector("#admin-schedules .admin-action-btn");
if (trigger) trigger.focus();
}
function submitCreateSchedule() {
var name = (document.getElementById("cs-name").value || "").trim();
var desc = (document.getElementById("cs-desc").value || "").trim();
var schedType = document.getElementById("cs-type").value;
var cronExpr = (document.getElementById("cs-cron").value || "").trim();
var atTime = document.getElementById("cs-at").value || "";
var targetMode = document.getElementById("cs-target").value;
var nodeId = (document.getElementById("cs-node").value || "").trim();
var model = (document.getElementById("cs-model").value || "").trim();
var message = (document.getElementById("cs-message").value || "").trim();
var autoApprove = document.getElementById("cs-autoapprove").checked;
var errEl = document.getElementById("create-schedule-error");
if (!name) return _showModalError(errEl, "Name is required");
if (!message) return _showModalError(errEl, "Initial message is required");
if (schedType === "cron" && !cronExpr)
return _showModalError(errEl, "Cron expression is required");
if (schedType === "at" && !atTime)
return _showModalError(errEl, "Run time is required");
// Normalize datetime-local to "YYYY-MM-DDTHH:MM:SS+00:00" (UTC)
if (schedType === "at" && atTime) {
if (atTime.length === 16) atTime += ":00";
else if (atTime.length > 19) atTime = atTime.slice(0, 19);
atTime += "+00:00";
}
if (targetMode === "node") targetMode = nodeId;
var btn = document.getElementById("cs-submit");
btn.disabled = true;
btn.textContent = "Creating\u2026";
authFetch("/v1/api/admin/schedules", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
name: name,
description: desc,
schedule_type: schedType,
cron_expr: cronExpr,
at_time: atTime,
target_mode: targetMode,
model: model,
initial_message: message,
auto_approve: autoApprove,
}),
})
.then(function (r) {
if (!r.ok)
return r.json().then(function (d) {
throw new Error(d.error || "Failed");
});
return r.json();
})
.then(function () {
hideCreateScheduleModal();
showToast("Schedule '" + name + "' created");
loadAdminSchedules();
})
.catch(function (err) {
btn.disabled = false;
btn.textContent = "Create";
_showModalError(errEl, err.message || "Failed to create schedule");
});
}
// --- Edit Schedule Modal ---
function toggleEditScheduleTypeFields() {
var t = document.getElementById("es-type").value;
document.getElementById("es-cron-group").style.display =
t === "cron" ? "" : "none";
document.getElementById("es-at-group").style.display =
t === "at" ? "" : "none";
if (t === "cron") document.getElementById("es-cron").focus();
else document.getElementById("es-at").focus();
}
function toggleEditScheduleNodeField() {
var v = document.getElementById("es-target").value;
document.getElementById("es-node-group").style.display =
v === "node" ? "" : "none";
if (v === "node") document.getElementById("es-node").focus();
}
function showEditScheduleModal(taskId) {
_editScheduleTriggerEl = document.activeElement;
authFetch("/v1/api/admin/schedules/" + encodeURIComponent(taskId))
.then(function (r) {
if (!r.ok) throw new Error("Not found");
return r.json();
})
.then(function (s) {
document.getElementById("es-id").value = s.task_id;
document.getElementById("es-name").value = s.name || "";
document.getElementById("es-desc").value = s.description || "";
document.getElementById("es-type").value = s.schedule_type;
document.getElementById("es-cron").value = s.cron_expr || "";
document.getElementById("es-at").value = (s.at_time || "").slice(0, 16);
var isSpecificNode =
s.target_mode &&
s.target_mode !== "auto" &&
s.target_mode !== "pool" &&
s.target_mode !== "all";
document.getElementById("es-target").value = isSpecificNode
? "node"
: s.target_mode;
document.getElementById("es-node").value = isSpecificNode
? s.target_mode
: "";
document.getElementById("es-model").value = s.model || "";
document.getElementById("es-message").value = s.initial_message || "";
document.getElementById("es-autoapprove").checked = !!s.auto_approve;
document.getElementById("es-enabled").checked = !!s.enabled;
toggleEditScheduleTypeFields();
toggleEditScheduleNodeField();
document.getElementById("edit-schedule-error").style.display = "none";
document.getElementById("es-submit").disabled = false;
document.getElementById("es-submit").textContent = "Save";
var overlay = document.getElementById("edit-schedule-overlay");
overlay.style.display = "flex";
_esTrapHandler = _installTrap(
"edit-schedule-overlay",
"edit-schedule-box",
);
setTimeout(function () {
document.getElementById("es-name").focus();
}, 50);
})
.catch(function () {
showToast("Failed to load schedule");
});
}
function hideEditScheduleModal() {
document.getElementById("edit-schedule-overlay").style.display = "none";
_esTrapHandler = _removeTrap(_esTrapHandler);
if (_editScheduleTriggerEl && _editScheduleTriggerEl.isConnected) {
_editScheduleTriggerEl.focus();
}
_editScheduleTriggerEl = null;
}
function submitEditSchedule() {
var taskId = document.getElementById("es-id").value;
var name = (document.getElementById("es-name").value || "").trim();
var message = (document.getElementById("es-message").value || "").trim();
var schedType = document.getElementById("es-type").value;
var cronExpr = (document.getElementById("es-cron").value || "").trim();
var targetMode = document.getElementById("es-target").value;
if (targetMode === "node")
targetMode = (document.getElementById("es-node").value || "").trim();
var atTime = document.getElementById("es-at").value || "";
if (atTime) {
if (atTime.length === 16) atTime += ":00";
else if (atTime.length > 19) atTime = atTime.slice(0, 19);
atTime += "+00:00";
}
var errEl = document.getElementById("edit-schedule-error");
if (!name) return _showModalError(errEl, "Name is required");
if (!message) return _showModalError(errEl, "Initial message is required");
if (schedType === "cron" && !cronExpr)
return _showModalError(errEl, "Cron expression is required");
if (schedType === "at" && !atTime)
return _showModalError(errEl, "Run time is required");
var btn = document.getElementById("es-submit");
btn.disabled = true;
btn.textContent = "Saving\u2026";
authFetch("/v1/api/admin/schedules/" + encodeURIComponent(taskId), {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
name: (document.getElementById("es-name").value || "").trim(),
description: (document.getElementById("es-desc").value || "").trim(),
schedule_type: document.getElementById("es-type").value,
cron_expr: (document.getElementById("es-cron").value || "").trim(),
at_time: atTime,
target_mode: targetMode,
model: (document.getElementById("es-model").value || "").trim(),
initial_message: (
document.getElementById("es-message").value || ""
).trim(),
auto_approve: document.getElementById("es-autoapprove").checked,
enabled: document.getElementById("es-enabled").checked,
}),
})
.then(function (r) {
if (!r.ok)
return r.json().then(function (d) {
throw new Error(d.error || "Failed");
});
return r.json();
})
.then(function () {
hideEditScheduleModal();
showToast("Schedule updated");
loadAdminSchedules();
})
.catch(function (err) {
btn.disabled = false;
btn.textContent = "Save";
_showModalError(errEl, err.message || "Failed to update schedule");
});
}
// --- Schedule Runs Modal ---
function showScheduleRuns(taskId) {
_runsScheduleTriggerEl = document.activeElement;
authFetch(
"/v1/api/admin/schedules/" + encodeURIComponent(taskId) + "/runs?limit=50",
)
.then(function (r) {
if (!r.ok) throw new Error("Not found");
return r.json();
})
.then(function (data) {
var runs = data.runs || [];
var container = document.getElementById("schedule-runs-table");
if (!runs.length) {
container.innerHTML = '<div class="dashboard-empty">No runs yet</div>';
} else {
var html =
'<div class="admin-colheaders sched-runs-grid" aria-hidden="true">' +
'<span class="admin-col">STARTED</span>' +
'<span class="admin-col">NODE</span>' +
'<span class="admin-col">STATUS</span>' +
'<span class="admin-col">ERROR</span></div>';
for (var i = 0; i < runs.length; i++) {
var r = runs[i];
var statusCls =
r.status === "dispatched"
? "sched-active"
: r.status === "failed"
? "sched-expired"
: "";
html +=
'<div class="admin-row sched-runs-grid">' +
'<span class="admin-col">' +
escapeHtml(r.started || "")
.slice(0, 19)
.replace("T", " ") +
"</span>" +
'<span class="admin-col">' +
escapeHtml(r.node_id || "\u2014") +
"</span>" +
'<span class="admin-col"><span class="' +
statusCls +
'">' +
escapeHtml(r.status) +
"</span></span>" +
'<span class="admin-col">' +
escapeHtml(r.error || "\u2014") +
"</span></div>";
}
container.innerHTML = html;
}
var overlay = document.getElementById("schedule-runs-overlay");
overlay.style.display = "flex";
_srTrapHandler = _installTrap(
"schedule-runs-overlay",
"schedule-runs-box",
);
})
.catch(function () {
showToast("Failed to load run history");
});
}
function hideScheduleRunsModal() {
document.getElementById("schedule-runs-overlay").style.display = "none";
_srTrapHandler = _removeTrap(_srTrapHandler);
if (_runsScheduleTriggerEl && _runsScheduleTriggerEl.isConnected) {
_runsScheduleTriggerEl.focus();
}
_runsScheduleTriggerEl = null;
}
// ---------------------------------------------------------------------------
// Create Channel Link Modal
// ---------------------------------------------------------------------------
@@ -644,7 +1158,7 @@ function _modalFocusTrap(boxId) {
var box = document.getElementById(boxId);
if (!box) return;
var focusable = box.querySelectorAll(
"input:not([disabled]), select:not([disabled]), button:not([disabled])",
"input:not([disabled]):not([type='hidden']), select:not([disabled]), textarea:not([disabled]), button:not([disabled])",
);
var visible = [];
for (var i = 0; i < focusable.length; i++) {
@@ -678,6 +1192,10 @@ function _installTrap(overlayId, boxId, trapRef) {
else if (overlayId === "token-created-overlay") hideTokenCreatedModal();
else if (overlayId === "create-channel-overlay")
hideCreateChannelModal();
else if (overlayId === "create-schedule-overlay")
hideCreateScheduleModal();
else if (overlayId === "edit-schedule-overlay") hideEditScheduleModal();
else if (overlayId === "schedule-runs-overlay") hideScheduleRunsModal();
else if (overlayId === "confirm-overlay") hideConfirmModal();
}
};
@@ -721,6 +1239,24 @@ document.addEventListener("keydown", function (e) {
hideCreateChannelModal();
return;
}
var cso = document.getElementById("create-schedule-overlay");
if (cso && cso.style.display !== "none") {
e.preventDefault();
hideCreateScheduleModal();
return;
}
var eso = document.getElementById("edit-schedule-overlay");
if (eso && eso.style.display !== "none") {
e.preventDefault();
hideEditScheduleModal();
return;
}
var sro = document.getElementById("schedule-runs-overlay");
if (sro && sro.style.display !== "none") {
e.preventDefault();
hideScheduleRunsModal();
return;
}
var cf = document.getElementById("confirm-overlay");
if (cf && cf.style.display !== "none") {
e.preventDefault();
@@ -735,7 +1271,7 @@ document.addEventListener("keydown", function (e) {
if (!tablist) return;
tablist.addEventListener("keydown", function (e) {
if (e.key !== "ArrowLeft" && e.key !== "ArrowRight") return;
var tabOrder = ["users", "tokens", "channels"];
var tabOrder = ["users", "tokens", "channels", "schedules"];
var idx = tabOrder.indexOf(_adminTab);
if (e.key === "ArrowRight") idx = (idx + 1) % tabOrder.length;
else idx = (idx - 1 + tabOrder.length) % tabOrder.length;
+125
View File
@@ -80,6 +80,7 @@
<button id="tab-users" class="admin-tab active" data-tab="users" role="tab" aria-selected="true" aria-controls="admin-users" tabindex="0" onclick="switchAdminTab('users')">Users</button>
<button id="tab-tokens" class="admin-tab" data-tab="tokens" role="tab" aria-selected="false" aria-controls="admin-tokens" tabindex="-1" onclick="switchAdminTab('tokens')">Tokens</button>
<button id="tab-channels" class="admin-tab" data-tab="channels" role="tab" aria-selected="false" aria-controls="admin-channels" tabindex="-1" onclick="switchAdminTab('channels')">Channels</button>
<button id="tab-schedules" class="admin-tab" data-tab="schedules" role="tab" aria-selected="false" aria-controls="admin-schedules" tabindex="-1" onclick="switchAdminTab('schedules')">Schedules</button>
</div>
<!-- Users Tab -->
@@ -142,6 +143,26 @@
<div class="dashboard-empty">Select a user to view channel links</div>
</div>
</div>
<!-- Schedules Tab -->
<div id="admin-schedules" class="admin-panel" role="tabpanel" aria-labelledby="tab-schedules" style="display:none">
<div class="admin-toolbar">
<span class="section-header" style="margin:0">SCHEDULED TASKS</span>
<button class="admin-action-btn" onclick="showCreateScheduleModal()">+ New schedule</button>
</div>
<div class="admin-colheaders" aria-hidden="true">
<span class="admin-col admin-col-sname">NAME</span>
<span class="admin-col admin-col-stype">TYPE</span>
<span class="admin-col admin-col-sschedule">SCHEDULE</span>
<span class="admin-col admin-col-starget">TARGET</span>
<span class="admin-col admin-col-snext">NEXT RUN</span>
<span class="admin-col admin-col-sstatus">STATUS</span>
<span class="admin-col admin-col-actions">ACTIONS</span>
</div>
<div id="admin-schedules-table" role="list" aria-label="Scheduled tasks" aria-live="polite">
<div class="dashboard-empty">Loading schedules...</div>
</div>
</div>
</div>
</div>
@@ -284,6 +305,110 @@ window.TURNSTONE_KB_SHORTCUTS = [
</div>
</div>
<!-- Create Schedule Modal -->
<div id="create-schedule-overlay" style="display:none" role="dialog" aria-modal="true" aria-labelledby="create-schedule-title">
<div id="create-schedule-box" class="admin-modal admin-modal-wide">
<h2 id="create-schedule-title">New Schedule</h2>
<div id="create-schedule-error" role="alert" aria-live="assertive"></div>
<label for="cs-name">Name</label>
<input id="cs-name" type="text" placeholder="Daily health check" autocomplete="off">
<label for="cs-desc">Description <span class="label-hint">optional</span></label>
<input id="cs-desc" type="text" placeholder="" autocomplete="off">
<label for="cs-type">Schedule type</label>
<select id="cs-type" onchange="toggleScheduleTypeFields()">
<option value="cron">Cron (recurring)</option>
<option value="at">At (one-shot)</option>
</select>
<div id="cs-cron-group">
<label for="cs-cron">Cron expression</label>
<input id="cs-cron" type="text" placeholder="0 9 * * MON-FRI" autocomplete="off" spellcheck="false" aria-describedby="cs-cron-hint">
<span id="cs-cron-hint" class="label-hint" style="display:block;margin-top:3px">min hour day month weekday</span>
</div>
<div id="cs-at-group" style="display:none">
<label for="cs-at">Run at</label>
<input id="cs-at" type="datetime-local">
</div>
<label for="cs-target">Target</label>
<select id="cs-target" onchange="toggleScheduleNodeField()">
<option value="auto">Auto (best available)</option>
<option value="pool">Pool (any bridge)</option>
<option value="all">All nodes</option>
<option value="node">Specific node...</option>
</select>
<div id="cs-node-group" style="display:none">
<label for="cs-node">Node ID</label>
<input id="cs-node" type="text" placeholder="node-001" autocomplete="off" spellcheck="false">
</div>
<label for="cs-model">Model <span class="label-hint">optional</span></label>
<input id="cs-model" type="text" placeholder="Default model" autocomplete="off">
<label for="cs-message">Initial message</label>
<textarea id="cs-message" rows="3" placeholder="What should the workstream do?"></textarea>
<label class="admin-checkbox"><input id="cs-autoapprove" type="checkbox"> Auto-approve tool calls</label>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideCreateScheduleModal()">Cancel</button>
<button id="cs-submit" class="modal-submit" onclick="submitCreateSchedule()">Create</button>
</div>
</div>
</div>
<!-- Edit Schedule Modal -->
<div id="edit-schedule-overlay" style="display:none" role="dialog" aria-modal="true" aria-labelledby="edit-schedule-title">
<div id="edit-schedule-box" class="admin-modal admin-modal-wide">
<h2 id="edit-schedule-title">Edit Schedule</h2>
<div id="edit-schedule-error" role="alert" aria-live="assertive"></div>
<input id="es-id" type="hidden">
<label for="es-name">Name</label>
<input id="es-name" type="text" autocomplete="off">
<label for="es-desc">Description</label>
<input id="es-desc" type="text" autocomplete="off">
<label for="es-type">Schedule type</label>
<select id="es-type" onchange="toggleEditScheduleTypeFields()">
<option value="cron">Cron (recurring)</option>
<option value="at">At (one-shot)</option>
</select>
<div id="es-cron-group">
<label for="es-cron">Cron expression</label>
<input id="es-cron" type="text" autocomplete="off" spellcheck="false">
</div>
<div id="es-at-group" style="display:none">
<label for="es-at">Run at</label>
<input id="es-at" type="datetime-local">
</div>
<label for="es-target">Target</label>
<select id="es-target" onchange="toggleEditScheduleNodeField()">
<option value="auto">Auto (best available)</option>
<option value="pool">Pool (any bridge)</option>
<option value="all">All nodes</option>
<option value="node">Specific node...</option>
</select>
<div id="es-node-group" style="display:none">
<label for="es-node">Node ID</label>
<input id="es-node" type="text" autocomplete="off" spellcheck="false">
</div>
<label for="es-model">Model</label>
<input id="es-model" type="text" autocomplete="off">
<label for="es-message">Initial message</label>
<textarea id="es-message" rows="3"></textarea>
<label class="admin-checkbox"><input id="es-autoapprove" type="checkbox"> Auto-approve tool calls</label>
<label class="admin-checkbox"><input id="es-enabled" type="checkbox"> Enabled</label>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideEditScheduleModal()">Cancel</button>
<button id="es-submit" class="modal-submit" onclick="submitEditSchedule()">Save</button>
</div>
</div>
</div>
<!-- Schedule Runs Modal -->
<div id="schedule-runs-overlay" style="display:none" role="dialog" aria-modal="true" aria-labelledby="schedule-runs-title">
<div id="schedule-runs-box" class="admin-modal admin-modal-wide">
<h2 id="schedule-runs-title">Run History</h2>
<div id="schedule-runs-table"></div>
<div class="modal-buttons">
<button class="modal-cancel" onclick="hideScheduleRunsModal()">Close</button>
</div>
</div>
</div>
<script src="/static/admin.js"></script>
<script src="/static/app.js"></script>
</body>
+63 -5
View File
@@ -832,6 +832,58 @@
.admin-btn-danger:hover { opacity: 1; background: rgba(248, 113, 113, 0.1); }
.admin-btn-danger:focus-visible { outline: 2px solid var(--red); outline-offset: 2px; }
.admin-btn-action {
background: none;
border: 1px solid var(--border-strong);
color: var(--fg-dim);
font-family: var(--font-display);
font-size: 10px;
font-weight: 500;
padding: 2px 8px;
border-radius: var(--radius-sm);
cursor: pointer;
opacity: 0.8;
transition: opacity 0.15s, background 0.15s;
margin-right: 4px;
}
.admin-btn-action:hover { opacity: 1; background: rgba(255, 255, 255, 0.05); color: var(--fg); }
.admin-btn-action:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
/* Schedules grid: NAME | TYPE | SCHEDULE | TARGET | NEXT RUN | STATUS | ACTIONS */
#admin-schedules .admin-colheaders,
#admin-schedules .admin-row {
grid-template-columns: 1.5fr 60px 1.2fr 80px 130px 70px 170px;
}
/* Schedule runs grid: STARTED | NODE | STATUS | ERROR */
.sched-runs-grid { grid-template-columns: 2fr 1fr 1fr 2fr; }
/* Schedule status indicators */
.sched-active { color: var(--green); font-weight: 500; }
.sched-disabled { color: var(--fg-dim); }
.sched-expired { color: var(--accent); }
/* Wide modal variant for schedule forms */
.admin-modal-wide { width: 480px; }
/* Checkbox labels inside admin modals */
.admin-modal label.admin-checkbox {
display: flex;
align-items: center;
gap: 8px;
font-size: 12px;
font-weight: 500;
text-transform: none;
letter-spacing: 0;
color: var(--fg);
cursor: pointer;
margin-top: 14px;
}
.admin-modal label.admin-checkbox input[type="checkbox"] {
width: auto;
margin: 0;
}
/* Admin modals (reuse new-ws-overlay pattern) */
.admin-modal {
background: var(--bg-surface);
@@ -871,7 +923,7 @@
margin-top: 12px;
}
.admin-modal label:first-of-type { margin-top: 0; }
.admin-modal input, .admin-modal select {
.admin-modal input:not([type="hidden"]), .admin-modal select, .admin-modal textarea {
width: 100%;
padding: 9px 12px;
background: var(--bg);
@@ -882,12 +934,13 @@
font-size: 13px;
transition: border-color 0.15s, box-shadow 0.15s;
}
.admin-modal input:focus, .admin-modal select:focus {
.admin-modal input:focus, .admin-modal select:focus, .admin-modal textarea:focus {
border-color: var(--accent);
outline: none;
box-shadow: 0 0 0 3px var(--accent-dim);
}
.admin-modal input::placeholder { color: var(--fg-dim); opacity: 0.6; }
.admin-modal input::placeholder, .admin-modal textarea::placeholder { color: var(--fg-dim); opacity: 0.6; }
.admin-modal textarea { resize: vertical; min-height: 40px; }
.admin-modal [role="alert"] { color: var(--red); font-size: 12px; margin-bottom: 8px; display: none; }
.modal-buttons { display: flex; gap: 10px; margin-top: 20px; }
@@ -925,7 +978,8 @@
.modal-submit:focus-visible { outline: 2px solid var(--fg-bright); outline-offset: 2px; }
.modal-submit:disabled { opacity: 0.4; cursor: not-allowed; filter: none; }
#create-user-overlay, #create-token-overlay, #token-created-overlay, #create-channel-overlay, #confirm-overlay {
#create-user-overlay, #create-token-overlay, #token-created-overlay, #create-channel-overlay, #confirm-overlay,
#create-schedule-overlay, #edit-schedule-overlay, #schedule-runs-overlay {
position: fixed;
inset: 0;
background: rgba(0, 0, 0, 0.7);
@@ -969,6 +1023,10 @@
grid-template-columns: 80px 1fr 80px;
}
#admin-channels .admin-col-created { display: none; }
#admin-schedules .admin-colheaders, #admin-schedules .admin-row {
grid-template-columns: 1fr 60px 80px 130px;
}
.admin-col-sschedule, .admin-col-starget, .admin-col-snext { display: none; }
}
/* ==========================================================================
@@ -981,7 +1039,7 @@
.node-link, .dash-cell-node, .pagination button { transition: none; }
.dash-row.has-link::after, .node-group-header::before { transition: none; }
#new-ws-box select, #new-ws-box input, #new-ws-buttons button { transition: none; }
.admin-tab, .admin-row, .admin-btn-danger { transition: none; }
.admin-tab, .admin-row, .admin-btn-danger, .admin-btn-action { transition: none; }
.admin-action-btn, .modal-cancel, .modal-submit { transition: none; }
.admin-modal input, .admin-modal select { transition: none; }
}
+1
View File
@@ -49,6 +49,7 @@ TOKEN_BYTES = 32 # 64 hex chars after prefix
JWT_ISSUER = "turnstone"
JWT_AUD_SERVER = "turnstone-server"
JWT_AUD_CONSOLE = "turnstone-console"
JWT_AUD_CHANNEL = "turnstone-channel"
_MIN_SECRET_LENGTH = 32 # 256 bits minimum for HMAC-SHA256
VALID_SCOPES: frozenset[str] = frozenset({"read", "write", "approve"})
+15 -1
View File
@@ -61,7 +61,7 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
},
"session": {
"instructions": "instructions",
"retention_days": "session_retention_days",
"retention_days": "retention_days",
"compact_max_tokens": "compact_max_tokens",
"auto_compact_pct": "auto_compact_pct",
},
@@ -70,6 +70,9 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
"truncation": "tool_truncation",
"agent_max_turns": "agent_max_turns",
"skip_permissions": "skip_permissions",
"search": "tool_search",
"search_threshold": "tool_search_threshold",
"search_max_results": "tool_search_max_results",
},
"server": {
"host": "host",
@@ -102,6 +105,7 @@ _CONFIG_MAP: dict[str, dict[str, str]] = {
},
"mcp": {
"config_path": "mcp_config",
"refresh_interval": "mcp_refresh_interval",
},
"ratelimit": {
"enabled": "ratelimit_enabled",
@@ -150,6 +154,16 @@ def get_tavily_key() -> str | None:
return _tavily_key
def nonneg_float(val: str) -> float:
"""Argparse type for non-negative floats (``>= 0``)."""
f = float(val)
if f < 0:
import argparse
raise argparse.ArgumentTypeError("must be >= 0")
return f
def apply_config(parser: argparse.ArgumentParser, sections: list[str]) -> None:
"""Set argparse defaults from config file.
+223 -10
View File
@@ -7,19 +7,33 @@ Architecture: the MCP SDK is fully async, but turnstone's ChatSession is
synchronous. We bridge the two by running a dedicated asyncio event loop
in a daemon thread. ``call_tool_sync`` dispatches coroutines onto that loop
via ``asyncio.run_coroutine_threadsafe``.
Tool refresh: three mechanisms keep tool lists up-to-date without restart:
1. Push notifications servers declaring ``tools.listChanged`` trigger
immediate refresh via ``ToolListChangedNotification``.
2. Periodic timer servers *without* push support are polled on a
staggered interval (configurable, default 4 h, seeded at launch).
3. Manual ``/mcp refresh [server]`` triggers ``refresh_sync()``.
"""
from __future__ import annotations
import asyncio
import contextlib
import json
import logging
import os
import random
import threading
import time
from contextlib import AsyncExitStack
from pathlib import Path
from typing import Any
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from collections.abc import Callable
import mcp.types as mcp_types
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from mcp.client.streamable_http import streamablehttp_client
@@ -28,6 +42,8 @@ from turnstone.core.config import load_config
log = logging.getLogger("turnstone.mcp")
_DEFAULT_REFRESH_INTERVAL: float = 14400 # 4 hours
# ---------------------------------------------------------------------------
# MCP ↔ OpenAI schema conversion
@@ -67,8 +83,15 @@ class MCPClientManager:
synchronous methods for tool discovery and invocation.
"""
def __init__(self, server_configs: dict[str, dict[str, Any]]) -> None:
def __init__(
self,
server_configs: dict[str, dict[str, Any]],
*,
refresh_interval: float = _DEFAULT_REFRESH_INTERVAL,
) -> None:
self._server_configs = server_configs
if refresh_interval < 0:
refresh_interval = 0.0
self._loop: asyncio.AbstractEventLoop | None = None
self._thread: threading.Thread | None = None
self._exit_stack: AsyncExitStack | None = None
@@ -80,6 +103,19 @@ class MCPClientManager:
self._connected = threading.Event()
self._error: str | None = None
# Per-server tool storage for surgical refresh
self._per_server_tools: dict[str, list[dict[str, Any]]] = {}
# Tracks which servers support push notifications
self._supports_list_changed: dict[str, bool] = {}
# Listener infrastructure (tool-change callbacks for ChatSession)
self._listeners: list[Callable[[], None]] = []
self._listeners_lock = threading.Lock()
# Periodic refresh for servers without push notifications
self._refresh_interval = refresh_interval
self._refresh_task: asyncio.Task[None] | None = None
# -- lifecycle -----------------------------------------------------------
def start(self) -> None:
@@ -108,6 +144,13 @@ class MCPClientManager:
self._connected.set()
# Start periodic refresh for servers without push notifications
needs_periodic = any(
not self._supports_list_changed.get(name, False) for name in self._sessions
)
if needs_periodic and self._refresh_interval > 0:
self._refresh_task = asyncio.get_running_loop().create_task(self._periodic_refresh())
async def _connect_one(self, name: str, cfg: dict[str, Any]) -> None:
"""Connect to a single MCP server and discover its tools."""
assert self._exit_stack is not None
@@ -135,26 +178,187 @@ class MCPClientManager:
)
read, write = await self._exit_stack.enter_async_context(stdio_client(params))
session = await self._exit_stack.enter_async_context(ClientSession(read, write))
# Register notification handler — lightweight; only acts on
# ToolListChangedNotification, which is a no-op if the server
# never sends it.
async def _on_notification(
msg: Any, # RequestResponder | ServerNotification | Exception
) -> None:
if isinstance(msg, mcp_types.ServerNotification) and isinstance(
msg.root, mcp_types.ToolListChangedNotification
):
log.info("Received tools/list_changed from '%s'", name)
try:
await self._refresh_server(name)
except Exception:
log.warning("Refresh after notification failed for '%s'", name, exc_info=True)
session = await self._exit_stack.enter_async_context(
ClientSession(read, write, message_handler=_on_notification) # type: ignore[arg-type]
)
await session.initialize()
self._sessions[name] = session
# Check push notification support
caps = session.get_server_capabilities()
tools_cap = getattr(caps, "tools", None) if caps else None
self._supports_list_changed[name] = bool(getattr(tools_cap, "listChanged", False))
# Discover tools
result = await session.list_tools()
server_tools: list[dict[str, Any]] = []
for tool in result.tools:
openai_def = _mcp_to_openai(name, tool)
prefixed = openai_def["function"]["name"]
self._tools.append(openai_def)
self._tool_map[prefixed] = (name, tool.name)
server_tools.append(_mcp_to_openai(name, tool))
self._per_server_tools[name] = server_tools
self._rebuild_tools()
push_status = " (push)" if self._supports_list_changed[name] else ""
log.info(
"Connected MCP server '%s'%d tool(s)",
"Connected MCP server '%s'%d tool(s)%s",
name,
len(result.tools),
push_status,
)
# -- tool refresh --------------------------------------------------------
def _rebuild_tools(self) -> None:
"""Rebuild merged ``_tools`` and ``_tool_map`` from per-server state.
Uses copy-on-write: builds new objects, then assigns atomically.
Concurrent readers see either the old or new snapshot both valid.
"""
new_tools: list[dict[str, Any]] = []
new_map: dict[str, tuple[str, str]] = {}
for srv_name, srv_tools in self._per_server_tools.items():
for tool in srv_tools:
prefixed: str = tool["function"]["name"]
new_tools.append(tool)
# Extract original name from the mcp__server__original pattern
original = prefixed.split("__", 2)[2] if prefixed.count("__") >= 2 else prefixed
new_map[prefixed] = (srv_name, original)
self._tools = new_tools
self._tool_map = new_map
self._notify_listeners()
async def _refresh_server(self, name: str) -> tuple[list[str], list[str]]:
"""Re-fetch tools for one server. Returns ``(added, removed)`` names."""
session = self._sessions.get(name)
if session is None:
raise RuntimeError(f"MCP server '{name}' is not connected")
old_names = {t["function"]["name"] for t in self._per_server_tools.get(name, [])}
result = await session.list_tools()
server_tools = [_mcp_to_openai(name, tool) for tool in result.tools]
new_names = {t["function"]["name"] for t in server_tools}
self._per_server_tools[name] = server_tools
self._rebuild_tools()
added = sorted(new_names - old_names)
removed = sorted(old_names - new_names)
if added or removed:
log.info(
"Refreshed MCP server '%s': +%d/-%d tool(s)",
name,
len(added),
len(removed),
)
return added, removed
async def _refresh_all(
self, server_name: str | None = None
) -> dict[str, tuple[list[str], list[str]]]:
"""Refresh tools for one or all servers.
For disconnected servers (in config but not connected), attempts
reconnect. Returns ``{server: (added, removed)}`` per server.
"""
results: dict[str, tuple[list[str], list[str]]] = {}
targets = [server_name] if server_name else list(self._server_configs.keys())
for name in targets:
try:
if name not in self._sessions:
# Attempt reconnect
cfg = self._server_configs.get(name)
if cfg:
log.info("Reconnecting MCP server '%s'", name)
await self._connect_one(name, cfg)
new_names = [
t["function"]["name"] for t in self._per_server_tools.get(name, [])
]
results[name] = (new_names, [])
continue
added, removed = await self._refresh_server(name)
results[name] = (added, removed)
except Exception:
log.warning("Refresh failed for MCP server '%s'", name, exc_info=True)
results[name] = ([], [])
return results
def refresh_sync(
self, server_name: str | None = None, timeout: int = 30
) -> dict[str, tuple[list[str], list[str]]]:
"""Refresh tools synchronously (blocks the calling thread).
Returns ``{server: (added_names, removed_names)}`` per server.
"""
assert self._loop is not None
future = asyncio.run_coroutine_threadsafe(self._refresh_all(server_name), self._loop)
return future.result(timeout=timeout)
async def _periodic_refresh(self) -> None:
"""Periodically refresh servers that lack push notifications."""
# Stagger start using a launch-time seed so cluster nodes don't
# all hit MCP servers simultaneously.
seed = random.Random(time.monotonic_ns() ^ os.getpid()).random()
initial_delay = seed * self._refresh_interval
await asyncio.sleep(initial_delay)
while True:
for name in list(self._server_configs):
if self._supports_list_changed.get(name, False):
continue # has push — skip
if name not in self._sessions:
continue # not connected — skip (reconnect on manual refresh)
try:
await self._refresh_server(name)
except Exception:
log.warning("Periodic refresh failed for '%s'", name, exc_info=True)
await asyncio.sleep(self._refresh_interval)
# -- listener infrastructure ---------------------------------------------
def add_listener(self, callback: Callable[[], None]) -> None:
"""Register a callback invoked when the tool list changes."""
with self._listeners_lock:
self._listeners.append(callback)
def remove_listener(self, callback: Callable[[], None]) -> None:
"""Unregister a tool-change callback."""
with self._listeners_lock, contextlib.suppress(ValueError):
self._listeners.remove(callback)
def _notify_listeners(self) -> None:
"""Invoke all registered listeners (runs on MCP background thread)."""
with self._listeners_lock:
listeners = list(self._listeners)
for cb in listeners:
try:
cb()
except Exception:
log.warning("Tool-change listener raised", exc_info=True)
# -- lifecycle (shutdown) ------------------------------------------------
def shutdown(self) -> None:
"""Close all MCP sessions and stop the background loop."""
# Cancel periodic refresh
if self._refresh_task and self._loop:
self._loop.call_soon_threadsafe(self._refresh_task.cancel)
if self._loop and self._exit_stack:
future = asyncio.run_coroutine_threadsafe(self._exit_stack.aclose(), self._loop)
try:
@@ -183,6 +387,11 @@ class MCPClientManager:
def server_count(self) -> int:
return len(self._sessions)
@property
def server_names(self) -> list[str]:
"""Return configured server names."""
return list(self._server_configs.keys())
# -- tool invocation -----------------------------------------------------
def call_tool_sync(
@@ -273,7 +482,11 @@ def load_mcp_config(config_path: str | None = None) -> dict[str, dict[str, Any]]
return {}
def create_mcp_client(config_path: str | None = None) -> MCPClientManager | None:
def create_mcp_client(
config_path: str | None = None,
*,
refresh_interval: float = _DEFAULT_REFRESH_INTERVAL,
) -> MCPClientManager | None:
"""Create and start an MCP client manager.
Returns *None* if no servers are configured.
@@ -282,6 +495,6 @@ def create_mcp_client(config_path: str | None = None) -> MCPClientManager | None
if not servers:
return None
mgr = MCPClientManager(servers)
mgr = MCPClientManager(servers, refresh_interval=refresh_interval)
mgr.start()
return mgr
+96 -110
View File
@@ -21,22 +21,11 @@ def normalize_key(key: str) -> str:
return key.lower().replace("-", "_").replace(" ", "_")
# -- Core session operations ---------------------------------------------------
def register_session(
session_id: str,
title: str | None = None,
node_id: str | None = None,
ws_id: str | None = None,
) -> None:
"""Create a sessions row for a new session (no-op if already exists)."""
with contextlib.suppress(Exception):
get_storage().register_session(session_id, title, node_id=node_id, ws_id=ws_id)
# -- Core conversation operations ---------------------------------------------
def save_message(
session_id: str,
ws_id: str,
role: str,
content: str | None,
tool_name: str | None = None,
@@ -47,111 +36,19 @@ def save_message(
"""Log a message to the conversations table."""
with contextlib.suppress(Exception):
get_storage().save_message(
session_id, role, content, tool_name, tool_args, tool_call_id, provider_data
ws_id, role, content, tool_name, tool_args, tool_call_id, provider_data
)
def load_session_messages(session_id: str) -> list[dict[str, Any]]:
"""Load messages for a session and reconstruct OpenAI message format."""
def load_messages(ws_id: str) -> list[dict[str, Any]]:
"""Load messages for a workstream and reconstruct OpenAI message format."""
try:
return get_storage().load_session_messages(session_id)
return get_storage().load_messages(ws_id)
except Exception:
return []
# -- Session management --------------------------------------------------------
def list_sessions(limit: int = 20) -> list[Any]:
"""List recent sessions with message counts."""
try:
return get_storage().list_sessions(limit)
except Exception:
return []
def delete_session(session_id: str) -> bool:
"""Delete a session and all its messages."""
try:
return get_storage().delete_session(session_id)
except Exception:
return False
def prune_sessions(
retention_days: int = 90,
log_fn: Callable[[str], None] | None = None,
) -> tuple[int, int]:
"""Prune orphaned and stale sessions."""
try:
orphans, stale = get_storage().prune_sessions(retention_days)
except Exception:
return (0, 0)
if log_fn and (orphans or stale):
parts = []
if orphans:
parts.append(f"{orphans} empty session{'s' if orphans != 1 else ''}")
if stale:
parts.append(
f"{stale} session{'s' if stale != 1 else ''} older than {retention_days} days"
)
log_fn(f"[turnstone] Session cleanup: removed {', '.join(parts)}.")
return (orphans, stale)
def resolve_session(alias_or_id: str) -> str | None:
"""Resolve an alias or session_id (or prefix) to a full session_id."""
try:
return get_storage().resolve_session(alias_or_id)
except Exception:
return None
# -- Session config ------------------------------------------------------------
def save_session_config(session_id: str, config: dict[str, str]) -> None:
"""Persist session configuration key/value pairs."""
with contextlib.suppress(Exception):
get_storage().save_session_config(session_id, config)
def load_session_config(session_id: str) -> dict[str, str]:
"""Load session configuration."""
try:
return get_storage().load_session_config(session_id)
except Exception:
return {}
# -- Session metadata ----------------------------------------------------------
def set_session_alias(session_id: str, alias: str) -> bool:
"""Set a human-friendly alias. Returns False if alias is taken."""
try:
return get_storage().set_session_alias(session_id, alias)
except Exception:
return False
def get_session_name(session_id: str) -> str | None:
"""Return the alias (or title) for a session, or None if unset."""
try:
return get_storage().get_session_name(session_id)
except Exception:
return None
def update_session_title(session_id: str, title: str) -> None:
"""Set or update the auto-generated title for a session."""
with contextlib.suppress(Exception):
get_storage().update_session_title(session_id, title)
# -- Workstream operations -----------------------------------------------------
# -- Workstream management ----------------------------------------------------
def register_workstream(
@@ -182,6 +79,95 @@ def list_workstreams(node_id: str | None = None, limit: int = 100) -> list[Any]:
return []
def list_workstreams_with_history(limit: int = 20) -> list[Any]:
"""List workstreams that have conversation messages."""
try:
return get_storage().list_workstreams_with_history(limit)
except Exception:
return []
def delete_workstream(ws_id: str) -> bool:
"""Delete a workstream and all its conversations + config."""
try:
return get_storage().delete_workstream(ws_id)
except Exception:
return False
def prune_workstreams(
retention_days: int = 90,
log_fn: Callable[[str], None] | None = None,
) -> tuple[int, int]:
"""Prune orphaned and stale workstreams."""
try:
orphans, stale = get_storage().prune_workstreams(retention_days)
except Exception:
return (0, 0)
if log_fn and (orphans or stale):
parts = []
if orphans:
parts.append(f"{orphans} empty workstream{'s' if orphans != 1 else ''}")
if stale:
parts.append(
f"{stale} workstream{'s' if stale != 1 else ''} older than {retention_days} days"
)
log_fn(f"[turnstone] Cleanup: removed {', '.join(parts)}.")
return (orphans, stale)
def resolve_workstream(alias_or_id: str) -> str | None:
"""Resolve an alias or ws_id (or prefix) to a full ws_id."""
try:
return get_storage().resolve_workstream(alias_or_id)
except Exception:
return None
# -- Workstream config --------------------------------------------------------
def save_workstream_config(ws_id: str, config: dict[str, str]) -> None:
"""Persist workstream configuration key/value pairs."""
with contextlib.suppress(Exception):
get_storage().save_workstream_config(ws_id, config)
def load_workstream_config(ws_id: str) -> dict[str, str]:
"""Load workstream configuration."""
try:
return get_storage().load_workstream_config(ws_id)
except Exception:
return {}
# -- Workstream metadata ------------------------------------------------------
def set_workstream_alias(ws_id: str, alias: str) -> bool:
"""Set a human-friendly alias. Returns False if alias is taken."""
try:
return get_storage().set_workstream_alias(ws_id, alias)
except Exception:
return False
def get_workstream_display_name(ws_id: str) -> str | None:
"""Return the alias (or title) for a workstream, or None if unset."""
try:
return get_storage().get_workstream_display_name(ws_id)
except Exception:
return None
def update_workstream_title(ws_id: str, title: str) -> None:
"""Set or update the auto-generated title for a workstream."""
with contextlib.suppress(Exception):
get_storage().update_workstream_title(ws_id, title)
# -- Key-value store (memories) ------------------------------------------------
+1 -7
View File
@@ -259,19 +259,13 @@ class MetricsCollector:
# Per-workstream metrics (only when data is provided)
if workstream_metrics:
# turnstone_workstream_info — exposes session_id as a label for joining,
# without propagating that high-cardinality label to counters.
lines.append(
"# HELP turnstone_workstream_info Workstream metadata"
" (join on session_id for per-session queries)"
)
lines.append("# HELP turnstone_workstream_info Workstream metadata")
lines.append("# TYPE turnstone_workstream_info gauge")
for wm in workstream_metrics:
lstr = _fmt_labels(
{
"ws_id": wm["ws_id"],
"name": wm["name"],
"session_id": wm["session_id"],
}
)
lines.append(f"turnstone_workstream_info{lstr} 1")
+38
View File
@@ -64,6 +64,9 @@ def _merge_consecutive(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
# Tool version for Anthropic's server-side web search (update when new version ships)
_WEB_SEARCH_TOOL_TYPE = "web_search_20250305"
# Tool search: server-side BM25 tool discovery for deferred tools
_TOOL_SEARCH_TOOL_TYPE = "tool_search_tool_bm25_20251119"
# -- model capabilities -------------------------------------------------------
_ANTHROPIC_DEFAULT = ModelCapabilities(
@@ -83,6 +86,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
supports_effort=True,
effort_levels=("low", "medium", "high", "max"),
supports_web_search=True,
supports_tool_search=True,
),
"claude-sonnet-4-6": ModelCapabilities(
context_window=200000,
@@ -92,6 +96,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
supports_effort=True,
effort_levels=("low", "medium", "high"),
supports_web_search=True,
supports_tool_search=True,
),
"claude-haiku-4-5": ModelCapabilities(
context_window=200000,
@@ -122,6 +127,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
token_param="max_tokens",
thinking_mode="manual",
supports_web_search=True,
supports_tool_search=True,
),
"claude-sonnet-4": ModelCapabilities(
context_window=200000,
@@ -129,6 +135,7 @@ _ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
token_param="max_tokens",
thinking_mode="manual",
supports_web_search=True,
supports_tool_search=True,
),
}
@@ -182,6 +189,31 @@ class AnthropicProvider:
filtered.append({"type": _WEB_SEARCH_TOOL_TYPE, "name": "web_search"})
return filtered
# -- tool search injection -----------------------------------------------
def _inject_tool_search(
self,
tools: list[dict[str, Any]],
caps: ModelCapabilities,
deferred_names: frozenset[str] | None = None,
) -> list[dict[str, Any]]:
"""Mark deferred tools and add native server-side search tool.
When the model supports tool search and ``deferred_names`` is provided,
tools whose name is in the deferred set get ``defer_loading: true``.
The BM25 search tool is appended so the model can discover them.
"""
if not caps.supports_tool_search or not deferred_names:
return tools
result = []
for tool in tools:
if tool.get("name", "") in deferred_names:
result.append({**tool, "defer_loading": True})
else:
result.append(tool)
result.append({"type": _TOOL_SEARCH_TOOL_TYPE, "name": "tool_search"})
return result
# -- shared param logic --------------------------------------------------
def _build_thinking_and_kwargs(
@@ -195,6 +227,7 @@ class AnthropicProvider:
system_prompt: str,
model: str,
tools: list[dict[str, Any]] | None,
deferred_names: frozenset[str] | None = None,
) -> dict[str, Any]:
"""Build the full kwargs dict with thinking mode and effort params."""
thinking_params: dict[str, Any] = {}
@@ -217,6 +250,7 @@ class AnthropicProvider:
if tools:
anthropic_tools = self.convert_tools(tools)
anthropic_tools = self._inject_web_search(anthropic_tools, caps)
anthropic_tools = self._inject_tool_search(anthropic_tools, caps, deferred_names)
kwargs["tools"] = anthropic_tools
kwargs.update(thinking_params)
@@ -371,6 +405,7 @@ class AnthropicProvider:
temperature: float = 0.5,
reasoning_effort: str = "medium",
extra_params: dict[str, Any] | None = None,
deferred_names: frozenset[str] | None = None,
) -> Iterator[StreamChunk]:
_ensure_anthropic()
caps = self.get_capabilities(model)
@@ -385,6 +420,7 @@ class AnthropicProvider:
system_prompt,
model,
tools,
deferred_names,
)
with client.messages.stream(**kwargs) as stream:
@@ -536,6 +572,7 @@ class AnthropicProvider:
temperature: float = 0.5,
reasoning_effort: str = "medium",
extra_params: dict[str, Any] | None = None,
deferred_names: frozenset[str] | None = None,
) -> CompletionResult:
_ensure_anthropic()
caps = self.get_capabilities(model)
@@ -550,6 +587,7 @@ class AnthropicProvider:
system_prompt,
model,
tools,
deferred_names,
)
response = client.messages.create(**kwargs)
+73 -1
View File
@@ -45,6 +45,14 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
reasoning_effort_values=("minimal", "low", "medium", "high"),
default_reasoning_effort="medium",
),
# GPT-5 pro — high reasoning only, extended output
"gpt-5-pro": ModelCapabilities(
context_window=400000,
max_output_tokens=272000,
supports_temperature=False,
reasoning_effort_values=("high",),
default_reasoning_effort="high",
),
# GPT-5.1 — temperature OK when reasoning_effort=none (default)
"gpt-5.1": ModelCapabilities(
context_window=400000,
@@ -59,6 +67,38 @@ _OPENAI_CAPABILITIES: dict[str, ModelCapabilities] = {
reasoning_effort_values=("none", "low", "medium", "high", "xhigh"),
default_reasoning_effort="none",
),
# GPT-5.2 pro — always-reasoning variant
"gpt-5.2-pro": ModelCapabilities(
context_window=400000,
max_output_tokens=128000,
supports_temperature=False,
reasoning_effort_values=("medium", "high", "xhigh"),
default_reasoning_effort="medium",
),
# GPT-5.3 — same capabilities as 5.2 (matches gpt-5.3-chat-latest, codex)
"gpt-5.3": ModelCapabilities(
context_window=400000,
max_output_tokens=128000,
reasoning_effort_values=("none", "low", "medium", "high", "xhigh"),
default_reasoning_effort="none",
),
# GPT-5.4 — 1M context window, native tool search
"gpt-5.4": ModelCapabilities(
context_window=1050000,
max_output_tokens=128000,
reasoning_effort_values=("none", "low", "medium", "high", "xhigh"),
default_reasoning_effort="none",
supports_tool_search=True,
),
# GPT-5.4 pro — always-reasoning, 1M context, native tool search
"gpt-5.4-pro": ModelCapabilities(
context_window=1050000,
max_output_tokens=128000,
supports_temperature=False,
reasoning_effort_values=("medium", "high", "xhigh"),
default_reasoning_effort="medium",
supports_tool_search=True,
),
# O-series reasoning models
"o1": ModelCapabilities(
context_window=200000,
@@ -145,7 +185,11 @@ class OpenAIProvider:
else:
kwargs["temperature"] = temperature
if caps.reasoning_effort_values and reasoning_effort and reasoning_effort != "none":
kwargs["reasoning_effort"] = reasoning_effort
# Validate against supported values; fall back to model default
if reasoning_effort in caps.reasoning_effort_values:
kwargs["reasoning_effort"] = reasoning_effort
elif caps.default_reasoning_effort and caps.default_reasoning_effort != "none":
kwargs["reasoning_effort"] = caps.default_reasoning_effort
# -- web search ----------------------------------------------------------
@@ -173,6 +217,30 @@ class OpenAIProvider:
kwargs["web_search_options"] = {}
return tools
# -- tool search ---------------------------------------------------------
def _apply_tool_search(
self,
caps: ModelCapabilities,
tools: list[dict[str, Any]] | None,
deferred_names: frozenset[str] | None = None,
) -> list[dict[str, Any]] | None:
"""Mark deferred tools with ``defer_loading: true`` for native search.
For GPT-5.4+ models that support tool search, OpenAI's API handles
discovery automatically no explicit search tool is needed.
"""
if not caps.supports_tool_search or not deferred_names or not tools:
return tools
result = []
for tool in tools:
name = tool.get("function", {}).get("name", "")
if name in deferred_names:
result.append({**tool, "defer_loading": True})
else:
result.append(tool)
return result
# -- streaming -----------------------------------------------------------
def create_streaming(
@@ -186,6 +254,7 @@ class OpenAIProvider:
temperature: float = 0.5,
reasoning_effort: str = "medium",
extra_params: dict[str, Any] | None = None,
deferred_names: frozenset[str] | None = None,
) -> Iterator[StreamChunk]:
caps = self.get_capabilities(model)
kwargs: dict[str, Any] = {
@@ -197,6 +266,7 @@ class OpenAIProvider:
}
self._apply_model_params(kwargs, caps, temperature, reasoning_effort)
tools = self._apply_web_search(kwargs, caps, tools)
tools = self._apply_tool_search(caps, tools, deferred_names)
if tools:
kwargs["tools"] = tools
if extra_params:
@@ -290,6 +360,7 @@ class OpenAIProvider:
temperature: float = 0.5,
reasoning_effort: str = "medium",
extra_params: dict[str, Any] | None = None,
deferred_names: frozenset[str] | None = None,
) -> CompletionResult:
caps = self.get_capabilities(model)
kwargs: dict[str, Any] = {
@@ -300,6 +371,7 @@ class OpenAIProvider:
}
self._apply_model_params(kwargs, caps, temperature, reasoning_effort)
tools = self._apply_web_search(kwargs, caps, tools)
tools = self._apply_tool_search(caps, tools, deferred_names)
if tools:
kwargs["tools"] = tools
if extra_params:
+3
View File
@@ -76,6 +76,7 @@ class ModelCapabilities:
reasoning_effort_values: tuple[str, ...] = ()
default_reasoning_effort: str = "medium"
supports_web_search: bool = False
supports_tool_search: bool = False
def _lookup_capabilities(
@@ -119,6 +120,7 @@ class LLMProvider(Protocol):
temperature: float = 0.5,
reasoning_effort: str = "medium",
extra_params: dict[str, Any] | None = None,
deferred_names: frozenset[str] | None = None,
) -> Iterator[StreamChunk]:
"""Create a streaming request, yielding normalized StreamChunks."""
...
@@ -134,6 +136,7 @@ class LLMProvider(Protocol):
temperature: float = 0.5,
reasoning_effort: str = "medium",
extra_params: dict[str, Any] | None = None,
deferred_names: frozenset[str] | None = None,
) -> CompletionResult:
"""Create a non-streaming request, returning a normalized result."""
...
+508 -78
View File
@@ -26,32 +26,35 @@ import httpx
from turnstone.core.config import get_tavily_key
from turnstone.core.edit import find_occurrences, pick_nearest
from turnstone.core.log import get_logger
from turnstone.core.memory import (
delete_memory,
delete_session,
get_session_name,
list_sessions,
delete_workstream,
get_workstream_display_name,
list_workstreams_with_history,
load_memories,
load_session_config,
load_session_messages,
load_messages,
load_workstream_config,
normalize_key,
register_session,
resolve_session,
resolve_workstream,
save_memory,
save_message,
save_session_config,
save_workstream_config,
search_history,
search_history_recent,
search_memories,
set_session_alias,
update_session_title,
set_workstream_alias,
update_workstream_title,
)
from turnstone.core.providers import create_provider
from turnstone.core.safety import is_command_blocked, sanitize_command
from turnstone.core.sandbox import execute_math_sandboxed
from turnstone.core.storage._registry import get_storage
from turnstone.core.tool_search import ToolSearchManager
from turnstone.core.tools import (
AGENT_AUTO_TOOLS,
AGENT_TOOLS,
BUILTIN_TOOL_NAMES,
PRIMARY_KEY_MAP,
TASK_AGENT_TOOLS,
TASK_AUTO_TOOLS,
@@ -61,6 +64,8 @@ from turnstone.core.tools import (
from turnstone.core.web import check_ssrf, strip_html
from turnstone.ui.colors import DIM, GRAY, GREEN, RED, RESET, YELLOW, bold, cyan, dim
log = get_logger(__name__)
if TYPE_CHECKING:
from collections.abc import Iterator
@@ -91,6 +96,43 @@ class SessionUI(Protocol):
def on_rename(self, name: str) -> None: ...
# ---------------------------------------------------------------------------
# Notify auth helper (module-level, lazy-init)
# ---------------------------------------------------------------------------
_notify_token_manager: Any = None
_notify_token_lock = threading.Lock()
def _notify_auth_headers() -> dict[str, str]:
"""Return Authorization headers for outbound notify requests."""
global _notify_token_manager
# Static token from env takes precedence
static_token = os.environ.get("TURNSTONE_CHANNEL_AUTH_TOKEN", "").strip()
if static_token:
return {"Authorization": f"Bearer {static_token}"}
# JWT via ServiceTokenManager
jwt_secret = os.environ.get("TURNSTONE_JWT_SECRET", "").strip()
if not jwt_secret:
return {}
with _notify_token_lock:
if _notify_token_manager is None:
from turnstone.core.auth import JWT_AUD_CHANNEL, ServiceTokenManager
_notify_token_manager = ServiceTokenManager(
user_id="system",
scopes=frozenset({"write"}),
source="service",
secret=jwt_secret,
audience=JWT_AUD_CHANNEL,
)
header: dict[str, str] = _notify_token_manager.bearer_header
return header
# ---------------------------------------------------------------------------
# ChatSession — the core engine
# ---------------------------------------------------------------------------
@@ -118,6 +160,9 @@ class ChatSession:
health_monitor: BackendHealthMonitor | None = None,
node_id: str | None = None,
ws_id: str | None = None,
tool_search: str = "auto",
tool_search_threshold: int = 20,
tool_search_max_results: int = 5,
):
self.client = client
self.model = model
@@ -151,10 +196,8 @@ class ChatSession:
self.debug = False
self.auto_approve = False
self._node_id = node_id
self._ws_id = ws_id
self._session_id = uuid.uuid4().hex
self._ws_id = ws_id or uuid.uuid4().hex
self._title_generated = False
register_session(self._session_id, node_id=self._node_id, ws_id=self._ws_id)
self._read_files: set[str] = set()
self.messages: list[dict[str, Any]] = []
self._last_usage: dict[str, int] | None = None
@@ -162,32 +205,51 @@ class ChatSession:
self._system_tokens = 0 # tokens for system_messages
self._assistant_pending_tokens = 0
self.creative_mode = False
self._notify_count = 0
# MCP tool integration: merge external tools with built-in
self._mcp_client = mcp_client
self._mcp_refresh_cb: Any = None # Callable | None (avoid import)
if mcp_client:
mcp_tools = mcp_client.get_tools()
self._tools = merge_mcp_tools(TOOLS, mcp_tools)
self._task_tools = merge_mcp_tools(TASK_AGENT_TOOLS, mcp_tools)
self._agent_tools = merge_mcp_tools(AGENT_TOOLS, mcp_tools)
# Register for tool-change notifications from MCP servers
self._mcp_refresh_cb = self._on_mcp_tools_changed
mcp_client.add_listener(self._mcp_refresh_cb)
else:
self._tools = TOOLS
self._task_tools = TASK_AGENT_TOOLS
self._agent_tools = AGENT_TOOLS
# Dynamic tool search: defer MCP tools when tool count is high
self._tool_search_setting = tool_search
self._tool_search_threshold = tool_search_threshold
self._tool_search_max_results = tool_search_max_results
self._tool_search: ToolSearchManager | None = None
if tool_search == "on" or (
tool_search == "auto" and len(self._tools) > tool_search_threshold
):
self._tool_search = ToolSearchManager(
self._tools,
always_on_names=set(BUILTIN_TOOL_NAMES),
threshold=tool_search_threshold,
max_results=tool_search_max_results,
)
self._init_system_messages()
self._save_config()
@property
def session_id(self) -> str:
return self._session_id
def ws_id(self) -> str:
return self._ws_id
@property
def model_alias(self) -> str | None:
return self._model_alias
def _save_config(self) -> None:
"""Persist LLM-affecting config so resumed sessions behave identically."""
save_session_config(
self._session_id,
"""Persist LLM-affecting config so resumed workstreams behave identically."""
save_workstream_config(
self._ws_id,
{
"temperature": str(self.temperature),
"reasoning_effort": self.reasoning_effort,
@@ -197,6 +259,93 @@ class ChatSession:
},
)
# -- MCP tool refresh ----------------------------------------------------
def _on_mcp_tools_changed(self) -> None:
"""Callback from MCPClientManager when the tool list changes.
Rebuilds merged tool lists and reconstructs ToolSearchManager.
Called on the MCP background thread. The work is O(n) where *n* is
the MCP tool count ``merge_mcp_tools`` is list concatenation and
``BM25Index`` construction over <50 tools completes in microseconds,
so this does not meaningfully block the MCP event loop.
Thread safety: each assignment creates a new object (copy-on-write).
Under CPython's GIL, individual reference assignments are atomic.
``_try_stream`` captures tools at call time, so a concurrent refresh
between turns is safe; mid-stream the LLM request already holds
the old snapshot.
"""
if not self._mcp_client:
return
mcp_tools = self._mcp_client.get_tools()
self._tools = merge_mcp_tools(TOOLS, mcp_tools)
self._task_tools = merge_mcp_tools(TASK_AGENT_TOOLS, mcp_tools)
self._agent_tools = merge_mcp_tools(AGENT_TOOLS, mcp_tools)
self._rebuild_tool_search()
def _rebuild_tool_search(self) -> None:
"""Reconstruct ToolSearchManager, preserving expanded tools."""
old_expanded = self._tool_search.get_expanded_names() if self._tool_search else []
if self._tool_search_setting == "on" or (
self._tool_search_setting == "auto" and len(self._tools) > self._tool_search_threshold
):
self._tool_search = ToolSearchManager(
self._tools,
always_on_names=set(BUILTIN_TOOL_NAMES),
threshold=self._tool_search_threshold,
max_results=self._tool_search_max_results,
)
# Restore previously expanded tools that still exist
if old_expanded:
self._tool_search.expand_visible(old_expanded)
else:
self._tool_search = None
def close(self) -> None:
"""Release resources (listener registrations, etc.)."""
if self._mcp_client and self._mcp_refresh_cb:
self._mcp_client.remove_listener(self._mcp_refresh_cb)
self._mcp_refresh_cb = None
def _handle_mcp_refresh(self, arg: str) -> None:
"""Handle ``/mcp refresh [server]``."""
assert self._mcp_client is not None
tokens = arg.split(None, 1) # ["refresh"] or ["refresh", "server"]
server_name: str | None = tokens[1] if len(tokens) > 1 else None
if server_name and server_name not in self._mcp_client.server_names:
known = ", ".join(self._mcp_client.server_names) or "(none)"
self.ui.on_error(f"Unknown MCP server: {server_name}. Known servers: {known}")
return
try:
results = self._mcp_client.refresh_sync(server_name)
except Exception as exc:
self.ui.on_error(f"MCP refresh failed: {exc}")
return
lines: list[str] = []
for srv, (added, removed) in sorted(results.items()):
if added or removed:
summary: list[str] = []
if added:
summary.append(f"+{len(added)} added")
if removed:
summary.append(f"-{len(removed)} removed")
lines.append(f" {srv}: {', '.join(summary)}")
for name in added:
lines.append(f" {GREEN}+ {name}{RESET}")
for name in removed:
lines.append(f" {RED}- {name}{RESET}")
else:
lines.append(f" {srv}: {dim('no changes')}")
header = "MCP refresh complete:"
self.ui.on_info(
"\n".join([header, *lines]) if lines else "MCP refresh complete: no servers to refresh."
)
def _truncate_output(self, output: str) -> str:
"""Truncate tool output to self.tool_truncation chars, keeping head + tail."""
limit = self.tool_truncation
@@ -256,32 +405,32 @@ class ChatSession:
# Take first line, strip quotes
title = raw.split("\n")[0].strip().strip('"').strip("'")
if title:
update_session_title(self._session_id, title[:80])
update_workstream_title(self._ws_id, title[:80])
except Exception:
pass # Title generation is non-critical
def resume_session(self, session_id: str) -> bool:
"""Load messages from a previous session and resume it.
def resume(self, ws_id: str) -> bool:
"""Load messages from a previous workstream and resume it.
Replaces the current conversation with the loaded messages,
adopting the old session_id so new messages continue in the same
session. Restores persisted config (temperature, reasoning_effort,
etc.) so the resumed session behaves identically to the original.
adopting the old ws_id so new messages continue in the same
workstream. Restores persisted config (temperature, reasoning_effort,
etc.) so the resumed workstream behaves identically to the original.
Returns True on success.
"""
messages = load_session_messages(session_id)
messages = load_messages(ws_id)
if not messages:
return False
self._session_id = session_id
self._ws_id = ws_id
self.messages = messages
self._read_files.clear()
self._last_usage = None
self._title_generated = True # don't re-title resumed sessions
self._title_generated = True # don't re-title resumed workstreams
self._msg_tokens = [
max(1, int(self._msg_char_count(m) / self._chars_per_token)) for m in self.messages
]
# Restore persisted config
config = load_session_config(session_id)
config = load_workstream_config(ws_id)
if config:
if "temperature" in config:
self.temperature = float(config["temperature"])
@@ -354,6 +503,14 @@ class ChatSession:
"Look up documentation → man:\n"
" man(page='tar')",
]
# Tool search hint (client-side mode only — native mode needs no hint)
if self._tool_search:
caps = self._provider.get_capabilities(self.model)
if not caps.supports_tool_search:
dev_parts.append(
"\n\nAdditional tools are available via tool_search. "
"Use it when you need a capability not in your current tool set."
)
if self.instructions:
dev_parts.append("")
dev_parts.append(self.instructions)
@@ -390,6 +547,41 @@ class ChatSession:
return {"chat_template_kwargs": kwargs}
return None
# -- tool search helpers --------------------------------------------------
def _get_active_tools(self) -> list[dict[str, Any]] | None:
"""Return the tool list to send to the LLM.
When tool search is active:
- Native mode (provider supports it): send all tools (provider
marks deferred ones with defer_loading).
- Client-side fallback: send visible tools + synthetic tool_search.
Without tool search: return self._tools unchanged.
"""
if self.creative_mode:
return None
if not self._tool_search:
return self._tools
# Check if provider supports native tool search
caps = self._provider.get_capabilities(self.model)
if caps.supports_tool_search:
# Provider handles defer_loading — send all tools
return self._tools
# Client-side fallback: visible tools + search tool
visible = self._tool_search.get_visible_tools()
return visible + [self._tool_search.get_search_tool_definition()]
def _get_deferred_names(self) -> frozenset[str] | None:
"""Return names of deferred tools for native provider search, or None."""
if not self._tool_search:
return None
caps = self._provider.get_capabilities(self.model)
if not caps.supports_tool_search:
return None # Client-side mode — no deferred names for provider
deferred = self._tool_search.get_deferred_tools()
return frozenset(name for t in deferred if (name := t.get("function", {}).get("name", "")))
# Retryable error names are now provided by LLMProvider.retryable_error_names.
_MAX_RETRIES = 3
_RETRY_BASE_DELAY = 1.0 # seconds
@@ -449,11 +641,12 @@ class ChatSession:
client=client,
model=model,
messages=msgs,
tools=self._tools if not self.creative_mode else None,
tools=self._get_active_tools(),
max_tokens=self.max_tokens,
temperature=self.temperature,
reasoning_effort=self.reasoning_effort,
extra_params=self._provider_extra_params(provider=prov),
deferred_names=self._get_deferred_names(),
)
except Exception as e:
ename = type(e).__name__
@@ -468,9 +661,10 @@ class ChatSession:
def send(self, user_input: str) -> None:
"""Send user input and handle the response loop (including tool calls)."""
self._notify_count = 0
self.messages.append({"role": "user", "content": user_input})
self._msg_tokens.append(max(1, int(len(user_input) / self._chars_per_token)))
save_message(self._session_id, "user", user_input)
save_message(self._ws_id, "user", user_input)
try:
while True:
@@ -506,9 +700,7 @@ class ChatSession:
provider_data = _json.dumps(assistant_msg["_provider_content"])
if content or provider_data is not None:
save_message(
self._session_id, "assistant", content, provider_data=provider_data
)
save_message(self._ws_id, "assistant", content, provider_data=provider_data)
if tc:
for call in tc:
fn = call.get("function", {})
@@ -519,7 +711,7 @@ class ChatSession:
"recall",
):
save_message(
self._session_id,
self._ws_id,
"tool_call",
None,
name,
@@ -569,7 +761,7 @@ class ChatSession:
"recall",
):
save_message(
self._session_id,
self._ws_id,
"tool_result",
output[:2000],
_tname,
@@ -845,7 +1037,8 @@ class ChatSession:
f"{GRAY}[request] model={self.model} "
f"max_tokens={self.max_tokens} temp={self.temperature} "
f"reasoning={self.reasoning_effort} "
f"tools={0 if self.creative_mode else len(self._tools)}{RESET}"
f"tools={0 if self.creative_mode else len(self._get_active_tools() or [])}"
f"{' (search)' if self._tool_search else ''}{RESET}"
)
lines.append(f"{GRAY}[request] {len(msgs)} messages:{RESET}")
for i, m in enumerate(msgs):
@@ -899,7 +1092,8 @@ class ChatSession:
# Calibrate chars_per_token ratio from actual usage.
all_msgs = self._full_messages() # system + self.messages (before append)
tool_def_chars = sum(len(json.dumps(t)) for t in self._tools)
active_tools = self._get_active_tools() or []
tool_def_chars = sum(len(json.dumps(t)) for t in active_tools)
total_chars = sum(self._msg_char_count(m) for m in all_msgs) + tool_def_chars
if total_chars > 0 and prompt_tok > 0:
self._chars_per_token = total_chars / prompt_tok
@@ -1243,11 +1437,13 @@ class ChatSession:
"man": self._prepare_man,
"web_fetch": self._prepare_web_fetch,
"web_search": self._prepare_web_search,
"tool_search": self._prepare_tool_search,
"task": self._prepare_task,
"plan": self._prepare_plan,
"remember": self._prepare_remember,
"recall": self._prepare_recall,
"forget": self._prepare_forget,
"notify": self._prepare_notify,
}
preparer = preparers.get(func_name)
if not preparer:
@@ -1724,6 +1920,48 @@ class ChatSession:
"topic": topic,
}
def _prepare_tool_search(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
"""Prepare a tool search query (client-side BM25 fallback)."""
query = (args.get("query") or "").strip()
if not query:
return {
"call_id": call_id,
"func_name": "tool_search",
"header": "\u2717 tool_search: empty query",
"preview": "",
"needs_approval": False,
"error": "Error: no query provided",
}
if not self._tool_search:
return {
"call_id": call_id,
"func_name": "tool_search",
"header": "\u2717 tool_search: not active",
"preview": "",
"needs_approval": False,
"error": "Tool search is not active.",
}
return {
"call_id": call_id,
"func_name": "tool_search",
"header": f"\u2699 tool_search: {query[:80]}",
"preview": f" {DIM}{query}{RESET}",
"needs_approval": False,
"execute": self._exec_tool_search,
"query": query,
}
def _exec_tool_search(self, item: dict[str, Any]) -> tuple[str, str]:
"""Execute a client-side tool search and expand visible tools."""
assert self._tool_search is not None
query = item["query"]
results = self._tool_search.search(query)
# Expand discovered tools into the visible set
names = [t.get("function", {}).get("name", "") for t in results]
self._tool_search.expand_visible(names)
output = self._tool_search.format_search_results(results)
return item["call_id"], output
def _prepare_task(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
"""Prepare a general-purpose sub-agent task for approval."""
prompt = (args.get("prompt") or "").strip()
@@ -2283,9 +2521,9 @@ class ChatSession:
)
def _exec_plan(self, item: dict[str, Any]) -> tuple[str, str]:
"""Run a planning agent and write the result to .plan-<session_id>.md."""
"""Run a planning agent and write the result to .plan-<ws_id>.md."""
call_id, prompt = item["call_id"], item["prompt"]
plan_path = f".plan-{self._session_id}.md"
plan_path = f".plan-{self._ws_id}.md"
# If plan was called before in this session, the previous assistant
# tool_call + tool result are already in self.messages — pass them
@@ -2394,6 +2632,195 @@ class ChatSession:
self.ui.on_tool_result(call_id, "recall", output)
return call_id, output
# -- Notify tool -----------------------------------------------------------
def _prepare_notify(self, call_id: str, args: dict[str, Any]) -> dict[str, Any]:
"""Prepare a channel notification."""
message = (args.get("message") or "").strip()
if not message:
return {
"call_id": call_id,
"func_name": "notify",
"header": "\u2717 notify: empty message",
"preview": "",
"needs_approval": False,
"error": "Error: message is required",
}
if len(message) > 2000:
return {
"call_id": call_id,
"func_name": "notify",
"header": "\u2717 notify: message too long",
"preview": "",
"needs_approval": False,
"error": "Error: message exceeds 2000 character limit",
}
username = (args.get("username") or "").strip()
channel_type = (args.get("channel_type") or "").strip()
channel_id = (args.get("channel_id") or "").strip()
title = (args.get("title") or "").strip()
has_username = bool(username)
has_direct = bool(channel_type and channel_id)
if has_username and has_direct:
return {
"call_id": call_id,
"func_name": "notify",
"header": "\u2717 notify: ambiguous target",
"preview": "",
"needs_approval": False,
"error": "Error: provide either username or channel_type+channel_id, not both",
}
if channel_type and not channel_id:
return {
"call_id": call_id,
"func_name": "notify",
"header": "\u2717 notify: incomplete target",
"preview": "",
"needs_approval": False,
"error": "Error: channel_id is required when channel_type is provided",
}
if channel_id and not channel_type:
return {
"call_id": call_id,
"func_name": "notify",
"header": "\u2717 notify: incomplete target",
"preview": "",
"needs_approval": False,
"error": "Error: channel_type is required when channel_id is provided",
}
if not has_username and not has_direct:
return {
"call_id": call_id,
"func_name": "notify",
"header": "\u2717 notify: no target",
"preview": "",
"needs_approval": False,
"error": "Error: provide username or channel_type+channel_id",
}
target_desc = f"@{username}" if has_username else f"{channel_type}:{channel_id}"
preview = message[:120] + ("..." if len(message) > 120 else "")
return {
"call_id": call_id,
"func_name": "notify",
"header": f"\u2709 notify \u2192 {target_desc}",
"preview": preview,
"needs_approval": False,
"execute": self._exec_notify,
"message": message,
"username": username,
"channel_type": channel_type,
"channel_id": channel_id,
"title": title,
}
_NOTIFY_MAX_RETRIES = 2
_NOTIFY_RETRY_DELAYS = (1.0, 3.0)
def _exec_notify(self, item: dict[str, Any]) -> tuple[str, str]:
"""Send a notification directly to the channel gateway via HTTP."""
call_id = item["call_id"]
if self._notify_count >= 5:
msg = "Error: notification rate limit exceeded (max 5 per turn)"
self.ui.on_tool_result(call_id, "notify", msg)
return call_id, msg
target: dict[str, str] = {}
if item.get("username"):
target["username"] = item["username"]
else:
target["channel_type"] = item["channel_type"]
target["channel_id"] = item["channel_id"]
payload = {
"target": target,
"message": item["message"],
"title": item.get("title", ""),
}
# Build auth headers for service-to-service call
auth_headers = _notify_auth_headers()
# Retry loop: attempt delivery, re-query services on each retry
# in case a gateway comes back online between attempts.
for attempt in range(1 + self._NOTIFY_MAX_RETRIES):
storage = get_storage()
services = storage.list_services("channel", max_age_seconds=120)
if not services:
if attempt < self._NOTIFY_MAX_RETRIES:
delay = self._NOTIFY_RETRY_DELAYS[attempt]
log.warning(
"notify.no_services",
attempt=attempt + 1,
max_retries=self._NOTIFY_MAX_RETRIES,
retry_delay=delay,
)
time.sleep(delay)
continue
log.warning("notify.no_services_exhausted")
msg = "Error: no channel gateway services available"
self.ui.on_tool_result(call_id, "notify", msg)
return call_id, msg
# Try first healthy gateway, fall back to next
last_error: str = ""
for svc in services:
url = svc["url"].rstrip("/") + "/v1/api/notify"
# SSRF guard: only allow http(s) URLs
if not url.startswith(("http://", "https://")):
continue
try:
resp = httpx.post(url, json=payload, timeout=10, headers=auth_headers)
if resp.status_code < 300:
# Check that at least one target was actually delivered
try:
data = resp.json()
except Exception:
last_error = "invalid gateway response"
continue
results = data.get("results") if isinstance(data, dict) else None
if isinstance(results, list) and any(
isinstance(r, dict) and r.get("status") == "sent" for r in results
):
self._notify_count += 1
msg = "Notification sent successfully"
self.ui.on_tool_result(call_id, "notify", msg)
return call_id, msg
last_error = "no successful deliveries"
continue
last_error = f"HTTP {resp.status_code}"
except Exception as exc:
last_error = type(exc).__name__
continue # try next gateway
# All gateways failed this attempt — retry if we have attempts left
if attempt < self._NOTIFY_MAX_RETRIES:
delay = self._NOTIFY_RETRY_DELAYS[attempt]
log.warning(
"notify.all_gateways_failed",
attempt=attempt + 1,
max_retries=self._NOTIFY_MAX_RETRIES,
last_error=last_error,
gateway_count=len(services),
retry_delay=delay,
)
time.sleep(delay)
else:
log.warning(
"notify.delivery_failed",
last_error=last_error,
gateway_count=len(services),
)
msg = "Error: notification delivery failed"
self.ui.on_tool_result(call_id, "notify", msg)
return call_id, msg
def _exec_write_file(self, item: dict[str, Any]) -> tuple[str, str]:
"""Write content to a file, creating parent directories as needed."""
call_id = item["call_id"]
@@ -2675,29 +3102,31 @@ class ChatSession:
self._read_files.clear()
self._last_usage = None
self._msg_tokens = []
self.ui.on_info("Context cleared (session preserved in database).")
self.ui.on_info("Context cleared (messages preserved in database).")
elif cmd == "/new":
from turnstone.core.memory import register_workstream
self.messages.clear()
self._read_files.clear()
self._last_usage = None
self._msg_tokens = []
self._session_id = uuid.uuid4().hex
self._ws_id = uuid.uuid4().hex
self._title_generated = False
register_session(self._session_id, node_id=self._node_id, ws_id=self._ws_id)
register_workstream(self._ws_id, node_id=self._node_id)
self._save_config()
self.ui.on_info("New session started.")
self.ui.on_info("New workstream started.")
elif cmd == "/sessions":
rows = list_sessions(limit=20)
elif cmd == "/workstreams":
rows = list_workstreams_with_history(limit=20)
if not rows:
self.ui.on_info("No saved sessions.")
self.ui.on_info("No saved workstreams.")
else:
lines = ["Sessions:\n"]
for sid, alias, title, _created, updated, count, *_extra in rows:
display_name = alias or sid
lines = ["Workstreams:\n"]
for wid, alias, title, _created, updated, count, *_extra in rows:
display_name = alias or wid
display_title = f" {title}" if title else ""
marker = " *" if sid == self._session_id else " "
marker = " *" if wid == self._ws_id else " "
lines.append(
f" {marker} {bold(display_name)}{display_title} "
f"{dim(f'{count} msgs, {updated}')}"
@@ -2707,30 +3136,29 @@ class ChatSession:
elif cmd == "/resume":
if not arg:
self.ui.on_info(
"Usage: /resume <alias_or_session_id>\n"
"Use /sessions to list available sessions."
"Usage: /resume <alias_or_ws_id>\nUse /workstreams to list available workstreams."
)
else:
target_id = resolve_session(arg.strip())
target_id = resolve_workstream(arg.strip())
if not target_id:
self.ui.on_info(f"Session not found: {arg.strip()}")
elif target_id == self._session_id:
self.ui.on_info("Already in that session.")
elif self.resume_session(target_id):
self.ui.on_info(f"Workstream not found: {arg.strip()}")
elif target_id == self._ws_id:
self.ui.on_info("Already in that workstream.")
elif self.resume(target_id):
self.ui.on_info(
f"Resumed session {bold(target_id)} ({len(self.messages)} messages loaded)"
f"Resumed {bold(target_id)} ({len(self.messages)} messages loaded)"
)
name = get_session_name(target_id)
name = get_workstream_display_name(target_id)
if name:
self.ui.on_rename(name)
else:
self.ui.on_info(f"Session {arg.strip()} has no messages.")
self.ui.on_info(f"Workstream {arg.strip()} has no messages.")
elif cmd == "/name":
if not arg:
self.ui.on_info(f"Current session: {self._session_id}")
elif set_session_alias(self._session_id, arg.strip()):
self.ui.on_info(f"Session named: {bold(arg.strip())}")
self.ui.on_info(f"Current workstream: {self._ws_id}")
elif set_workstream_alias(self._ws_id, arg.strip()):
self.ui.on_info(f"Workstream named: {bold(arg.strip())}")
self.ui.on_rename(arg.strip())
else:
self.ui.on_info(f"Alias '{arg.strip()}' is already in use.")
@@ -2738,18 +3166,18 @@ class ChatSession:
elif cmd == "/delete":
if not arg:
self.ui.on_info(
"Usage: /delete <alias_or_session_id>\nUse /sessions to list sessions."
"Usage: /delete <alias_or_ws_id>\nUse /workstreams to list workstreams."
)
else:
target_id = resolve_session(arg.strip())
target_id = resolve_workstream(arg.strip())
if not target_id:
self.ui.on_info(f"Session not found: {arg.strip()}")
elif target_id == self._session_id:
self.ui.on_info("Cannot delete the active session.")
elif delete_session(target_id):
self.ui.on_info(f"Deleted session {arg.strip()}")
self.ui.on_info(f"Workstream not found: {arg.strip()}")
elif target_id == self._ws_id:
self.ui.on_info("Cannot delete the active workstream.")
elif delete_workstream(target_id):
self.ui.on_info(f"Deleted workstream {arg.strip()}")
else:
self.ui.on_info(f"Failed to delete session {arg.strip()}")
self.ui.on_info(f"Failed to delete workstream {arg.strip()}")
elif cmd == "/history":
query = arg.strip() if arg else None
@@ -2859,6 +3287,8 @@ class ChatSession:
elif cmd == "/mcp":
if not self._mcp_client:
self.ui.on_info("No MCP servers configured.")
elif arg and arg.split()[0] == "refresh":
self._handle_mcp_refresh(arg)
else:
tools = self._mcp_client.get_tools()
if not tools:
@@ -2877,13 +3307,13 @@ class ChatSession:
[
"── Slash Commands ─────────────────────────────────────",
" /instructions <text> Set developer instructions",
" /clear Clear context (session preserved in database)",
" /new Start a new session (old session stays resumable)",
" /clear Clear context (workstream preserved in database)",
" /new Start a new workstream (old one stays resumable)",
"",
" /sessions List saved sessions",
" /resume <id|alias> Resume a previous session",
" /name <alias> Name the current session",
" /delete <id|alias> Delete a saved session",
" /workstreams List saved workstreams",
" /resume <id|alias> Resume a previous workstream",
" /name <alias> Name the current workstream",
" /delete <id|alias> Delete a saved workstream",
"",
" /history [query] Search conversation history (or show recent)",
" /compact Compact conversation (summarize old messages)",
@@ -2893,7 +3323,7 @@ class ChatSession:
" /reason [low|med|high] Set/show reasoning effort",
" /creative Toggle creative writing mode (no tools)",
" /debug Toggle raw SSE delta logging",
" /mcp List connected MCP tools",
" /mcp [refresh [server]] List or refresh MCP tools",
" /help Show this help",
" /exit Exit (also: Ctrl+D)",
"────────────────────────────────────────────────────────",
+35 -10
View File
@@ -16,6 +16,9 @@ def run_migrations(storage: Any, backend: str) -> None:
For SQLite backends, also handles bootstrapping existing databases
that were created before the migration system existed.
For PostgreSQL, acquires an advisory lock so only one process runs
migrations at a time (multiple containers share the same database).
"""
from alembic import command
from alembic.config import Config
@@ -31,13 +34,32 @@ def run_migrations(storage: Any, backend: str) -> None:
if backend == "sqlite":
_bootstrap_existing_sqlite(engine, cfg)
try:
command.upgrade(cfg, "head")
except Exception as exc:
if backend == "sqlite":
if backend == "postgresql":
_run_with_pg_lock(engine, cfg)
else:
try:
command.upgrade(cfg, "head")
except Exception as exc:
log.warning("Migration failed (non-fatal for SQLite): %s", exc)
else:
raise
def _run_with_pg_lock(engine: Any, cfg: Any) -> None:
"""Run Alembic upgrade under a PostgreSQL advisory lock.
Advisory lock ID 7_475_283 (arbitrary, derived from 'turnstone').
``pg_advisory_lock`` blocks until the lock is available, so
concurrent containers wait in line rather than racing.
"""
import sqlalchemy as sa
from alembic import command
with engine.connect() as conn:
conn.execute(sa.text("SELECT pg_advisory_lock(7475283)"))
try:
command.upgrade(cfg, "head")
finally:
conn.execute(sa.text("SELECT pg_advisory_unlock(7475283)"))
conn.commit()
def _bootstrap_existing_sqlite(engine: Any, cfg: Any) -> None:
@@ -58,11 +80,14 @@ def _bootstrap_existing_sqlite(engine: Any, cfg: Any) -> None:
if has_alembic:
return # Already managed by Alembic
# Check if sessions table exists (indicates pre-existing database)
has_sessions = conn.execute(
sa.text("SELECT 1 FROM sqlite_master WHERE type='table' AND name='sessions'")
# Check if a known table exists (indicates pre-existing database)
has_tables = conn.execute(
sa.text(
"SELECT 1 FROM sqlite_master WHERE type='table' "
"AND name IN ('sessions', 'workstreams')"
)
).fetchone()
if has_sessions:
if has_tables:
log.info("Bootstrapping existing database into Alembic (stamping at baseline)")
command.stamp(cfg, "001")
+327 -123
View File
@@ -13,9 +13,8 @@ from turnstone.core.storage._schema import (
conversations,
memories,
metadata,
session_config,
sessions,
users,
workstream_config,
workstreams,
)
from turnstone.core.storage._sqlite import _reconstruct_messages
@@ -38,40 +37,11 @@ class PostgreSQLBackend:
if create_tables:
metadata.create_all(self._engine)
# -- Core session operations -----------------------------------------------
def register_session(
self,
session_id: str,
title: str | None = None,
node_id: str | None = None,
ws_id: str | None = None,
user_id: str | None = None,
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
# Use dialect-neutral upsert pattern
existing = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.session_id == session_id)
).fetchone()
if not existing:
conn.execute(
sa.insert(sessions),
{
"session_id": session_id,
"title": title,
"node_id": node_id,
"ws_id": ws_id,
"user_id": user_id,
"created": now,
"updated": now,
},
)
conn.commit()
# -- Core conversation operations ------------------------------------------
def save_message(
self,
session_id: str,
ws_id: str,
role: str,
content: str | None,
tool_name: str | None = None,
@@ -84,7 +54,7 @@ class PostgreSQLBackend:
conn.execute(
sa.insert(conversations),
{
"session_id": session_id,
"ws_id": ws_id,
"timestamp": now,
"role": role,
"content": content,
@@ -95,11 +65,11 @@ class PostgreSQLBackend:
},
)
conn.execute(
sa.update(sessions).where(sessions.c.session_id == session_id).values(updated=now)
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(updated=now)
)
conn.commit()
def load_session_messages(self, session_id: str) -> list[dict[str, Any]]:
def load_messages(self, ws_id: str) -> list[dict[str, Any]]:
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(
@@ -110,174 +80,149 @@ class PostgreSQLBackend:
conversations.c.tool_call_id,
conversations.c.provider_data,
)
.where(conversations.c.session_id == session_id)
.where(conversations.c.ws_id == ws_id)
.order_by(conversations.c.id)
).fetchall()
return _reconstruct_messages(list(rows), session_id)
return _reconstruct_messages(list(rows), ws_id)
# -- Session management ----------------------------------------------------
# -- Workstream management -------------------------------------------------
def list_sessions(self, limit: int = 20) -> list[Any]:
def list_workstreams_with_history(self, limit: int = 20) -> list[Any]:
with self._engine.connect() as conn:
return list(
conn.execute(
sa.text(
"SELECT s.session_id, s.alias, s.title, s.created, s.updated, "
"SELECT w.ws_id, w.alias, w.title, w.created, w.updated, "
"(SELECT COUNT(*) FROM conversations c "
" WHERE c.session_id = s.session_id), "
"s.node_id, s.ws_id "
"FROM sessions s "
" WHERE c.ws_id = w.ws_id), "
"w.node_id "
"FROM workstreams w "
"WHERE EXISTS "
" (SELECT 1 FROM conversations c WHERE c.session_id = s.session_id) "
"ORDER BY s.updated DESC LIMIT :limit"
" (SELECT 1 FROM conversations c WHERE c.ws_id = w.ws_id) "
"ORDER BY w.updated DESC LIMIT :limit"
),
{"limit": limit},
).fetchall()
)
def delete_session(self, session_id: str) -> bool:
with self._engine.connect() as conn:
conn.execute(sa.delete(conversations).where(conversations.c.session_id == session_id))
conn.execute(sa.delete(session_config).where(session_config.c.session_id == session_id))
conn.execute(sa.delete(sessions).where(sessions.c.session_id == session_id))
conn.commit()
return True
def prune_sessions(self, retention_days: int = 90) -> tuple[int, int]:
def prune_workstreams(self, retention_days: int = 90) -> tuple[int, int]:
orphans = stale = 0
with self._engine.connect() as conn:
# 1. Remove sessions with no messages
# 1. Remove workstreams with no messages
orphan_rows = conn.execute(
sa.text(
"SELECT session_id FROM sessions "
"SELECT ws_id FROM workstreams "
"WHERE NOT EXISTS "
" (SELECT 1 FROM conversations c "
" WHERE c.session_id = sessions.session_id)"
" WHERE c.ws_id = workstreams.ws_id)"
)
).fetchall()
orphan_ids = [r[0] for r in orphan_rows]
if orphan_ids:
conn.execute(
sa.delete(session_config).where(session_config.c.session_id.in_(orphan_ids))
sa.delete(workstream_config).where(workstream_config.c.ws_id.in_(orphan_ids))
)
result = conn.execute(
sa.delete(sessions).where(sessions.c.session_id.in_(orphan_ids))
sa.delete(workstreams).where(workstreams.c.ws_id.in_(orphan_ids))
)
orphans = result.rowcount
# 2. Remove old unnamed sessions
# 2. Remove old unnamed workstreams
if retention_days > 0:
cutoff = (datetime.now(UTC) - timedelta(days=retention_days)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
stale_rows = conn.execute(
sa.select(sessions.c.session_id).where(
sessions.c.alias.is_(None),
sessions.c.updated < cutoff,
sa.select(workstreams.c.ws_id).where(
workstreams.c.alias.is_(None),
workstreams.c.updated < cutoff,
)
).fetchall()
stale_ids = [r[0] for r in stale_rows]
if stale_ids:
conn.execute(
sa.delete(session_config).where(session_config.c.session_id.in_(stale_ids))
sa.delete(conversations).where(conversations.c.ws_id.in_(stale_ids))
)
conn.execute(
sa.delete(workstream_config).where(workstream_config.c.ws_id.in_(stale_ids))
)
result = conn.execute(
sa.delete(sessions).where(sessions.c.session_id.in_(stale_ids))
sa.delete(workstreams).where(workstreams.c.ws_id.in_(stale_ids))
)
stale = result.rowcount
conn.commit()
return (orphans, stale)
def resolve_session(self, alias_or_id: str) -> str | None:
def resolve_workstream(self, alias_or_id: str) -> str | None:
with self._engine.connect() as conn:
# 1. Exact alias
row = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.alias == alias_or_id)
sa.select(workstreams.c.ws_id).where(workstreams.c.alias == alias_or_id)
).fetchone()
if row:
return str(row[0])
# 2. Exact session_id
# 2. Exact ws_id
row = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.session_id == alias_or_id)
sa.select(workstreams.c.ws_id).where(workstreams.c.ws_id == alias_or_id)
).fetchone()
if row:
return str(row[0])
# 3. Prefix match
rows = conn.execute(
sa.select(sessions.c.session_id).where(
sessions.c.session_id.like(alias_or_id + "%")
)
sa.select(workstreams.c.ws_id).where(workstreams.c.ws_id.like(alias_or_id + "%"))
).fetchall()
if len(rows) == 1:
return str(rows[0][0])
# 4. Legacy: check conversations
row = conn.execute(
sa.select(sa.distinct(conversations.c.session_id))
.where(conversations.c.session_id == alias_or_id)
.limit(1)
).fetchone()
if row:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
existing = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.session_id == row[0])
).fetchone()
if not existing:
conn.execute(
sa.insert(sessions),
{"session_id": row[0], "created": now, "updated": now},
)
conn.commit()
return str(row[0])
return None
# -- Session config --------------------------------------------------------
# -- Workstream config -----------------------------------------------------
def save_session_config(self, session_id: str, config: dict[str, str]) -> None:
def save_workstream_config(self, ws_id: str, config: dict[str, str]) -> None:
with self._engine.connect() as conn:
for key, value in config.items():
# Upsert: delete + insert
conn.execute(
sa.delete(session_config).where(
session_config.c.session_id == session_id,
session_config.c.key == key,
sa.delete(workstream_config).where(
workstream_config.c.ws_id == ws_id,
workstream_config.c.key == key,
)
)
conn.execute(
sa.insert(session_config),
{"session_id": session_id, "key": key, "value": value},
sa.insert(workstream_config),
{"ws_id": ws_id, "key": key, "value": value},
)
conn.commit()
def load_session_config(self, session_id: str) -> dict[str, str]:
def load_workstream_config(self, ws_id: str) -> dict[str, str]:
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(session_config.c.key, session_config.c.value).where(
session_config.c.session_id == session_id
sa.select(workstream_config.c.key, workstream_config.c.value).where(
workstream_config.c.ws_id == ws_id
)
).fetchall()
return {row[0]: row[1] for row in rows}
# -- Session metadata ------------------------------------------------------
# -- Workstream metadata ---------------------------------------------------
def set_session_alias(self, session_id: str, alias: str) -> bool:
def set_workstream_alias(self, ws_id: str, alias: str) -> bool:
with self._engine.connect() as conn:
existing = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.alias == alias)
sa.select(workstreams.c.ws_id).where(workstreams.c.alias == alias)
).fetchone()
if existing and existing[0] != session_id:
if existing and existing[0] != ws_id:
return False
conn.execute(
sa.update(sessions).where(sessions.c.session_id == session_id).values(alias=alias)
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(alias=alias)
)
conn.commit()
return True
def get_session_name(self, session_id: str) -> str | None:
def get_workstream_display_name(self, ws_id: str) -> str | None:
with self._engine.connect() as conn:
row = conn.execute(
sa.select(sessions.c.alias, sessions.c.title).where(
sessions.c.session_id == session_id
sa.select(workstreams.c.alias, workstreams.c.title).where(
workstreams.c.ws_id == ws_id
)
).fetchone()
if row:
@@ -285,10 +230,10 @@ class PostgreSQLBackend:
return str(value) if value is not None else None
return None
def update_session_title(self, session_id: str, title: str) -> None:
def update_workstream_title(self, ws_id: str, title: str) -> None:
with self._engine.connect() as conn:
conn.execute(
sa.update(sessions).where(sessions.c.session_id == session_id).values(title=title)
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(title=title)
)
conn.commit()
@@ -359,6 +304,8 @@ class PostgreSQLBackend:
name: str = "",
state: str = "idle",
user_id: str | None = None,
alias: str | None = None,
title: str | None = None,
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
@@ -374,6 +321,8 @@ class PostgreSQLBackend:
"user_id": user_id,
"name": name,
"state": state,
"alias": alias,
"title": title,
"created": now,
"updated": now,
},
@@ -402,6 +351,8 @@ class PostgreSQLBackend:
def delete_workstream(self, ws_id: str) -> bool:
with self._engine.connect() as conn:
conn.execute(sa.delete(conversations).where(conversations.c.ws_id == ws_id))
conn.execute(sa.delete(workstream_config).where(workstream_config.c.ws_id == ws_id))
result = conn.execute(sa.delete(workstreams).where(workstreams.c.ws_id == ws_id))
conn.commit()
return result.rowcount > 0
@@ -436,7 +387,7 @@ class PostgreSQLBackend:
return list(
conn.execute(
sa.text(
"SELECT c.timestamp, c.session_id, c.role, c.content, c.tool_name "
"SELECT c.timestamp, c.ws_id, c.role, c.content, c.tool_name "
"FROM conversations c "
"WHERE to_tsvector('english', COALESCE(c.content, '')) "
" @@ plainto_tsquery('english', :query) "
@@ -452,7 +403,7 @@ class PostgreSQLBackend:
return list(
conn.execute(
sa.text(
"SELECT timestamp, session_id, role, content, tool_name "
"SELECT timestamp, ws_id, role, content, tool_name "
"FROM conversations WHERE content ILIKE :pattern "
"ORDER BY timestamp DESC LIMIT :limit"
),
@@ -466,22 +417,13 @@ class PostgreSQLBackend:
return list(
conn.execute(
sa.text(
"SELECT timestamp, session_id, role, content, tool_name "
"SELECT timestamp, ws_id, role, content, tool_name "
"FROM conversations ORDER BY timestamp DESC LIMIT :limit"
),
{"limit": capped},
).fetchall()
)
# -- Session lookup by workstream ------------------------------------------
def get_session_id_by_ws(self, ws_id: str) -> str | None:
with self._engine.connect() as conn:
row = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.ws_id == ws_id)
).fetchone()
return str(row[0]) if row else None
# -- User identity operations -----------------------------------------------
def create_user(
@@ -879,6 +821,268 @@ class PostgreSQLBackend:
conn.commit()
return result.rowcount > 0
# -- Scheduled tasks -------------------------------------------------------
def create_scheduled_task(
self,
task_id: str,
name: str,
description: str,
schedule_type: str,
cron_expr: str,
at_time: str,
target_mode: str,
model: str,
initial_message: str,
auto_approve: bool,
auto_approve_tools: list[str],
created_by: str,
next_run: str,
) -> None:
from sqlalchemy.dialects import postgresql
from turnstone.core.storage._schema import scheduled_tasks
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
conn.execute(
postgresql.insert(scheduled_tasks)
.values(
task_id=task_id,
name=name,
description=description,
schedule_type=schedule_type,
cron_expr=cron_expr,
at_time=at_time,
target_mode=target_mode,
model=model,
initial_message=initial_message,
auto_approve=1 if auto_approve else 0,
auto_approve_tools=",".join(auto_approve_tools),
enabled=1,
created_by=created_by,
next_run=next_run,
created=now,
updated=now,
)
.on_conflict_do_nothing()
)
conn.commit()
def get_scheduled_task(self, task_id: str) -> dict[str, Any] | None:
from turnstone.core.storage._schema import scheduled_tasks
with self._engine.connect() as conn:
row = conn.execute(
sa.select(scheduled_tasks).where(scheduled_tasks.c.task_id == task_id)
).fetchone()
if row is None:
return None
return dict(row._mapping)
def list_scheduled_tasks(self) -> list[dict[str, Any]]:
from turnstone.core.storage._schema import scheduled_tasks
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(scheduled_tasks).order_by(scheduled_tasks.c.created.desc())
).fetchall()
return [dict(r._mapping) for r in rows]
_UPDATABLE_TASK_FIELDS = frozenset(
{
"name",
"description",
"schedule_type",
"cron_expr",
"at_time",
"target_mode",
"model",
"initial_message",
"auto_approve",
"auto_approve_tools",
"enabled",
"last_run",
"next_run",
"updated",
}
)
def update_scheduled_task(self, task_id: str, **fields: Any) -> bool:
from turnstone.core.storage._schema import scheduled_tasks
fields = {k: v for k, v in fields.items() if k in self._UPDATABLE_TASK_FIELDS}
fields["updated"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
if "auto_approve" in fields:
fields["auto_approve"] = 1 if fields["auto_approve"] else 0
if "auto_approve_tools" in fields and isinstance(fields["auto_approve_tools"], list):
fields["auto_approve_tools"] = ",".join(fields["auto_approve_tools"])
if "enabled" in fields:
fields["enabled"] = 1 if fields["enabled"] else 0
with self._engine.connect() as conn:
result = conn.execute(
sa.update(scheduled_tasks)
.where(scheduled_tasks.c.task_id == task_id)
.values(**fields)
)
conn.commit()
return result.rowcount > 0
def delete_scheduled_task(self, task_id: str) -> bool:
from turnstone.core.storage._schema import scheduled_task_runs, scheduled_tasks
with self._engine.connect() as conn:
conn.execute(
sa.delete(scheduled_task_runs).where(scheduled_task_runs.c.task_id == task_id)
)
result = conn.execute(
sa.delete(scheduled_tasks).where(scheduled_tasks.c.task_id == task_id)
)
conn.commit()
return result.rowcount > 0
def list_due_tasks(self, now: str) -> list[dict[str, Any]]:
from turnstone.core.storage._schema import scheduled_tasks
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(scheduled_tasks)
.where(
(scheduled_tasks.c.enabled == 1)
& (scheduled_tasks.c.next_run <= now)
& (scheduled_tasks.c.next_run != "")
)
.order_by(scheduled_tasks.c.next_run)
.limit(100)
).fetchall()
return [dict(r._mapping) for r in rows]
def record_task_run(
self,
run_id: str,
task_id: str,
node_id: str,
ws_id: str,
correlation_id: str,
started: str,
status: str,
error: str,
) -> None:
from turnstone.core.storage._schema import scheduled_task_runs
with self._engine.connect() as conn:
conn.execute(
sa.insert(scheduled_task_runs),
{
"run_id": run_id,
"task_id": task_id,
"node_id": node_id,
"ws_id": ws_id,
"correlation_id": correlation_id,
"started": started,
"status": status,
"error": error,
},
)
conn.commit()
def list_task_runs(self, task_id: str, limit: int = 50) -> list[dict[str, Any]]:
from turnstone.core.storage._schema import scheduled_task_runs
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(scheduled_task_runs)
.where(scheduled_task_runs.c.task_id == task_id)
.order_by(scheduled_task_runs.c.started.desc())
.limit(limit)
).fetchall()
return [dict(r._mapping) for r in rows]
def prune_task_runs(self, retention_days: int = 90) -> int:
from datetime import timedelta
from turnstone.core.storage._schema import scheduled_task_runs
cutoff = (datetime.now(UTC) - timedelta(days=retention_days)).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
result = conn.execute(
sa.delete(scheduled_task_runs).where(scheduled_task_runs.c.started < cutoff)
)
conn.commit()
return result.rowcount
# -- Service registry ------------------------------------------------------
def register_service(
self, service_type: str, service_id: str, url: str, metadata: str = "{}"
) -> None:
from turnstone.core.storage._schema import services
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
from sqlalchemy.dialects.postgresql import insert as pg_insert
stmt = pg_insert(services).values(
service_type=service_type,
service_id=service_id,
url=url,
metadata=metadata,
last_heartbeat=now,
created=now,
)
stmt = stmt.on_conflict_do_update(
index_elements=[services.c.service_type, services.c.service_id],
set_={"url": url, "metadata": metadata, "last_heartbeat": now},
)
conn.execute(stmt)
conn.commit()
def heartbeat_service(self, service_type: str, service_id: str) -> bool:
from turnstone.core.storage._schema import services
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
result = conn.execute(
sa.update(services)
.where(
(services.c.service_type == service_type)
& (services.c.service_id == service_id)
)
.values(last_heartbeat=now)
)
conn.commit()
return result.rowcount > 0
def list_services(self, service_type: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
from turnstone.core.storage._schema import services
cutoff = (datetime.now(UTC) - timedelta(seconds=max_age_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(services)
.where(
(services.c.service_type == service_type)
& (services.c.last_heartbeat >= cutoff)
)
.order_by(services.c.last_heartbeat.desc())
).fetchall()
return [dict(r._mapping) for r in rows]
def deregister_service(self, service_type: str, service_id: str) -> bool:
from turnstone.core.storage._schema import services
with self._engine.connect() as conn:
result = conn.execute(
sa.delete(services).where(
(services.c.service_type == service_type)
& (services.c.service_id == service_id)
)
)
conn.commit()
return result.rowcount > 0
# -- Lifecycle -------------------------------------------------------------
def close(self) -> None:
+111 -47
View File
@@ -9,26 +9,15 @@ from typing import Any, Protocol, runtime_checkable
class StorageBackend(Protocol):
"""Protocol that every storage backend adapter must implement.
Provides session management, conversation persistence, key-value storage
Provides workstream management, conversation persistence, key-value storage
(for memories), and full-text search.
"""
# -- Core session operations -----------------------------------------------
def register_session(
self,
session_id: str,
title: str | None = None,
node_id: str | None = None,
ws_id: str | None = None,
user_id: str | None = None,
) -> None:
"""Create a sessions row for a new session (no-op if already exists)."""
...
# -- Core conversation operations ------------------------------------------
def save_message(
self,
session_id: str,
ws_id: str,
role: str,
content: str | None,
tool_name: str | None = None,
@@ -39,50 +28,46 @@ class StorageBackend(Protocol):
"""Log a message to the conversations table."""
...
def load_session_messages(self, session_id: str) -> list[dict[str, Any]]:
"""Load messages for a session and reconstruct OpenAI message format."""
def load_messages(self, ws_id: str) -> list[dict[str, Any]]:
"""Load messages for a workstream and reconstruct OpenAI message format."""
...
# -- Session management ----------------------------------------------------
# -- Workstream management -------------------------------------------------
def list_sessions(self, limit: int = 20) -> list[Any]:
"""List recent sessions with message counts, ordered by updated DESC."""
def list_workstreams_with_history(self, limit: int = 20) -> list[Any]:
"""List workstreams that have messages, ordered by updated DESC."""
...
def delete_session(self, session_id: str) -> bool:
"""Delete a session and all its messages. Returns True on success."""
def prune_workstreams(self, retention_days: int = 90) -> tuple[int, int]:
"""Remove orphaned + stale unnamed workstreams. Returns (orphans, stale)."""
...
def prune_sessions(self, retention_days: int = 90) -> tuple[int, int]:
"""Remove orphaned + stale unnamed sessions. Returns (orphans, stale)."""
def resolve_workstream(self, alias_or_id: str) -> str | None:
"""Resolve an alias or ws_id (or prefix) to a full ws_id."""
...
def resolve_session(self, alias_or_id: str) -> str | None:
"""Resolve an alias or session_id (or prefix) to a full session_id."""
# -- Workstream config -----------------------------------------------------
def save_workstream_config(self, ws_id: str, config: dict[str, str]) -> None:
"""Persist workstream configuration key/value pairs."""
...
# -- Session config --------------------------------------------------------
def save_session_config(self, session_id: str, config: dict[str, str]) -> None:
"""Persist session configuration key/value pairs."""
def load_workstream_config(self, ws_id: str) -> dict[str, str]:
"""Load workstream configuration. Returns empty dict if none stored."""
...
def load_session_config(self, session_id: str) -> dict[str, str]:
"""Load session configuration. Returns empty dict if none stored."""
...
# -- Workstream metadata ---------------------------------------------------
# -- Session metadata ------------------------------------------------------
def set_session_alias(self, session_id: str, alias: str) -> bool:
def set_workstream_alias(self, ws_id: str, alias: str) -> bool:
"""Set a human-friendly alias. Returns False if alias is taken."""
...
def get_session_name(self, session_id: str) -> str | None:
"""Return the alias (or title) for a session, or None if unset."""
def get_workstream_display_name(self, ws_id: str) -> str | None:
"""Return the alias (or title) for a workstream, or None if unset."""
...
def update_session_title(self, session_id: str, title: str) -> None:
"""Set or update the auto-generated title for a session."""
def update_workstream_title(self, ws_id: str, title: str) -> None:
"""Set or update the auto-generated title for a workstream."""
...
# -- Generic key-value store (backs memories table) ------------------------
@@ -116,6 +101,8 @@ class StorageBackend(Protocol):
name: str = "",
state: str = "idle",
user_id: str | None = None,
alias: str | None = None,
title: str | None = None,
) -> None:
"""Create a workstreams row (no-op if already exists)."""
...
@@ -129,7 +116,7 @@ class StorageBackend(Protocol):
...
def delete_workstream(self, ws_id: str) -> bool:
"""Delete a workstream. Returns True on success."""
"""Delete a workstream and all its conversations + config."""
...
def list_workstreams(self, node_id: str | None = None, limit: int = 100) -> list[Any]:
@@ -139,7 +126,7 @@ class StorageBackend(Protocol):
# -- Conversation search ---------------------------------------------------
def search_history(self, query: str, limit: int = 20) -> list[Any]:
"""Search conversation history. Returns (timestamp, session_id, role, content, tool_name)."""
"""Search conversation history. Returns (timestamp, ws_id, role, content, tool_name)."""
...
def search_history_recent(self, limit: int = 20) -> list[Any]:
@@ -219,12 +206,6 @@ class StorageBackend(Protocol):
"""Remove a channel user mapping. Returns True if existed."""
...
# -- Session lookup by workstream ------------------------------------------
def get_session_id_by_ws(self, ws_id: str) -> str | None:
"""Find the session_id associated with a workstream. Returns None if not found."""
...
# -- Channel routing -------------------------------------------------------
def create_channel_route(
@@ -249,6 +230,89 @@ class StorageBackend(Protocol):
"""Remove a channel route. Returns True if existed."""
...
# -- Scheduled tasks -------------------------------------------------------
def create_scheduled_task(
self,
task_id: str,
name: str,
description: str,
schedule_type: str,
cron_expr: str,
at_time: str,
target_mode: str,
model: str,
initial_message: str,
auto_approve: bool,
auto_approve_tools: list[str],
created_by: str,
next_run: str,
) -> None:
"""Create a scheduled task. No-op if task_id already exists."""
...
def get_scheduled_task(self, task_id: str) -> dict[str, Any] | None:
"""Return scheduled task dict or None."""
...
def list_scheduled_tasks(self) -> list[dict[str, Any]]:
"""Return all scheduled tasks ordered by created DESC."""
...
def update_scheduled_task(self, task_id: str, **fields: Any) -> bool:
"""Update specified fields on a scheduled task. Returns True if found."""
...
def delete_scheduled_task(self, task_id: str) -> bool:
"""Delete a scheduled task and its run history. Returns True if found."""
...
def list_due_tasks(self, now: str) -> list[dict[str, Any]]:
"""Return enabled tasks whose next_run <= now, ordered by next_run."""
...
def record_task_run(
self,
run_id: str,
task_id: str,
node_id: str,
ws_id: str,
correlation_id: str,
started: str,
status: str,
error: str,
) -> None:
"""Record a scheduled task execution."""
...
def list_task_runs(self, task_id: str, limit: int = 50) -> list[dict[str, Any]]:
"""List run history for a task, ordered by started DESC."""
...
def prune_task_runs(self, retention_days: int = 90) -> int:
"""Delete task runs older than retention_days. Returns count deleted."""
...
# -- Service registry ------------------------------------------------------
def register_service(
self, service_type: str, service_id: str, url: str, metadata: str = "{}"
) -> None:
"""Register or update a service instance. Upserts by (service_type, service_id)."""
...
def heartbeat_service(self, service_type: str, service_id: str) -> bool:
"""Update last_heartbeat for a registered service. Returns False if not found."""
...
def list_services(self, service_type: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
"""Return healthy services of a given type (heartbeat within max_age_seconds)."""
...
def deregister_service(self, service_type: str, service_id: str) -> bool:
"""Remove a service registration. Returns True if existed."""
...
# -- Lifecycle -------------------------------------------------------------
def close(self) -> None:
+71 -25
View File
@@ -22,7 +22,7 @@ conversations = sa.Table(
"conversations",
metadata,
sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
sa.Column("session_id", sa.Text, nullable=False, index=True),
sa.Column("ws_id", sa.Text, nullable=False, index=True),
sa.Column("timestamp", sa.Text, nullable=False),
sa.Column("role", sa.Text, nullable=False),
sa.Column("content", sa.Text),
@@ -32,32 +32,14 @@ conversations = sa.Table(
sa.Column("provider_data", sa.Text),
)
sessions = sa.Table(
"sessions",
metadata,
sa.Column("session_id", sa.Text, primary_key=True),
sa.Column("alias", sa.Text, unique=True),
sa.Column("title", sa.Text),
sa.Column("node_id", sa.Text),
sa.Column("ws_id", sa.Text),
sa.Column("user_id", sa.Text),
sa.Column("created", sa.Text, nullable=False),
sa.Column("updated", sa.Text, nullable=False),
)
# Additional indexes on sessions (name-based to avoid duplication with SA's auto-index)
sa.Index("idx_sessions_alias", sessions.c.alias)
sa.Index("idx_sessions_updated", sessions.c.updated)
sa.Index("idx_sessions_node_id", sessions.c.node_id)
sa.Index("idx_sessions_ws_id", sessions.c.ws_id)
sa.Index("idx_sessions_user_id", sessions.c.user_id)
workstreams = sa.Table(
"workstreams",
metadata,
sa.Column("ws_id", sa.Text, primary_key=True),
sa.Column("node_id", sa.Text),
sa.Column("user_id", sa.Text),
sa.Column("alias", sa.Text, unique=True),
sa.Column("title", sa.Text),
sa.Column("name", sa.Text, nullable=False, server_default=""),
sa.Column("state", sa.Text, nullable=False, server_default="idle"),
sa.Column("created", sa.Text, nullable=False),
@@ -67,14 +49,15 @@ workstreams = sa.Table(
sa.Index("idx_workstreams_node_id", workstreams.c.node_id)
sa.Index("idx_workstreams_state", workstreams.c.state)
sa.Index("idx_workstreams_user_id", workstreams.c.user_id)
sa.Index("idx_workstreams_alias", workstreams.c.alias)
session_config = sa.Table(
"session_config",
workstream_config = sa.Table(
"workstream_config",
metadata,
sa.Column("session_id", sa.Text, nullable=False),
sa.Column("ws_id", sa.Text, nullable=False),
sa.Column("key", sa.Text, nullable=False),
sa.Column("value", sa.Text),
sa.PrimaryKeyConstraint("session_id", "key"),
sa.PrimaryKeyConstraint("ws_id", "key"),
)
# ---------------------------------------------------------------------------
@@ -137,3 +120,66 @@ channel_routes = sa.Table(
)
sa.Index("idx_channel_routes_ws", channel_routes.c.ws_id)
# ---------------------------------------------------------------------------
# Scheduled task tables
# ---------------------------------------------------------------------------
scheduled_tasks = sa.Table(
"scheduled_tasks",
metadata,
sa.Column("task_id", sa.Text, primary_key=True),
sa.Column("name", sa.Text, nullable=False),
sa.Column("description", sa.Text, nullable=False, server_default=""),
sa.Column("schedule_type", sa.Text, nullable=False), # "cron" or "at"
sa.Column("cron_expr", sa.Text, nullable=False, server_default=""),
sa.Column("at_time", sa.Text, nullable=False, server_default=""), # ISO8601
sa.Column("target_mode", sa.Text, nullable=False, server_default="auto"),
sa.Column("model", sa.Text, nullable=False, server_default=""),
sa.Column("initial_message", sa.Text, nullable=False),
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
sa.Column("auto_approve_tools", sa.Text, nullable=False, server_default=""),
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
sa.Column("created_by", sa.Text, nullable=False, server_default=""),
sa.Column("last_run", sa.Text),
sa.Column("next_run", sa.Text),
sa.Column("created", sa.Text, nullable=False),
sa.Column("updated", sa.Text, nullable=False),
)
sa.Index("idx_scheduled_tasks_enabled", scheduled_tasks.c.enabled)
sa.Index("idx_scheduled_tasks_next_run", scheduled_tasks.c.next_run)
scheduled_task_runs = sa.Table(
"scheduled_task_runs",
metadata,
sa.Column("run_id", sa.Text, primary_key=True),
sa.Column("task_id", sa.Text, nullable=False),
sa.Column("node_id", sa.Text, nullable=False, server_default=""),
sa.Column("ws_id", sa.Text, nullable=False, server_default=""),
sa.Column("correlation_id", sa.Text, nullable=False, server_default=""),
sa.Column("started", sa.Text, nullable=False),
sa.Column("status", sa.Text, nullable=False, server_default="dispatched"),
sa.Column("error", sa.Text, nullable=False, server_default=""),
)
sa.Index("idx_scheduled_task_runs_task_id", scheduled_task_runs.c.task_id)
sa.Index("idx_scheduled_task_runs_started", scheduled_task_runs.c.started)
# ---------------------------------------------------------------------------
# Service registry
# ---------------------------------------------------------------------------
services = sa.Table(
"services",
metadata,
sa.Column("service_type", sa.Text, nullable=False),
sa.Column("service_id", sa.Text, nullable=False),
sa.Column("url", sa.Text, nullable=False),
sa.Column("metadata", sa.Text, nullable=False, server_default="{}"),
sa.Column("last_heartbeat", sa.Text, nullable=False),
sa.Column("created", sa.Text, nullable=False),
sa.PrimaryKeyConstraint("service_type", "service_id"),
)
sa.Index("idx_services_type_heartbeat", services.c.service_type, services.c.last_heartbeat)
+326 -121
View File
@@ -15,9 +15,8 @@ from turnstone.core.storage._schema import (
conversations,
memories,
metadata,
session_config,
sessions,
users,
workstream_config,
workstreams,
)
@@ -82,35 +81,11 @@ class SQLiteBackend:
except Exception:
self._fts5_available = False
# -- Core session operations -----------------------------------------------
def register_session(
self,
session_id: str,
title: str | None = None,
node_id: str | None = None,
ws_id: str | None = None,
user_id: str | None = None,
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
conn.execute(
sa.insert(sessions).prefix_with("OR IGNORE"),
{
"session_id": session_id,
"title": title,
"node_id": node_id,
"ws_id": ws_id,
"user_id": user_id,
"created": now,
"updated": now,
},
)
conn.commit()
# -- Core conversation operations ------------------------------------------
def save_message(
self,
session_id: str,
ws_id: str,
role: str,
content: str | None,
tool_name: str | None = None,
@@ -123,7 +98,7 @@ class SQLiteBackend:
result = conn.execute(
sa.insert(conversations),
{
"session_id": session_id,
"ws_id": ws_id,
"timestamp": now,
"role": role,
"content": content,
@@ -145,13 +120,13 @@ class SQLiteBackend:
)
except Exception:
self._fts5_available = False
# Bump session updated timestamp
# Bump workstream updated timestamp
conn.execute(
sa.update(sessions).where(sessions.c.session_id == session_id).values(updated=now)
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(updated=now)
)
conn.commit()
def load_session_messages(self, session_id: str) -> list[dict[str, Any]]:
def load_messages(self, ws_id: str) -> list[dict[str, Any]]:
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(
@@ -162,52 +137,44 @@ class SQLiteBackend:
conversations.c.tool_call_id,
conversations.c.provider_data,
)
.where(conversations.c.session_id == session_id)
.where(conversations.c.ws_id == ws_id)
.order_by(conversations.c.id)
).fetchall()
return _reconstruct_messages(list(rows), session_id)
return _reconstruct_messages(list(rows), ws_id)
# -- Session management ----------------------------------------------------
# -- Workstream management -------------------------------------------------
def list_sessions(self, limit: int = 20) -> list[Any]:
def list_workstreams_with_history(self, limit: int = 20) -> list[Any]:
with self._engine.connect() as conn:
return list(
conn.execute(
sa.text(
"SELECT s.session_id, s.alias, s.title, s.created, s.updated, "
"SELECT w.ws_id, w.alias, w.title, w.created, w.updated, "
"(SELECT COUNT(*) FROM conversations c "
" WHERE c.session_id = s.session_id), "
"s.node_id, s.ws_id "
"FROM sessions s "
" WHERE c.ws_id = w.ws_id), "
"w.node_id "
"FROM workstreams w "
"WHERE EXISTS "
" (SELECT 1 FROM conversations c WHERE c.session_id = s.session_id) "
"ORDER BY s.updated DESC LIMIT :limit"
" (SELECT 1 FROM conversations c WHERE c.ws_id = w.ws_id) "
"ORDER BY w.updated DESC LIMIT :limit"
),
{"limit": limit},
).fetchall()
)
def delete_session(self, session_id: str) -> bool:
with self._engine.connect() as conn:
conn.execute(sa.delete(conversations).where(conversations.c.session_id == session_id))
conn.execute(sa.delete(session_config).where(session_config.c.session_id == session_id))
conn.execute(sa.delete(sessions).where(sessions.c.session_id == session_id))
conn.commit()
return True
def prune_sessions(self, retention_days: int = 90) -> tuple[int, int]:
def prune_workstreams(self, retention_days: int = 90) -> tuple[int, int]:
orphans = stale = 0
with self._engine.connect() as conn:
# 1. Remove sessions with no messages
# 1. Remove workstreams with no messages
orphan_ids = [
row[0]
for row in conn.execute(
sa.text(
"SELECT session_id FROM sessions "
"SELECT ws_id FROM workstreams "
"WHERE NOT EXISTS "
" (SELECT 1 FROM conversations c "
" WHERE c.session_id = sessions.session_id)"
" WHERE c.ws_id = workstreams.ws_id)"
)
).fetchall()
]
@@ -215,16 +182,16 @@ class SQLiteBackend:
placeholders = ",".join([":p" + str(i) for i in range(len(orphan_ids))])
params = {f"p{i}": oid for i, oid in enumerate(orphan_ids)}
conn.execute(
sa.text(f"DELETE FROM session_config WHERE session_id IN ({placeholders})"),
sa.text(f"DELETE FROM workstream_config WHERE ws_id IN ({placeholders})"),
params,
)
result = conn.execute(
sa.text(f"DELETE FROM sessions WHERE session_id IN ({placeholders})"),
sa.text(f"DELETE FROM workstreams WHERE ws_id IN ({placeholders})"),
params,
)
orphans = result.rowcount
# 2. Remove old unnamed sessions
# 2. Remove old unnamed workstreams
if retention_days > 0:
cutoff = (datetime.now(UTC) - timedelta(days=retention_days)).strftime(
"%Y-%m-%dT%H:%M:%S"
@@ -233,7 +200,7 @@ class SQLiteBackend:
row[0]
for row in conn.execute(
sa.text(
"SELECT session_id FROM sessions "
"SELECT ws_id FROM workstreams "
"WHERE alias IS NULL AND updated < :cutoff"
),
{"cutoff": cutoff},
@@ -243,11 +210,15 @@ class SQLiteBackend:
placeholders = ",".join([":p" + str(i) for i in range(len(stale_ids))])
params = {f"p{i}": sid for i, sid in enumerate(stale_ids)}
conn.execute(
sa.text(f"DELETE FROM session_config WHERE session_id IN ({placeholders})"),
sa.text(f"DELETE FROM workstream_config WHERE ws_id IN ({placeholders})"),
params,
)
conn.execute(
sa.text(f"DELETE FROM conversations WHERE ws_id IN ({placeholders})"),
params,
)
result = conn.execute(
sa.text(f"DELETE FROM sessions WHERE session_id IN ({placeholders})"),
sa.text(f"DELETE FROM workstreams WHERE ws_id IN ({placeholders})"),
params,
)
stale = result.rowcount
@@ -255,94 +226,71 @@ class SQLiteBackend:
conn.commit()
return (orphans, stale)
def resolve_session(self, alias_or_id: str) -> str | None:
def resolve_workstream(self, alias_or_id: str) -> str | None:
with self._engine.connect() as conn:
# 1. Exact alias match
row = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.alias == alias_or_id)
sa.select(workstreams.c.ws_id).where(workstreams.c.alias == alias_or_id)
).fetchone()
if row:
return str(row[0])
# 2. Exact session_id match
# 2. Exact ws_id match
row = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.session_id == alias_or_id)
sa.select(workstreams.c.ws_id).where(workstreams.c.ws_id == alias_or_id)
).fetchone()
if row:
return str(row[0])
# 3. Session_id prefix match
# 3. ws_id prefix match
rows = conn.execute(
sa.select(sessions.c.session_id).where(
sessions.c.session_id.like(alias_or_id + "%")
)
sa.select(workstreams.c.ws_id).where(workstreams.c.ws_id.like(alias_or_id + "%"))
).fetchall()
if len(rows) == 1:
return str(rows[0][0])
# 4. Legacy: check conversations table
row = conn.execute(
sa.text(
"SELECT DISTINCT session_id FROM conversations WHERE session_id = :sid LIMIT 1"
),
{"sid": alias_or_id},
).fetchone()
if row:
# Auto-register legacy session
conn.execute(
sa.text(
"INSERT OR IGNORE INTO sessions "
"(session_id, created, updated) VALUES ("
":sid, "
"(SELECT MIN(timestamp) FROM conversations WHERE session_id = :sid), "
"(SELECT MAX(timestamp) FROM conversations WHERE session_id = :sid))"
),
{"sid": row[0]},
)
conn.commit()
return str(row[0])
return None
# -- Session config --------------------------------------------------------
# -- Workstream config -----------------------------------------------------
def save_session_config(self, session_id: str, config: dict[str, str]) -> None:
def save_workstream_config(self, ws_id: str, config: dict[str, str]) -> None:
with self._engine.connect() as conn:
for key, value in config.items():
conn.execute(
sa.text(
"INSERT OR REPLACE INTO session_config "
"(session_id, key, value) VALUES (:sid, :key, :value)"
"INSERT OR REPLACE INTO workstream_config "
"(ws_id, key, value) VALUES (:wid, :key, :value)"
),
{"sid": session_id, "key": key, "value": value},
{"wid": ws_id, "key": key, "value": value},
)
conn.commit()
def load_session_config(self, session_id: str) -> dict[str, str]:
def load_workstream_config(self, ws_id: str) -> dict[str, str]:
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(session_config.c.key, session_config.c.value).where(
session_config.c.session_id == session_id
sa.select(workstream_config.c.key, workstream_config.c.value).where(
workstream_config.c.ws_id == ws_id
)
).fetchall()
return {row[0]: row[1] for row in rows}
# -- Session metadata ------------------------------------------------------
# -- Workstream metadata ---------------------------------------------------
def set_session_alias(self, session_id: str, alias: str) -> bool:
def set_workstream_alias(self, ws_id: str, alias: str) -> bool:
with self._engine.connect() as conn:
existing = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.alias == alias)
sa.select(workstreams.c.ws_id).where(workstreams.c.alias == alias)
).fetchone()
if existing and existing[0] != session_id:
if existing and existing[0] != ws_id:
return False
conn.execute(
sa.update(sessions).where(sessions.c.session_id == session_id).values(alias=alias)
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(alias=alias)
)
conn.commit()
return True
def get_session_name(self, session_id: str) -> str | None:
def get_workstream_display_name(self, ws_id: str) -> str | None:
with self._engine.connect() as conn:
row = conn.execute(
sa.select(sessions.c.alias, sessions.c.title).where(
sessions.c.session_id == session_id
sa.select(workstreams.c.alias, workstreams.c.title).where(
workstreams.c.ws_id == ws_id
)
).fetchone()
if row:
@@ -350,10 +298,10 @@ class SQLiteBackend:
return str(value) if value is not None else None
return None
def update_session_title(self, session_id: str, title: str) -> None:
def update_workstream_title(self, ws_id: str, title: str) -> None:
with self._engine.connect() as conn:
conn.execute(
sa.update(sessions).where(sessions.c.session_id == session_id).values(title=title)
sa.update(workstreams).where(workstreams.c.ws_id == ws_id).values(title=title)
)
conn.commit()
@@ -428,6 +376,8 @@ class SQLiteBackend:
name: str = "",
state: str = "idle",
user_id: str | None = None,
alias: str | None = None,
title: str | None = None,
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
@@ -437,6 +387,8 @@ class SQLiteBackend:
"ws_id": ws_id,
"node_id": node_id,
"user_id": user_id,
"alias": alias,
"title": title,
"name": name,
"state": state,
"created": now,
@@ -467,6 +419,8 @@ class SQLiteBackend:
def delete_workstream(self, ws_id: str) -> bool:
with self._engine.connect() as conn:
conn.execute(sa.delete(conversations).where(conversations.c.ws_id == ws_id))
conn.execute(sa.delete(workstream_config).where(workstream_config.c.ws_id == ws_id))
result = conn.execute(sa.delete(workstreams).where(workstreams.c.ws_id == ws_id))
conn.commit()
return result.rowcount > 0
@@ -500,7 +454,7 @@ class SQLiteBackend:
return list(
conn.execute(
sa.text(
"SELECT c.timestamp, c.session_id, c.role, c.content, c.tool_name "
"SELECT c.timestamp, c.ws_id, c.role, c.content, c.tool_name "
"FROM conversations_fts f "
"JOIN conversations c ON c.id = f.rowid "
"WHERE conversations_fts MATCH :query "
@@ -512,7 +466,7 @@ class SQLiteBackend:
return list(
conn.execute(
sa.text(
"SELECT timestamp, session_id, role, content, tool_name "
"SELECT timestamp, ws_id, role, content, tool_name "
"FROM conversations WHERE content LIKE :pattern ESCAPE '\\' "
"ORDER BY timestamp DESC LIMIT :limit"
),
@@ -526,7 +480,7 @@ class SQLiteBackend:
return list(
conn.execute(
sa.text(
"SELECT timestamp, session_id, role, content, tool_name "
"SELECT timestamp, ws_id, role, content, tool_name "
"FROM conversations ORDER BY timestamp DESC LIMIT :limit"
),
{"limit": capped},
@@ -732,15 +686,6 @@ class SQLiteBackend:
conn.commit()
return result.rowcount > 0
# -- Session lookup by workstream ------------------------------------------
def get_session_id_by_ws(self, ws_id: str) -> str | None:
with self._engine.connect() as conn:
row = conn.execute(
sa.select(sessions.c.session_id).where(sessions.c.ws_id == ws_id)
).fetchone()
return str(row[0]) if row else None
# -- Channel user mapping ---------------------------------------------------
def create_channel_user(self, channel_type: str, channel_user_id: str, user_id: str) -> None:
@@ -929,13 +874,273 @@ class SQLiteBackend:
conn.commit()
return result.rowcount > 0
# -- Scheduled tasks -------------------------------------------------------
def create_scheduled_task(
self,
task_id: str,
name: str,
description: str,
schedule_type: str,
cron_expr: str,
at_time: str,
target_mode: str,
model: str,
initial_message: str,
auto_approve: bool,
auto_approve_tools: list[str],
created_by: str,
next_run: str,
) -> None:
from turnstone.core.storage._schema import scheduled_tasks
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
conn.execute(
sa.insert(scheduled_tasks).prefix_with("OR IGNORE"),
{
"task_id": task_id,
"name": name,
"description": description,
"schedule_type": schedule_type,
"cron_expr": cron_expr,
"at_time": at_time,
"target_mode": target_mode,
"model": model,
"initial_message": initial_message,
"auto_approve": 1 if auto_approve else 0,
"auto_approve_tools": ",".join(auto_approve_tools),
"enabled": 1,
"created_by": created_by,
"next_run": next_run,
"created": now,
"updated": now,
},
)
conn.commit()
def get_scheduled_task(self, task_id: str) -> dict[str, Any] | None:
from turnstone.core.storage._schema import scheduled_tasks
with self._engine.connect() as conn:
row = conn.execute(
sa.select(scheduled_tasks).where(scheduled_tasks.c.task_id == task_id)
).fetchone()
if row is None:
return None
return dict(row._mapping)
def list_scheduled_tasks(self) -> list[dict[str, Any]]:
from turnstone.core.storage._schema import scheduled_tasks
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(scheduled_tasks).order_by(scheduled_tasks.c.created.desc())
).fetchall()
return [dict(r._mapping) for r in rows]
_UPDATABLE_TASK_FIELDS = frozenset(
{
"name",
"description",
"schedule_type",
"cron_expr",
"at_time",
"target_mode",
"model",
"initial_message",
"auto_approve",
"auto_approve_tools",
"enabled",
"last_run",
"next_run",
"updated",
}
)
def update_scheduled_task(self, task_id: str, **fields: Any) -> bool:
from turnstone.core.storage._schema import scheduled_tasks
fields = {k: v for k, v in fields.items() if k in self._UPDATABLE_TASK_FIELDS}
fields["updated"] = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
# Normalize boolean → int for auto_approve
if "auto_approve" in fields:
fields["auto_approve"] = 1 if fields["auto_approve"] else 0
if "auto_approve_tools" in fields and isinstance(fields["auto_approve_tools"], list):
fields["auto_approve_tools"] = ",".join(fields["auto_approve_tools"])
if "enabled" in fields:
fields["enabled"] = 1 if fields["enabled"] else 0
with self._engine.connect() as conn:
result = conn.execute(
sa.update(scheduled_tasks)
.where(scheduled_tasks.c.task_id == task_id)
.values(**fields)
)
conn.commit()
return result.rowcount > 0
def delete_scheduled_task(self, task_id: str) -> bool:
from turnstone.core.storage._schema import scheduled_task_runs, scheduled_tasks
with self._engine.connect() as conn:
conn.execute(
sa.delete(scheduled_task_runs).where(scheduled_task_runs.c.task_id == task_id)
)
result = conn.execute(
sa.delete(scheduled_tasks).where(scheduled_tasks.c.task_id == task_id)
)
conn.commit()
return result.rowcount > 0
def list_due_tasks(self, now: str) -> list[dict[str, Any]]:
from turnstone.core.storage._schema import scheduled_tasks
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(scheduled_tasks)
.where(
(scheduled_tasks.c.enabled == 1)
& (scheduled_tasks.c.next_run <= now)
& (scheduled_tasks.c.next_run != "")
)
.order_by(scheduled_tasks.c.next_run)
.limit(100)
).fetchall()
return [dict(r._mapping) for r in rows]
def record_task_run(
self,
run_id: str,
task_id: str,
node_id: str,
ws_id: str,
correlation_id: str,
started: str,
status: str,
error: str,
) -> None:
from turnstone.core.storage._schema import scheduled_task_runs
with self._engine.connect() as conn:
conn.execute(
sa.insert(scheduled_task_runs),
{
"run_id": run_id,
"task_id": task_id,
"node_id": node_id,
"ws_id": ws_id,
"correlation_id": correlation_id,
"started": started,
"status": status,
"error": error,
},
)
conn.commit()
def list_task_runs(self, task_id: str, limit: int = 50) -> list[dict[str, Any]]:
from turnstone.core.storage._schema import scheduled_task_runs
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(scheduled_task_runs)
.where(scheduled_task_runs.c.task_id == task_id)
.order_by(scheduled_task_runs.c.started.desc())
.limit(limit)
).fetchall()
return [dict(r._mapping) for r in rows]
def prune_task_runs(self, retention_days: int = 90) -> int:
from datetime import timedelta
from turnstone.core.storage._schema import scheduled_task_runs
cutoff = (datetime.now(UTC) - timedelta(days=retention_days)).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
result = conn.execute(
sa.delete(scheduled_task_runs).where(scheduled_task_runs.c.started < cutoff)
)
conn.commit()
return result.rowcount
# -- Service registry ------------------------------------------------------
def register_service(
self, service_type: str, service_id: str, url: str, metadata: str = "{}"
) -> None:
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from turnstone.core.storage._schema import services
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
stmt = sqlite_insert(services).values(
service_type=service_type,
service_id=service_id,
url=url,
metadata=metadata,
last_heartbeat=now,
created=now,
)
stmt = stmt.on_conflict_do_update(
index_elements=["service_type", "service_id"],
set_={"url": url, "metadata": metadata, "last_heartbeat": now},
)
with self._engine.connect() as conn:
conn.execute(stmt)
conn.commit()
def heartbeat_service(self, service_type: str, service_id: str) -> bool:
from turnstone.core.storage._schema import services
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._engine.connect() as conn:
result = conn.execute(
sa.update(services)
.where(
(services.c.service_type == service_type)
& (services.c.service_id == service_id)
)
.values(last_heartbeat=now)
)
conn.commit()
return result.rowcount > 0
def list_services(self, service_type: str, max_age_seconds: int = 120) -> list[dict[str, str]]:
from turnstone.core.storage._schema import services
cutoff = (datetime.now(UTC) - timedelta(seconds=max_age_seconds)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
with self._engine.connect() as conn:
rows = conn.execute(
sa.select(services)
.where(
(services.c.service_type == service_type)
& (services.c.last_heartbeat >= cutoff)
)
.order_by(services.c.last_heartbeat.desc())
).fetchall()
return [dict(r._mapping) for r in rows]
def deregister_service(self, service_type: str, service_id: str) -> bool:
from turnstone.core.storage._schema import services
with self._engine.connect() as conn:
result = conn.execute(
sa.delete(services).where(
(services.c.service_type == service_type)
& (services.c.service_id == service_id)
)
)
conn.commit()
return result.rowcount > 0
# -- Lifecycle -------------------------------------------------------------
def close(self) -> None:
self._engine.dispose()
def _reconstruct_messages(rows: list[Any], session_id: str) -> list[dict[str, Any]]:
def _reconstruct_messages(rows: list[Any], ws_id: str) -> list[dict[str, Any]]:
"""Reconstruct OpenAI message format from stored conversation rows.
Handles tool_call / tool_result grouping and incomplete turn repair.
@@ -973,7 +1178,7 @@ def _reconstruct_messages(rows: list[Any], session_id: str) -> list[dict[str, An
while i < len(rows) and rows[i][0] == "tool_call":
_, _, tn, ta, stored_tc_id, _ = rows[i]
call_id = stored_tc_id or f"call_{session_id}_{i}"
call_id = stored_tc_id or f"call_{ws_id}_{i}"
assistant_msg["tool_calls"].append(
{
"id": call_id,
@@ -0,0 +1,62 @@
"""Scheduled tasks and run history tables.
Revision ID: 004
Revises: 003
Create Date: 2026-03-05
"""
import sqlalchemy as sa
from alembic import op
revision = "004"
down_revision = "003"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"scheduled_tasks",
sa.Column("task_id", sa.Text, primary_key=True),
sa.Column("name", sa.Text, nullable=False),
sa.Column("description", sa.Text, nullable=False, server_default=""),
sa.Column("schedule_type", sa.Text, nullable=False),
sa.Column("cron_expr", sa.Text, nullable=False, server_default=""),
sa.Column("at_time", sa.Text, nullable=False, server_default=""),
sa.Column("target_mode", sa.Text, nullable=False, server_default="auto"),
sa.Column("model", sa.Text, nullable=False, server_default=""),
sa.Column("initial_message", sa.Text, nullable=False),
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
sa.Column("auto_approve_tools", sa.Text, nullable=False, server_default=""),
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
sa.Column("created_by", sa.Text, nullable=False, server_default=""),
sa.Column("last_run", sa.Text),
sa.Column("next_run", sa.Text),
sa.Column("created", sa.Text, nullable=False),
sa.Column("updated", sa.Text, nullable=False),
)
op.create_index("idx_scheduled_tasks_enabled", "scheduled_tasks", ["enabled"])
op.create_index("idx_scheduled_tasks_next_run", "scheduled_tasks", ["next_run"])
op.create_table(
"scheduled_task_runs",
sa.Column("run_id", sa.Text, primary_key=True),
sa.Column("task_id", sa.Text, nullable=False),
sa.Column("node_id", sa.Text, nullable=False, server_default=""),
sa.Column("ws_id", sa.Text, nullable=False, server_default=""),
sa.Column("correlation_id", sa.Text, nullable=False, server_default=""),
sa.Column("started", sa.Text, nullable=False),
sa.Column("status", sa.Text, nullable=False, server_default="dispatched"),
sa.Column("error", sa.Text, nullable=False, server_default=""),
)
op.create_index("idx_scheduled_task_runs_task_id", "scheduled_task_runs", ["task_id"])
op.create_index("idx_scheduled_task_runs_started", "scheduled_task_runs", ["started"])
def downgrade() -> None:
op.drop_index("idx_scheduled_task_runs_started", "scheduled_task_runs")
op.drop_index("idx_scheduled_task_runs_task_id", "scheduled_task_runs")
op.drop_table("scheduled_task_runs")
op.drop_index("idx_scheduled_tasks_next_run", "scheduled_tasks")
op.drop_index("idx_scheduled_tasks_enabled", "scheduled_tasks")
op.drop_table("scheduled_tasks")
@@ -0,0 +1,33 @@
"""Service registry table.
Revision ID: 005
Revises: 004
Create Date: 2026-03-05
"""
import sqlalchemy as sa
from alembic import op
revision = "005"
down_revision = "004"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"services",
sa.Column("service_type", sa.Text, nullable=False),
sa.Column("service_id", sa.Text, nullable=False),
sa.Column("url", sa.Text, nullable=False),
sa.Column("metadata", sa.Text, nullable=False, server_default="{}"),
sa.Column("last_heartbeat", sa.Text, nullable=False),
sa.Column("created", sa.Text, nullable=False),
sa.PrimaryKeyConstraint("service_type", "service_id"),
)
op.create_index("idx_services_type_heartbeat", "services", ["service_type", "last_heartbeat"])
def downgrade() -> None:
op.drop_index("idx_services_type_heartbeat", "services")
op.drop_table("services")
@@ -0,0 +1,131 @@
"""Normalize session_id into ws_id — merge sessions table into workstreams.
Conversations and config are now keyed by ws_id (workstream identity) instead
of a separate session_id. The sessions table is dropped; its alias/title
columns move to workstreams.
Revision ID: 006
Revises: 005
Create Date: 2026-03-07
"""
import sqlalchemy as sa
from alembic import op
revision = "006"
down_revision = "005"
branch_labels = None
depends_on = None
def upgrade() -> None:
# 1. Add alias and title columns to workstreams.
op.add_column("workstreams", sa.Column("alias", sa.Text))
op.add_column("workstreams", sa.Column("title", sa.Text))
op.create_index("idx_workstreams_alias", "workstreams", ["alias"], unique=True)
conn = op.get_bind()
# 2. Copy alias/title from sessions → workstreams (for rows that have a ws_id).
conn.execute(
sa.text(
"UPDATE workstreams SET "
" alias = (SELECT s.alias FROM sessions s WHERE s.ws_id = workstreams.ws_id), "
" title = (SELECT s.title FROM sessions s WHERE s.ws_id = workstreams.ws_id) "
"WHERE EXISTS (SELECT 1 FROM sessions s WHERE s.ws_id = workstreams.ws_id)"
)
)
# 3. Create workstream rows for sessions that have a ws_id but no
# corresponding workstream row yet.
# The NOT EXISTS guard makes this safe on both SQLite and PostgreSQL.
conn.execute(
sa.text(
"INSERT INTO workstreams "
"(ws_id, node_id, alias, title, state, created, updated) "
"SELECT s.ws_id, s.node_id, s.alias, s.title, 'closed', s.created, s.updated "
"FROM sessions s "
"WHERE s.ws_id IS NOT NULL AND s.ws_id != '' "
" AND NOT EXISTS (SELECT 1 FROM workstreams w WHERE w.ws_id = s.ws_id)"
)
)
# 4. Create workstream rows for sessions WITHOUT a ws_id
# (use session_id as ws_id).
conn.execute(
sa.text(
"INSERT INTO workstreams "
"(ws_id, node_id, alias, title, state, created, updated) "
"SELECT s.session_id, s.node_id, s.alias, s.title, 'closed', s.created, s.updated "
"FROM sessions s "
"WHERE (s.ws_id IS NULL OR s.ws_id = '') "
" AND NOT EXISTS (SELECT 1 FROM workstreams w WHERE w.ws_id = s.session_id)"
)
)
# 5. Rename conversations.session_id → conversations.ws_id and remap values.
# For sessions with ws_id: map session_id → ws_id.
# For sessions without ws_id: session_id stays (used as ws_id).
op.alter_column("conversations", "session_id", new_column_name="ws_id")
conn.execute(
sa.text(
"UPDATE conversations SET ws_id = ("
" SELECT COALESCE(NULLIF(s.ws_id, ''), s.session_id) "
" FROM sessions s WHERE s.session_id = conversations.ws_id"
") "
"WHERE EXISTS ("
" SELECT 1 FROM sessions s WHERE s.session_id = conversations.ws_id"
")"
)
)
# 6. Rename session_config → workstream_config with ws_id column.
op.rename_table("session_config", "workstream_config")
op.alter_column("workstream_config", "session_id", new_column_name="ws_id")
conn.execute(
sa.text(
"UPDATE workstream_config SET ws_id = ("
" SELECT COALESCE(NULLIF(s.ws_id, ''), s.session_id) "
" FROM sessions s WHERE s.session_id = workstream_config.ws_id"
") "
"WHERE EXISTS ("
" SELECT 1 FROM sessions s WHERE s.session_id = workstream_config.ws_id"
")"
)
)
# 7. Drop the sessions table.
op.drop_table("sessions")
def downgrade() -> None:
# Recreate the sessions table.
op.create_table(
"sessions",
sa.Column("session_id", sa.Text, primary_key=True),
sa.Column("alias", sa.Text, unique=True),
sa.Column("title", sa.Text),
sa.Column("node_id", sa.Text),
sa.Column("ws_id", sa.Text),
sa.Column("user_id", sa.Text),
sa.Column("created", sa.Text, nullable=False),
sa.Column("updated", sa.Text, nullable=False),
)
op.create_index("idx_sessions_alias", "sessions", ["alias"])
op.create_index("idx_sessions_updated", "sessions", ["updated"])
op.create_index("idx_sessions_node_id", "sessions", ["node_id"])
op.create_index("idx_sessions_ws_id", "sessions", ["ws_id"])
# Reverse config table rename.
op.alter_column("workstream_config", "ws_id", new_column_name="session_id")
op.rename_table("workstream_config", "session_config")
# Reverse conversations column rename.
op.alter_column("conversations", "ws_id", new_column_name="session_id")
# Drop alias/title from workstreams.
op.drop_index("idx_workstreams_alias", "workstreams")
op.drop_column("workstreams", "title")
op.drop_column("workstreams", "alias")
+247
View File
@@ -0,0 +1,247 @@
"""Dynamic tool search — BM25 index and session-scoped visibility manager.
When the total tool count exceeds a configurable threshold, deferred tools
are hidden from the LLM and discoverable via a ``tool_search`` function.
Native providers (Anthropic, OpenAI) handle search server-side; local
models (vLLM, llama.cpp) use the client-side BM25 fallback here.
"""
from __future__ import annotations
import math
import re
from collections import Counter
from typing import Any
# ---------------------------------------------------------------------------
# BM25 index — lightweight, pure-Python, zero external deps
# ---------------------------------------------------------------------------
_SPLIT_RE = re.compile(r"[_\-./\s]+")
def _tokenize(text: str) -> list[str]:
"""Split text on whitespace, underscores, hyphens, dots."""
return [t.lower() for t in _SPLIT_RE.split(text) if t]
class BM25Index:
"""Okapi BM25 index over tool name + description text."""
def __init__(self, documents: list[str], *, k1: float = 1.5, b: float = 0.75) -> None:
self.k1 = k1
self.b = b
self._docs = documents
self._doc_tokens: list[list[str]] = [_tokenize(d) for d in documents]
self._doc_lens = [len(t) for t in self._doc_tokens]
self._avgdl = sum(self._doc_lens) / max(len(self._doc_lens), 1)
self._n = len(documents)
# Document frequency per term
self._df: Counter[str] = Counter()
for tokens in self._doc_tokens:
for term in set(tokens):
self._df[term] += 1
def search(self, query: str, k: int = 5) -> list[int]:
"""Return indices of top-k documents sorted by descending BM25 score."""
q_tokens = _tokenize(query)
if not q_tokens:
return []
scores: list[tuple[float, int]] = []
for idx, doc_tokens in enumerate(self._doc_tokens):
score = self._score(q_tokens, doc_tokens, self._doc_lens[idx])
if score > 0:
scores.append((score, idx))
scores.sort(key=lambda x: (-x[0], x[1]))
return [idx for _, idx in scores[:k]]
def _score(self, q_tokens: list[str], doc_tokens: list[str], dl: int) -> float:
tf_map: Counter[str] = Counter(doc_tokens)
score = 0.0
for term in q_tokens:
if term not in tf_map:
continue
tf = tf_map[term]
df = self._df.get(term, 0)
idf = math.log((self._n - df + 0.5) / (df + 0.5) + 1.0)
numerator = tf * (self.k1 + 1)
denominator = tf + self.k1 * (1 - self.b + self.b * dl / self._avgdl)
score += idf * numerator / denominator
return score
# ---------------------------------------------------------------------------
# Tool search manager — partitions tools, tracks visibility
# ---------------------------------------------------------------------------
_MCP_PREFIX_RE = re.compile(r"^mcp__(.+?)__")
def _tool_name(tool: dict[str, Any]) -> str:
"""Extract function name from an OpenAI-format tool dict."""
fn: dict[str, Any] = tool.get("function", {})
name: str = fn.get("name", "")
return name
def _tool_text(tool: dict[str, Any]) -> str:
"""Build searchable text from tool name + description."""
fn = tool.get("function", {})
return f"{fn.get('name', '')} {fn.get('description', '')}"
def _mcp_server_summary(tools: list[dict[str, Any]]) -> str:
"""Summarise deferred tools by MCP server prefix for the hint."""
servers: Counter[str] = Counter()
other = 0
for tool in tools:
name = _tool_name(tool)
m = _MCP_PREFIX_RE.match(name)
if m:
servers[m.group(1)] += 1
else:
other += 1
parts = [f"{srv} ({cnt} tool{'s' if cnt != 1 else ''})" for srv, cnt in sorted(servers.items())]
if other:
parts.append(f"other ({other} tool{'s' if other != 1 else ''})")
return ", ".join(parts)
class ToolSearchManager:
"""Session-scoped tool visibility manager with BM25 search.
Partitions tools into always-on (built-in) and deferred (MCP) sets.
Tracks which deferred tools have been discovered and expanded into
the visible set for the current session.
"""
def __init__(
self,
all_tools: list[dict[str, Any]],
always_on_names: set[str],
*,
threshold: int = 20,
max_results: int = 5,
) -> None:
self._all_tools = all_tools
self._always_on: list[dict[str, Any]] = []
self._deferred: list[dict[str, Any]] = []
self._deferred_by_name: dict[str, dict[str, Any]] = {}
self._expanded: dict[str, None] = {} # ordered set (preserves discovery order)
self._threshold = threshold
self._max_results = max_results
for tool in all_tools:
name = _tool_name(tool)
if name in always_on_names:
self._always_on.append(tool)
else:
self._deferred.append(tool)
self._deferred_by_name[name] = tool
# BM25 index over deferred tools
texts = [_tool_text(t) for t in self._deferred]
self._index = BM25Index(texts)
# Pre-compute server summary for the search tool description
self._server_hint = _mcp_server_summary(self._deferred)
def should_activate(self) -> bool:
"""Return True if tool search should be active (enough tools)."""
return len(self._all_tools) > self._threshold
def get_visible_tools(self) -> list[dict[str, Any]]:
"""Return always-on tools + any expanded (discovered) tools."""
result = list(self._always_on)
for name in self._expanded:
tool = self._deferred_by_name.get(name)
if tool:
result.append(tool)
return result
def get_deferred_tools(self) -> list[dict[str, Any]]:
"""Return tools that are currently deferred (not yet discovered)."""
return [t for t in self._deferred if _tool_name(t) not in self._expanded]
def get_all_tools(self) -> list[dict[str, Any]]:
"""Return the full tool list (for native provider modes)."""
return list(self._all_tools)
def search(self, query: str) -> list[dict[str, Any]]:
"""Search deferred tools by query, return top-k matches.
Already-expanded tools are excluded so every result is genuinely new.
"""
# Request extra results to compensate for filtering out expanded tools
indices = self._index.search(query, k=self._max_results + len(self._expanded))
results = []
for i in indices:
if _tool_name(self._deferred[i]) not in self._expanded:
results.append(self._deferred[i])
if len(results) >= self._max_results:
break
return results
def get_expanded_names(self) -> list[str]:
"""Return names of currently expanded (discovered) tools."""
return list(self._expanded.keys())
def expand_visible(self, tool_names: list[str]) -> list[dict[str, Any]]:
"""Promote discovered tools to the visible set.
Returns the newly-expanded tool definitions (excludes tools
that were already visible).
"""
newly_added = []
for name in tool_names:
if name not in self._expanded and name in self._deferred_by_name:
self._expanded[name] = None
newly_added.append(self._deferred_by_name[name])
return newly_added
def get_search_tool_definition(self) -> dict[str, Any]:
"""Return the synthetic ``tool_search`` function tool definition.
The description includes a dynamic hint listing available MCP
server names and tool counts so the model can craft specific queries.
"""
desc = (
"Search for available tools by keyword. Returns matching tool "
"names and descriptions. Use this when you need a capability "
"not available in your current tool set."
)
if self._server_hint:
desc += f" Available tool servers: {self._server_hint}."
return {
"type": "function",
"function": {
"name": "tool_search",
"description": desc,
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search query describing the capability you need.",
},
},
"required": ["query"],
},
},
}
def format_search_results(self, tools: list[dict[str, Any]]) -> str:
"""Format search results as text for the tool_search response."""
if not tools:
return "No matching tools found. Try a different search query."
lines = []
for tool in tools:
fn = tool.get("function", {})
name = fn.get("name", "")
desc = fn.get("description", "")
lines.append(f"- **{name}**: {desc}")
return (
f"Found {len(tools)} matching tool(s):\n"
+ "\n".join(lines)
+ "\n\nThese tools are now available for use."
)
+1
View File
@@ -37,6 +37,7 @@ TASK_AGENT_TOOLS = [t for t in TOOLS if _META[t["function"]["name"]].get("task_a
AGENT_AUTO_TOOLS = {n for n, m in _META.items() if m.get("auto_approve")}
TASK_AUTO_TOOLS = {n for n, m in _META.items() if m.get("auto_approve")}
PRIMARY_KEY_MAP = {n: m["primary_key"] for n, m in _META.items() if "primary_key" in m}
BUILTIN_TOOL_NAMES = frozenset(_META)
def merge_mcp_tools(

Some files were not shown because too many files have changed in this diff Show More