ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

Initial commit — turnstone multi-node AI orchestration platform.

Browse Source
This commit is contained in:
Patrick Buckley
2026-03-02 00:24:29 -08:00
commit 0d6252dd7d
98 changed files with 25098 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+14
View File
@@ -0,0 +1,14 @@
__pycache__/
*.py[cod]
*.egg-info/
dist/
build/
.venv/
venv/
.env
*.db
.git/
.pytest_cache/
.mypy_cache/
.ruff_cache/
.hypothesis/
.env.example
+85
View File
@@ -0,0 +1,85 @@
# =============================================================================
# Turnstone Docker Compose — Environment Configuration
# Copy to .env and fill in your values: cp .env.example .env
# =============================================================================
# ---------------------------------------------------------------------------
# LLM Backend
# ---------------------------------------------------------------------------
# OpenAI-compatible API URL (vLLM, llama.cpp, OpenAI, etc.)
LLM_BASE_URL=http://host.docker.internal:8000/v1
# API key for the LLM backend ("dummy" for local servers without auth)
OPENAI_API_KEY=dummy
# Tavily API key for web_search tool (optional)
TAVILY_API_KEY=
# ---------------------------------------------------------------------------
# Redis
# ---------------------------------------------------------------------------
# Redis password (leave empty for no authentication)
REDIS_PASSWORD=
# Host port for Redis
REDIS_PORT=6379
# ---------------------------------------------------------------------------
# Server
# ---------------------------------------------------------------------------
# Host port for the turnstone web UI
SERVER_PORT=8080
# Set to any non-empty value to auto-approve all tool calls
SKIP_PERMISSIONS=
# ---------------------------------------------------------------------------
# Bridge
# ---------------------------------------------------------------------------
# Heartbeat TTL in seconds
HEARTBEAT_TTL=60
# Seconds to wait for external approval responses
APPROVAL_TIMEOUT=300
# ---------------------------------------------------------------------------
# Console (Cluster Dashboard)
# ---------------------------------------------------------------------------
# Host port for the cluster dashboard
CONSOLE_PORT=8090
# Seconds between node polling cycles
CONSOLE_POLL_INTERVAL=10
# ---------------------------------------------------------------------------
# Auth (optional)
# ---------------------------------------------------------------------------
# Set to "1" to require Bearer token authentication
TURNSTONE_AUTH_ENABLED=
# Bearer token for server/bridge/console authentication
TURNSTONE_AUTH_TOKEN=
# ---------------------------------------------------------------------------
# Simulator (used with: docker compose --profile sim up)
# ---------------------------------------------------------------------------
# Number of simulated nodes
SIM_NODES=100
# Scenario: steady, burst, node_failure, directed, lifecycle
SIM_SCENARIO=steady
# Scenario duration in seconds
SIM_DURATION=60
# Messages per second (steady scenario)
SIM_MPS=5.0
# Log level
SIM_LOG_LEVEL=INFO
# Random seed for reproducibility (leave empty for random)
SIM_SEED=
# Path to write JSON metrics report (leave empty to skip)
SIM_METRICS_FILE=
.gitattributes
+1
View File
@@ -0,0 +1 @@
*.png filter=lfs diff=lfs merge=lfs -text
.gitignore
+19
View File
@@ -0,0 +1,19 @@
__pycache__/
*.py[cod]
*$py.class
*.egg-info/
dist/
build/
.eggs/
*.egg
.venv/
venv/
.env
*.so
.mypy_cache/
.ruff_cache/
.pytest_cache/
*.db
.plan.md
.plan-*.md
.hypothesis/
CLA.md
+54
View File
@@ -0,0 +1,54 @@
# Turnstone Contributor License Agreement
Thank you for your interest in contributing to Turnstone. This Contributor
License Agreement ("Agreement") is between you ("Contributor") and Patrick
Buckley ("Maintainer"), and applies to all Contributions submitted to the
Turnstone project at https://github.com/turnstonelabs/turnstone.
By submitting a Contribution, you agree to the following terms:
## 1. Definitions
"Contribution" means any original work of authorship, including modifications
or additions to existing work, that you submit to the project via pull request,
issue, or any other means.
## 2. Grant of Copyright License
You grant the Maintainer a perpetual, worldwide, non-exclusive, royalty-free,
irrevocable copyright license to reproduce, prepare derivative works of,
publicly display, publicly perform, sublicense, and distribute your
Contributions and any derivative works thereof under any license, including
proprietary licenses.
## 3. Grant of Patent License
You grant the Maintainer a perpetual, worldwide, non-exclusive, royalty-free,
irrevocable patent license to make, have made, use, offer to sell, sell,
import, and otherwise transfer your Contributions, where such license applies
only to patent claims licensable by you that are necessarily infringed by your
Contributions alone or in combination with the project.
## 4. Representations
You represent that:
- Each Contribution is your original creation and you have the legal right to
grant the above licenses.
- Your Contribution does not violate any third party's intellectual property or
other rights.
- If your employer has rights to intellectual property that you create, you have
received permission to make the Contribution on behalf of that employer, or
your employer has waived such rights.
## 5. No Obligation
You understand that the decision to include your Contribution in the project is
entirely at the Maintainer's discretion. This Agreement does not obligate the
Maintainer to use or incorporate your Contribution.
## Signing
By submitting a pull request to the Turnstone repository, and commenting
"I have read the CLA and I agree to its terms," you indicate your acceptance
of this Agreement.
CONTRIBUTING.md
+51
View File
@@ -0,0 +1,51 @@
# Contributing to Turnstone
Thanks for your interest in contributing! Here's what you need to know.
## Contributor License Agreement
All contributors must agree to the [Contributor License Agreement](CLA.md)
before their pull request can be merged. When you open your first PR, the CLA
Assistant bot will ask you to sign by commenting on the PR. This is a one-time
process.
The CLA allows us to distribute Turnstone under both open-source and commercial
licenses. Your contributions remain your own — you are granting a license, not
transferring ownership.
## Getting Started
1. Fork the repository
2. Create a branch for your change
3. Make your changes
4. Run the tests: `pytest`
5. Open a pull request
## Development Setup
```
python -m venv .venv
source .venv/bin/activate
pip install -e ".[test]"
```
## Guidelines
- Keep pull requests focused — one change per PR
- Add tests for new functionality
- Follow existing code style and patterns
- Update documentation if your change affects user-facing behavior
## Reporting Issues
Open an issue at https://github.com/turnstonelabs/turnstone/issues with:
- What you expected to happen
- What actually happened
- Steps to reproduce
- Python version and OS
## License
By contributing, you agree that your contributions will be licensed under the
project's [Business Source License 1.1](LICENSE).
Dockerfile
+46
View File
@@ -0,0 +1,46 @@
# =============================================================================
# Turnstone — multi-stage Docker build
# Single image for all services: server, bridge, console, sim, eval
# =============================================================================
# ----------------------------------------------------------------------------
# Stage 1: Builder — build the wheel
# ----------------------------------------------------------------------------
FROM python:3.13-slim AS builder
WORKDIR /build
RUN pip install --no-cache-dir hatchling
COPY pyproject.toml README.md LICENSE ./
COPY turnstone/ turnstone/
RUN pip wheel --no-deps --wheel-dir /build/wheels .
# ----------------------------------------------------------------------------
# Stage 2: Runtime — slim image with the installed package
# ----------------------------------------------------------------------------
FROM python:3.13-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
# Non-root user
RUN useradd --create-home --shell /bin/bash turnstone
# Install the wheel with all optional extras (redis for mq/console/sim)
COPY --from=builder /build/wheels/*.whl /tmp/wheels/
RUN pip install --no-cache-dir "$(ls /tmp/wheels/*.whl)[mq,console,sim]" \
&& rm -rf /tmp/wheels
# Health check script (stdlib only, no pip deps needed)
COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
# Data directory — SQLite DB is created in CWD
WORKDIR /data
RUN chown turnstone:turnstone /data
USER turnstone
# Default command (overridden per service in compose.yaml)
CMD ["turnstone-server", "--host", "0.0.0.0", "--port", "8080"]
LICENSE
+62
View File
@@ -0,0 +1,62 @@
License text copyright (c) 2020 MariaDB Corporation Ab, All Rights Reserved.
"Business Source License" is a trademark of MariaDB Corporation Ab.
Parameters
Licensor: Patrick Buckley
Licensed Work: Turnstone 0.2.0. The Licensed Work is (c) 2025-2026 Patrick Buckley.
Additional Use Grant: You may make production use of the Licensed Work, provided
your use does not include providing the Licensed Work to third
parties as a hosted or managed service, where the service
provides users with access to any substantial set of the
features or functionality of the Licensed Work.
Change Date: 2030-03-01
Change License: Apache License, Version 2.0
For information about alternative licensing arrangements for the Licensed Work,
please contact buckleypm@gmail.com.
Notice
Business Source License 1.1
Terms
The Licensor hereby grants you the right to copy, modify, create derivative
works, redistribute, and make non-production use of the Licensed Work. The
Licensor may make an Additional Use Grant, above, permitting limited production use.
Effective on the Change Date, or the fourth anniversary of the first publicly
available distribution of a specific version of the Licensed Work under this
License, whichever comes first, the Licensor hereby grants you rights under
the terms of the Change License, and the rights granted in the paragraph
above terminate.
If your use of the Licensed Work does not comply with the requirements
currently in effect as described in this License, you must purchase a
commercial license from the Licensor, its affiliated entities, or authorized
resellers, or you must refrain from using the Licensed Work.
All copies of the original and modified Licensed Work, and derivative works
of the Licensed Work, are subject to this License. This License applies
separately for each version of the Licensed Work and the Change Date may vary
for each version of the Licensed Work released by Licensor.
You must conspicuously display this License on each original or modified copy
of the Licensed Work. If you receive the Licensed Work in original or
modified form from a third party, the terms and conditions set forth in this
License apply to your use of that work.
Any use of the Licensed Work in violation of this License will automatically
terminate your rights under this License for the current and all other
versions of the Licensed Work.
This License does not grant you any right in any trademark or logo of
Licensor or its affiliates (provided that you may use a trademark or logo of
Licensor as expressly required by this License).
TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
TITLE.
README.md
+264
View File
@@ -0,0 +1,264 @@
# Turnstone
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers, driven by message queues or interactive interfaces.
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) — a bird that flips rocks to expose what's hiding underneath.
## What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. It runs as:
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
- **Multi-node clusters** — generic work load-balances across nodes, directed work routes to a specific server
- **Cluster dashboard** — real-time view of all nodes, workstreams, and resource utilization
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
```
External System → Message Queue → Bridge (per node) → Turnstone Server → LLM + Tools
↓
Pub/Sub → Progress Events → External System
↓
turnstone-console → Cluster Dashboard (browser)
```
## Quickstart
### Interactive (terminal)
```bash
pip install turnstone
turnstone --base-url http://localhost:8000/v1
```
### Interactive (browser)
```bash
turnstone-server --port 8080 --base-url http://localhost:8000/v1
```
### Queue-driven (programmatic)
```bash
pip install turnstone[mq]
turnstone-bridge --server-url http://localhost:8080 --redis-host localhost
```
```python
from turnstone.mq import TurnstoneClient
with TurnstoneClient() as client:
# Generic — any available node picks it up
result = client.send_and_wait("Analyze the error logs", auto_approve=True)
print(result.content)
# Directed — must run on a specific server
result = client.send_and_wait(
"Check disk I/O on this server",
target_node="server-12",
auto_approve=True,
)
```
### Cluster dashboard
```bash
pip install turnstone[console]
turnstone-console --redis-host localhost --port 8090
```
Then open `http://localhost:8090` for the cluster-wide dashboard.
### Docker
```bash
cp .env.example .env # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose up # starts redis + server + bridge + console
```
Console dashboard at http://localhost:8090. See [docs/docker.md](docs/docker.md) for configuration, scaling, and profiles.
### Simulator
Test the multi-node stack at scale without an LLM backend:
```bash
docker compose --profile sim up redis console sim
```
Or standalone:
```bash
pip install turnstone[sim]
turnstone-sim --nodes 100 --scenario steady --duration 60 --mps 10
```
See [docs/simulator.md](docs/simulator.md) for scenarios, CLI reference, and metrics.
All frontends connect to any OpenAI-compatible API (vLLM, NVIDIA NIM/NGC, llama.cpp, OpenAI, etc.) and auto-detect the model.
## Architecture
```
turnstone/
├── core/ # UI-agnostic engine
│ ├── session.py # ChatSession — multi-turn loop, tool dispatch, agents
│ ├── tools.py # Tool definitions (auto-loaded from JSON)
│ ├── workstream.py # WorkstreamManager — parallel independent sessions
│ ├── config.py # Unified TOML config (~/.config/turnstone/config.toml)
│ ├── memory.py # SQLite persistence (memories, conversations, FTS5)
│ ├── metrics.py # Prometheus-compatible metrics collector
│ ├── edit.py # File editing (fuzzy match, indentation)
│ ├── safety.py # Path validation, sandbox checks
│ ├── sandbox.py # Command sandboxing
│ └── web.py # Web fetch/search helpers
├── mq/ # Message queue integration
│ ├── protocol.py # Typed message dataclasses (JSON serialization)
│ ├── broker.py # Abstract MessageBroker + RedisBroker
│ ├── bridge.py # Bridge service (queue ↔ HTTP API, multi-node routing)
│ └── client.py # TurnstoneClient — Python API for external systems
├── console/ # Cluster dashboard
│ ├── collector.py # ClusterCollector — aggregates all nodes via Redis + HTTP
│ ├── server.py # Dashboard HTTP server + SSE
│ └── static/ # Cluster dashboard web UI
├── tools/ # Tool schemas (one JSON file per tool)
├── ui/ # Frontend assets and terminal rendering
│ └── static/ # Web UI (HTML, CSS, JS)
├── sim/ # Cluster simulator
│ ├── cluster.py # SimCluster — orchestrates N nodes + dispatchers
│ ├── node.py # SimNode + SimWorkstream — protocol-compatible node
│ ├── engine.py # LLM + tool execution simulation
│ ├── scenario.py # 5 workload scenarios (steady, burst, node_failure, …)
│ ├── metrics.py # Latency, throughput, utilization collection
│ └── cli.py # CLI entry point (turnstone-sim)
├── cli.py # Terminal frontend (+ /cluster commands for console)
├── server.py # Web frontend (HTTP + SSE)
└── eval.py # Evaluation and prompt optimization harness
docs/
├── architecture.md # System architecture and threading model
├── api-reference.md # Web server API and SSE event reference
├── console.md # Cluster dashboard service (turnstone-console)
├── docker.md # Docker Compose deployment and configuration
├── simulator.md # Cluster simulator usage and scenarios
├── tools.md # Tool schemas, execution pipeline, approval flow
└── eval.md # Evaluation harness internals
```
## Multi-node routing
Each Turnstone server runs a bridge process. Bridges share a Redis instance for coordination:
| Redis Key | Purpose |
|-----------|---------|
| `turnstone:inbound` | Shared work queue — generic tasks, any node |
| `turnstone:inbound:{node_id}` | Per-node queue — directed tasks |
| `turnstone:ws:{ws_id}` | Workstream ownership — auto-routes follow-ups |
| `turnstone:node:{node_id}` | Node heartbeat + metadata for discovery |
| `turnstone:events:{ws_id}` | Per-workstream event pub/sub |
| `turnstone:events:global` | Global event pub/sub |
| `turnstone:events:cluster` | Cluster-wide state changes (for turnstone-console) |
**Routing rules:**
1. Message has `target_node` → routes to that node's queue
2. Message has `ws_id` → looks up owner, routes to owning node
3. Neither → shared queue, next available bridge picks it up
Bridges BLPOP from their per-node queue (priority) then the shared queue. Directed work always takes precedence.
## Tools
14 built-in tools, 2 agent tools:
| Tool | Description | Auto-approved |
|------|-------------|:---:|
| `bash` | Execute shell commands | |
| `read_file` | Read file contents | yes |
| `write_file` | Write/create files | |
| `edit_file` | Fuzzy-match file editing | |
| `search` | Search files by name/content | yes |
| `math` | Sandboxed Python evaluation | |
| `man` | Read man pages | yes |
| `web_fetch` | Fetch URL content | |
| `web_search` | Search via Tavily API | |
| `remember` | Save persistent facts | yes |
| `recall` | Search memories and history | yes |
| `forget` | Remove a memory | yes |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
## Configuration
All entry points read `~/.config/turnstone/config.toml`. CLI flags override config values.
```toml
[api]
base_url = "http://localhost:8000/v1"
api_key = ""
tavily_key = ""
[model]
name = "" # empty = auto-detect
temperature = 0.5
reasoning_effort = "medium"
[tools]
timeout = 30
skip_permissions = false
[server]
host = "0.0.0.0"
port = 8080
[redis]
host = "localhost"
port = 6379
password = ""
[bridge]
server_url = "http://localhost:8080"
node_id = "" # empty = hostname_xxxx
[console]
host = "0.0.0.0"
port = 8090
url = "http://localhost:8090" # used by CLI /cluster commands
poll_interval = 10
```
Precedence: CLI args > environment variables > config.toml > defaults.
## Workstreams
Parallel independent conversations, each with its own session and state:
| Symbol | State | Meaning |
|--------|-------|---------|
| `·` | idle | Waiting for input |
| `◌` | thinking | Model is generating |
| `▸` | running | Tool execution in progress |
| `◆` | attention | Waiting for approval |
| `✖` | error | Something went wrong |
Idle workstreams are automatically cleaned up after 2 hours (configurable). In multi-node deployments, workstream ownership is tracked in Redis — follow-up messages auto-route to the owning node.
## Monitoring
`/metrics` endpoint exposes Prometheus-format metrics:
- `turnstone_tokens_total{direction}` — prompt/completion token counters
- `turnstone_tool_calls_total{tool}` — per-tool invocation counts
- `turnstone_workstream_context_ratio{ws_id}` — per-workstream context utilization
- `turnstone_http_request_duration_seconds` — request latency histogram
- `turnstone_workstreams_by_state{state}` — workstream state gauges
Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
## Requirements
- Python 3.11+
- An OpenAI-compatible API endpoint ([vLLM](https://github.com/vllm-project/vllm), [NVIDIA NIM](https://build.nvidia.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), etc.)
- Redis (for message queue bridge — `pip install turnstone[mq]`)
## License
[Business Source License 1.1](LICENSE) — free for all use except hosting as a managed service. Converts to Apache 2.0 on 2030-03-01.
compose.yaml
+194
View File
@@ -0,0 +1,194 @@
# =============================================================================
# Turnstone Docker Compose Stack
#
# Usage:
# Full stack: docker compose up
# With simulator: docker compose --profile sim up
# Sim only: docker compose --profile sim up redis console sim
# Scale bridges: docker compose up --scale bridge=3
# =============================================================================
name: turnstone
networks:
turnstone-net:
driver: bridge
volumes:
redis-data:
turnstone-data:
services:
# -------------------------------------------------------------------
# Redis — message broker, pub/sub, node registry
# -------------------------------------------------------------------
redis:
image: redis:7-alpine
command:
- sh
- -c
- >-
redis-server
--save 60 1
--loglevel warning
$${REDIS_PASSWORD:+--requirepass $$REDIS_PASSWORD}
ports:
- "${REDIS_PORT:-6379}:6379"
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
volumes:
- redis-data:/data
networks:
- turnstone-net
healthcheck:
test:
- CMD-SHELL
- redis-cli $${REDIS_PASSWORD:+-a $$REDIS_PASSWORD} ping | grep -q PONG
interval: 5s
timeout: 3s
retries: 5
start_period: 5s
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-server — Web UI + chat workstreams + LLM interaction
# -------------------------------------------------------------------
server:
build:
context: .
dockerfile: Dockerfile
command:
- sh
- -c
- >-
turnstone-server
--host 0.0.0.0
--port 8080
--base-url "$${LLM_BASE_URL}"
--api-key "$${OPENAI_API_KEY}"
$${SKIP_PERMISSIONS:+--skip-permissions}
ports:
- "${SERVER_PORT:-8080}:8080"
volumes:
- turnstone-data:/data
environment:
- LLM_BASE_URL=${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
- OPENAI_API_KEY=${OPENAI_API_KEY:-dummy}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SKIP_PERMISSIONS=${SKIP_PERMISSIONS:-}
- TURNSTONE_AUTH_ENABLED=${TURNSTONE_AUTH_ENABLED:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- turnstone-net
depends_on:
redis:
condition: service_healthy
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-bridge — Redis <-> HTTP bridge for multi-node routing
# Node ID auto-generated from container hostname (no --node-id needed)
# -------------------------------------------------------------------
bridge:
build:
context: .
dockerfile: Dockerfile
command:
- turnstone-bridge
- --server-url=http://server:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-300}
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
networks:
- turnstone-net
depends_on:
server:
condition: service_healthy
redis:
condition: service_healthy
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-console — Cluster dashboard
# -------------------------------------------------------------------
console:
build:
context: .
dockerfile: Dockerfile
command:
- turnstone-console
- --host=0.0.0.0
- --port=8090
- --redis-host=redis
- --redis-port=6379
- --poll-interval=${CONSOLE_POLL_INTERVAL:-10}
ports:
- "${CONSOLE_PORT:-8090}:8090"
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_AUTH_ENABLED=${TURNSTONE_AUTH_ENABLED:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
networks:
- turnstone-net
depends_on:
redis:
condition: service_healthy
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8090/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 10s
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-sim — Multi-node cluster simulator (no LLM needed)
# Start with: docker compose --profile sim up
# -------------------------------------------------------------------
sim:
build:
context: .
dockerfile: Dockerfile
profiles:
- sim
command:
- sh
- -c
- >-
turnstone-sim
--nodes "$${SIM_NODES}"
--scenario "$${SIM_SCENARIO}"
--duration "$${SIM_DURATION}"
--mps "$${SIM_MPS}"
--redis-host redis
--redis-port 6379
--log-level "$${SIM_LOG_LEVEL}"
$${SIM_SEED:+--seed $$SIM_SEED}
$${SIM_METRICS_FILE:+--metrics-file $$SIM_METRICS_FILE}
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- SIM_NODES=${SIM_NODES:-100}
- SIM_SCENARIO=${SIM_SCENARIO:-steady}
- SIM_DURATION=${SIM_DURATION:-60}
- SIM_MPS=${SIM_MPS:-5.0}
- SIM_LOG_LEVEL=${SIM_LOG_LEVEL:-INFO}
- SIM_SEED=${SIM_SEED:-}
- SIM_METRICS_FILE=${SIM_METRICS_FILE:-}
networks:
- turnstone-net
depends_on:
redis:
condition: service_healthy
restart: "no"
demo.svg
+221
View File
@@ -0,0 +1,221 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" font-family="ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace" font-size="13">
<style>
@keyframes pulse-green { 0%,100% { opacity:0.5 } 50% { opacity:1 } }
@keyframes pulse-yellow { 0%,100% { opacity:0.4 } 50% { opacity:1 } }
@keyframes pulse-blue { 0%,100% { opacity:0.3 } 50% { opacity:1 } }
@keyframes fadein { from { opacity:0 } to { opacity:1 } }
.pg { animation: pulse-green 2s infinite }
.py { animation: pulse-yellow 1.8s infinite }
.pb { animation: pulse-blue 2.2s infinite }
.f1 { animation: fadein 0.4s 0.2s both }
.f2 { animation: fadein 0.4s 0.4s both }
.f3 { animation: fadein 0.4s 0.6s both }
.f4 { animation: fadein 0.4s 0.8s both }
.f5 { animation: fadein 0.4s 1.0s both }
.f6 { animation: fadein 0.4s 1.3s both }
.f7 { animation: fadein 0.4s 1.5s both }
.f8 { animation: fadein 0.4s 1.7s both }
.f9 { animation: fadein 0.4s 1.9s both }
.f10 { animation: fadein 0.4s 2.1s both }
.f11 { animation: fadein 0.4s 2.3s both }
.f12 { animation: fadein 0.4s 2.5s both }
</style>
<!-- Window chrome -->
<rect rx="10" width="860" height="520" fill="#1a1b26"/>
<rect width="860" height="36" rx="10" fill="#16161e"/>
<rect y="26" width="860" height="10" fill="#16161e"/>
<circle cx="20" cy="18" r="6" fill="#f7768e"/>
<circle cx="40" cy="18" r="6" fill="#e0af68"/>
<circle cx="60" cy="18" r="6" fill="#9ece6a"/>
<text x="430" y="22" text-anchor="middle" fill="#565f89" font-size="12">turnstone — console</text>
<!-- Header -->
<rect y="36" width="860" height="30" fill="#24283b"/>
<rect y="66" width="860" height="1" fill="#3b4261"/>
<text x="16" y="56" fill="#7aa2f7" font-size="14" font-weight="bold">turnstone console</text>
<text x="200" y="56" fill="#565f89" font-size="12">6 nodes · 10 workstreams</text>
<!-- ====== State cards ====== -->
<g transform="translate(16, 78)" class="f1" opacity="0">
<!-- RUN card -->
<rect x="0" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="0" y="0" width="156" height="3" rx="6" fill="#9ece6a"/>
<text x="78" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">3</text>
<text x="78" y="50" text-anchor="middle" fill="#565f89" font-size="10">▸ RUN</text>
<!-- THINK card -->
<rect x="168" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="168" y="0" width="156" height="3" rx="6" fill="#7aa2f7"/>
<text x="246" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">2</text>
<text x="246" y="50" text-anchor="middle" fill="#565f89" font-size="10">◌ THINK</text>
<!-- ATTN card -->
<rect x="336" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="336" y="0" width="156" height="3" rx="6" fill="#e0af68"/>
<text x="414" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">1</text>
<text x="414" y="50" text-anchor="middle" fill="#565f89" font-size="10">◆ ATTN</text>
<!-- ERR card -->
<rect x="504" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="504" y="0" width="156" height="3" rx="6" fill="#f7768e"/>
<text x="582" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">0</text>
<text x="582" y="50" text-anchor="middle" fill="#565f89" font-size="10">✖ ERR</text>
<!-- IDLE card -->
<rect x="672" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="672" y="0" width="156" height="3" rx="6" fill="#565f89"/>
<text x="750" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">4</text>
<text x="750" y="50" text-anchor="middle" fill="#565f89" font-size="10">· IDLE</text>
</g>
<!-- Aggregate bar -->
<text x="16" y="160" fill="#565f89" font-size="11" class="f2" opacity="0">197k tokens · 42 tool calls</text>
<!-- ====== NODES section ====== -->
<text x="16" y="182" fill="#7aa2f7" font-size="12" font-weight="bold" class="f3" opacity="0">NODES</text>
<!-- Node column headers -->
<g transform="translate(0, 190)" class="f4" opacity="0">
<rect width="860" height="20" fill="#24283b"/>
<rect y="20" width="860" height="1" fill="#3b4261"/>
<text y="14" fill="#565f89" font-size="10" letter-spacing="0.5">
<tspan x="36">NODE</tspan>
<tspan x="560">WS</tspan>
<tspan x="610">RUN</tspan>
<tspan x="660">ATTN</tspan>
<tspan x="710">TOKENS</tspan>
<tspan x="790">LOAD</tspan>
</text>
</g>
<!-- Node rows -->
<g transform="translate(0, 214)">
<!-- Node 1: db-west-04 — 3 ws, 1 running, has-running bar -->
<g class="f5" opacity="0">
<rect y="0" width="860" height="38" fill="#1a1b26"/>
<rect y="0" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="19" r="4" fill="#9ece6a"/>
<text x="36" y="23" fill="#a9b1d6" font-size="12" font-weight="bold">db-west-04</text>
<text x="566" y="23" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="23" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="23" fill="#565f89" font-size="11">0</text>
<text x="710" y="23" fill="#565f89" font-size="11">57.6k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="15" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="15" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="23" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 2: api-east-01 — 3 ws, 1 attention, has-attention bar -->
<g class="f6" opacity="0">
<rect y="40" width="860" height="38" fill="#24283b"/>
<rect y="40" width="3" height="38" fill="#e0af68"/>
<circle cx="22" cy="59" r="4" fill="#9ece6a"/>
<text x="36" y="63" fill="#a9b1d6" font-size="12" font-weight="bold">api-east-01</text>
<text x="566" y="63" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="63" fill="#565f89" font-size="11">0</text>
<text x="666" y="63" fill="#a9b1d6" font-size="11">1</text>
<text x="710" y="63" fill="#565f89" font-size="11">109k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="55" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="55" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="63" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 3: sre-node-03 — 2 ws, 1 running, has-running bar -->
<g class="f7" opacity="0">
<rect y="80" width="860" height="38" fill="#1a1b26"/>
<rect y="80" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="99" r="4" fill="#9ece6a"/>
<text x="36" y="103" fill="#a9b1d6" font-size="12" font-weight="bold">sre-node-03</text>
<text x="566" y="103" fill="#a9b1d6" font-size="11">2</text>
<text x="616" y="103" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="103" fill="#565f89" font-size="11">0</text>
<text x="710" y="103" fill="#565f89" font-size="11">64.4k</text>
<!-- Load bar: 2/10 = 20% -->
<rect x="770" y="95" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="95" width="12" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="103" fill="#565f89" font-size="11">20%</text>
</g>
<!-- Node 4: analytics-02 — 1 ws, thinking, has-thinking bar -->
<g class="f8" opacity="0">
<rect y="120" width="860" height="38" fill="#24283b"/>
<rect y="120" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="139" r="4" fill="#9ece6a"/>
<text x="36" y="143" fill="#a9b1d6" font-size="12" font-weight="bold">analytics-02</text>
<text x="566" y="143" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="143" fill="#565f89" font-size="11">0</text>
<text x="666" y="143" fill="#565f89" font-size="11">0</text>
<text x="710" y="143" fill="#565f89" font-size="11">18.3k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="135" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="135" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="143" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 5: data-ops-05 — 1 ws, thinking, has-thinking bar -->
<g class="f9" opacity="0">
<rect y="160" width="860" height="38" fill="#1a1b26"/>
<rect y="160" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="179" r="4" fill="#9ece6a"/>
<text x="36" y="183" fill="#a9b1d6" font-size="12" font-weight="bold">data-ops-05</text>
<text x="566" y="183" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="183" fill="#565f89" font-size="11">0</text>
<text x="666" y="183" fill="#565f89" font-size="11">0</text>
<text x="710" y="183" fill="#565f89" font-size="11">8.7k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="175" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="175" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="183" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 6: ml-gpu-07 — 0 ws, empty, no bar -->
<g class="f10" opacity="0">
<rect y="200" width="860" height="38" fill="#24283b"/>
<rect y="200" width="3" height="38" fill="transparent"/>
<circle cx="22" cy="219" r="4" fill="#9ece6a"/>
<text x="36" y="223" fill="#a9b1d6" font-size="12" font-weight="bold">ml-gpu-07</text>
<text x="566" y="223" fill="#565f89" font-size="11">0</text>
<text x="616" y="223" fill="#565f89" font-size="11">0</text>
<text x="666" y="223" fill="#565f89" font-size="11">0</text>
<text x="710" y="223" fill="#565f89" font-size="11">0</text>
<!-- Load bar: 0/10 = 0% (empty track) -->
<rect x="770" y="215" width="60" height="6" rx="3" fill="#292e42"/>
<text x="842" y="223" fill="#565f89" font-size="11">0%</text>
</g>
</g>
<!-- ====== Footer ====== -->
<g transform="translate(0, 468)" class="f12" opacity="0">
<rect width="860" height="1" fill="#3b4261"/>
<rect y="1" width="860" height="24" fill="#16161e"/>
<circle cx="20" cy="14" r="3" fill="#9ece6a"/>
<text x="28" y="18" fill="#565f89" font-size="10">db-west-04</text>
<circle cx="120" cy="14" r="3" fill="#9ece6a"/>
<text x="128" y="18" fill="#565f89" font-size="10">api-east-01</text>
<circle cx="225" cy="14" r="3" fill="#9ece6a"/>
<text x="233" y="18" fill="#565f89" font-size="10">sre-node-03</text>
<circle cx="335" cy="14" r="3" fill="#9ece6a"/>
<text x="343" y="18" fill="#565f89" font-size="10">analytics-02</text>
<circle cx="450" cy="14" r="3" fill="#9ece6a"/>
<text x="458" y="18" fill="#565f89" font-size="10">data-ops-05</text>
<circle cx="560" cy="14" r="3" fill="#9ece6a"/>
<text x="568" y="18" fill="#565f89" font-size="10">ml-gpu-07</text>
<text x="680" y="18" fill="#3b4261" font-size="10">258k tokens · 42 calls · 12m</text>
</g>
<!-- Bottom edge -->
<rect y="493" width="860" height="27" fill="#16161e"/>
<rect y="510" width="860" height="10" rx="10" fill="#16161e"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docker/healthcheck.py
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env python3
"""Health check for turnstone containers.
Usage: healthcheck.py <url>
Exit 0 if the endpoint returns {"status": "ok"}, exit 1 otherwise.
Uses only stdlib — no pip dependencies required.
"""
import json
import sys
import urllib.request
def main() -> None:
if len(sys.argv) != 2:
print("Usage: healthcheck.py <url>", file=sys.stderr)
sys.exit(1)
url = sys.argv[1]
try:
req = urllib.request.Request(url, method="GET")
with urllib.request.urlopen(req, timeout=5) as resp:
data = json.loads(resp.read().decode())
if data.get("status") == "ok":
sys.exit(0)
print(f"Unhealthy: {data}", file=sys.stderr)
sys.exit(1)
except Exception as exc:
print(f"Health check failed: {exc}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
docs/api-reference.md
+702
View File
@@ -0,0 +1,702 @@
# turnstone Web Server API Reference
## Overview
`turnstone-server` exposes a browser-based chat UI backed by a Python stdlib HTTP
server (`socketserver.ThreadingMixIn` + `http.server.HTTPServer`). The server
uses **Server-Sent Events (SSE)** for real-time streaming and **HTTP POST** for
user actions.
All API responses use `Content-Type: application/json` unless otherwise noted.
CORS headers (`Access-Control-Allow-Origin: *`) are included on every response.
The server supports multiple concurrent **workstreams** (tabs), each backed by
an independent `ChatSession` and event queue.
---
## Endpoints
### `GET /`
Serves the embedded single-page application (HTML, CSS, and JavaScript inlined
in a single document). The SPA connects to the SSE and POST endpoints listed
below.
**Response:** `text/html; charset=utf-8`
---
### `GET /api/events?ws_id=<id>`
Opens a Server-Sent Events stream scoped to a single workstream. The connection
remains open indefinitely; the server pushes events as they occur.
**Query parameters:**
| Parameter | Type | Required | Description |
|-----------|--------|----------|----------------------------|
| `ws_id` | string | yes | Workstream identifier |
**Error:** Returns `404` with `{"error": "Unknown workstream"}` if `ws_id` is
not recognized.
#### Connection lifecycle
1. **`connected`** -- sent immediately on connect.
```json
{
"type": "connected",
"model": "kappa_20b_131k",
"skip_permissions": false
}
```
`skip_permissions` reflects the workstream's current auto-approve state. It is
`true` if the server was started with `--skip-permissions` or if the user chose
"Always approve" via the approval prompt during the session.
2. **`history`** -- replays the full conversation history so the client can
rebuild its UI.
```json
{
"type": "history",
"messages": [
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hi there!", "tool_calls": null},
{"role": "tool", "content": "..."}
]
}
```
Each message in the `messages` array has:
| Field | Type | Description |
|--------------|-------------------|-----------------------------------------------|
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
Each entry in `tool_calls`:
| Field | Type | Description |
|-------------|--------|------------------------------------|
| `name` | string | Function name (e.g. `"bash"`) |
| `arguments` | string | JSON-encoded argument string |
#### Streaming events
After the initial `connected` and `history` frames, the server streams
real-time events as the model generates a response:
**`thinking_start`** -- the model has begun generating (shown as a spinner).
```json
{"type": "thinking_start"}
```
**`thinking_stop`** -- the spinner phase is over.
```json
{"type": "thinking_stop"}
```
**`reasoning`** -- a chunk of chain-of-thought reasoning text.
```json
{"type": "reasoning", "text": "Let me think about this..."}
```
**`content`** -- a chunk of the assistant's visible reply.
```json
{"type": "content", "text": "Here is the answer: "}
```
**`stream_end`** -- the model has finished generating. The client should
finalize any in-progress assistant message.
```json
{"type": "stream_end"}
```
**`tool_info`** -- one or more tool calls that were auto-approved (no user
action required).
```json
{
"type": "tool_info",
"items": [
{
"header": "bash: ls -la",
"preview": "",
"func_name": "bash",
"approval_label": "bash",
"needs_approval": false,
"error": null
}
]
}
```
**`approve_request`** -- one or more tool calls that require user approval. The
client must respond via `POST /api/approve`.
```json
{
"type": "approve_request",
"items": [
{
"header": "bash: rm -rf /tmp/build",
"preview": "",
"func_name": "bash",
"approval_label": "bash",
"needs_approval": true,
"error": null
}
]
}
```
Each item in `items` (shared by `tool_info` and `approve_request`):
| Field | Type | Description |
|------------------|-------------|--------------------------------------------------|
| `header` | string | Human-readable header line for the tool call |
| `preview` | string | Diff or argument preview (may be empty) |
| `func_name` | string | Function name (e.g. `"bash"`, `"edit_file"`) |
| `approval_label` | string | Display label for the approval prompt |
| `needs_approval` | bool | Whether this call requires explicit approval |
| `error` | string/null | Error description if the call was malformed |
**`tool_result`** -- output from a completed tool execution.
```json
{"type": "tool_result", "name": "bash", "output": "file1.py\nfile2.py\n"}
```
**`status`** -- token usage statistics, sent after each model turn.
```json
{
"type": "status",
"prompt_tokens": 1024,
"completion_tokens": 256,
"total_tokens": 1280,
"context_window": 131072,
"pct": 1.0,
"effort": "medium"
}
```
| Field | Type | Description |
|---------------------|--------|----------------------------------------------|
| `prompt_tokens` | int | Tokens in the prompt |
| `completion_tokens` | int | Tokens generated by the model |
| `total_tokens` | int | `prompt_tokens + completion_tokens` |
| `context_window` | int | Total context window size in tokens |
| `pct` | float | Percentage of context window used |
| `effort` | string | Reasoning effort level (`low`/`medium`/`high`) |
**`plan_review`** -- the model is proposing a plan and wants feedback. The
client must respond via `POST /api/plan`.
```json
{"type": "plan_review", "content": "Step 1: ...\nStep 2: ..."}
```
**`info`** -- an informational message (e.g. command output).
```json
{"type": "info", "message": "Session cleared."}
```
**`error`** -- an error message.
```json
{"type": "error", "message": "Error: connection timed out"}
```
**`busy_error`** -- sent when a new message arrives while the model is already
processing.
```json
{"type": "busy_error", "message": "Already processing a request. Please wait."}
```
**`clear_ui`** -- instructs the client to clear all displayed messages (sent
after `/clear` or `/new` commands).
```json
{"type": "clear_ui"}
```
#### Keepalive
The server sends an SSE comment every 5 seconds when no events are pending:
```
: keepalive
```
This prevents proxies and browsers from closing the connection due to
inactivity.
#### Generation mechanism
Each new SSE connection to a workstream increments an internal
`_sse_generation` counter. The previous SSE handler detects the generation
mismatch and exits its event loop, ensuring only one active SSE connection per
workstream at a time. The event queue is drained of stale events before the new
connection begins streaming.
---
### `GET /api/events/global`
Opens a Server-Sent Events stream that broadcasts state-change events across
all workstreams. This is used by the tab bar to display per-workstream activity
indicators.
**Events:**
```json
{"type": "ws_state", "ws_id": "abc123", "state": "thinking"}
```
| Field | Type | Description |
|---------|--------|--------------------------|
| `ws_id` | string | Workstream identifier |
| `state` | string | Current workstream state |
Possible `state` values:
| State | Description |
|-------------|-------------------------------------------------|
| `idle` | No active processing |
| `thinking` | Model is generating a response |
| `running` | Tool execution in progress |
| `attention` | Waiting for user input (approval or plan review)|
| `error` | An error occurred |
**Fan-out pattern:** Each connected client receives its own bounded queue
(`maxsize=500`). A dedicated fan-out thread reads from the shared global queue
and copies each event to every client queue. If a client queue is full, the
event is silently dropped for that client.
**Keepalive:** Same as `/api/events` -- an SSE comment every 5 seconds.
---
### `GET /api/workstreams`
Returns a list of all active workstreams.
**Response:**
```json
{
"workstreams": [
{"id": "abc123", "name": "default", "state": "idle", "session_id": "a1b2c3d4e5f6"},
{"id": "def456", "name": "hacker-news", "state": "thinking", "session_id": "c5d6e7f8a9b0"}
]
}
```
Each workstream object:
| Field | Type | Description |
|--------------|-------------|--------------------------------------------------------|
| `id` | string | Unique workstream routing identifier |
| `name` | string | Display name (alias if set, otherwise `ws-xxxx`) |
| `state` | string | Current state (see state values above) |
| `session_id` | string/null | Session ID of the workstream's `ChatSession`, used for deduplication against `/api/sessions` |
---
### `GET /api/sessions`
Returns a list of saved sessions from the database, ordered by most recently
updated.
**Response:**
```json
{
"sessions": [
{
"session_id": "a1b2c3d4e5f6",
"alias": "refactor",
"title": "JWT Authentication Refactor",
"created": "2026-03-01 10:00:00",
"updated": "2026-03-01 11:30:00",
"message_count": 42
}
]
}
```
Each session object:
| Field | Type | Description |
|-----------------|-------------|--------------------------------------------|
| `session_id` | string | Unique 12-char hex session identifier |
| `alias` | string/null | User-assigned short name |
| `title` | string/null | LLM-generated title |
| `created` | string | ISO timestamp of session creation |
| `updated` | string | ISO timestamp of last message |
| `message_count` | int | Number of messages in the session |
---
### `POST /api/send`
Sends a user message to a workstream. Spawns a daemon worker thread that calls
`session.send()` and streams results back via the SSE channel.
**Request body:**
```json
{"message": "Explain how the server works", "ws_id": "abc123"}
```
| Field | Type | Required | Description |
|-----------|--------|----------|-------------------------|
| `message` | string | yes | The user's message text |
| `ws_id` | string | yes | Target workstream ID |
**Response (success):**
```json
{"status": "ok"}
```
**Response (busy):** Returned if the workstream's worker thread is still alive
from a previous request. Also pushes a `busy_error` event to the SSE stream.
```json
{"status": "busy"}
```
**Error responses:**
| Status | Body | Condition |
|--------|------------------------------------|------------------------|
| 400 | `{"error": "Empty message"}` | Message is empty |
| 404 | `{"error": "Unknown workstream"}` | `ws_id` not found |
---
### `POST /api/approve`
Responds to a tool approval request. The SSE stream must have previously sent
an `approve_request` event for the given workstream.
**Request body:**
```json
{"approved": true, "feedback": null, "always": false, "ws_id": "abc123"}
```
| Field | Type | Required | Description |
|------------|-------------|----------|--------------------------------------------------|
| `approved` | bool | yes | `true` to approve, `false` to deny |
| `feedback` | string/null | no | Optional feedback text (sent as denial reason) |
| `always` | bool | no | If `true` and `approved`, enables auto-approve |
| `ws_id` | string | yes | Target workstream ID |
When `always` is `true` and `approved` is `true`, the workstream's WebUI
instance sets `auto_approve = True`, causing all subsequent tool calls to be
automatically approved without prompting.
**Response:**
```json
{"status": "ok"}
```
**Error:** `404` with `{"error": "Unknown workstream"}` if `ws_id` is invalid.
---
### `POST /api/plan`
Responds to a plan review dialog. The SSE stream must have previously sent a
`plan_review` event for the given workstream.
**Request body:**
```json
{"feedback": "", "ws_id": "abc123"}
```
| Field | Type | Required | Description |
|------------|--------|----------|---------------------------------------------------------|
| `feedback` | string | yes | Feedback text; empty string means approval |
| `ws_id` | string | yes | Target workstream ID |
To approve the plan, send an empty string for `feedback`. To reject or request
changes, send a non-empty feedback string (e.g. `"reject"` or specific
revision instructions).
**Response:**
```json
{"status": "ok"}
```
**Error:** `404` with `{"error": "Unknown workstream"}` if `ws_id` is invalid.
---
### `POST /api/command`
Executes a slash command in the given workstream.
**Request body:**
```json
{"command": "/clear", "ws_id": "abc123"}
```
| Field | Type | Required | Description |
|-----------|--------|----------|------------------------------------|
| `command` | string | yes | The slash command (e.g. `/clear`) |
| `ws_id` | string | yes | Target workstream ID |
If the command is `/clear` or `/new`, the server pushes a `clear_ui` SSE event
to instruct the client to reset its message display. If the command is
`/resume`, the server pushes `clear_ui` followed by a `history` event
containing the resumed session's messages.
**Response:**
```json
{"status": "ok"}
```
**Error responses:**
| Status | Body | Condition |
|--------|------------------------------------|----------------------|
| 400 | `{"error": "Empty command"}` | Command is empty |
| 404 | `{"error": "Unknown workstream"}` | `ws_id` not found |
---
### `POST /api/workstreams/new`
Creates a new workstream. The server supports up to 10 concurrent workstreams.
**Request body:**
```json
{}
```
No fields are required. The body can be empty or an empty JSON object.
**Response (success):**
```json
{"ws_id": "ghi789", "name": "ws-3"}
```
| Field | Type | Description |
|---------|--------|------------------------------------|
| `ws_id` | string | Unique ID of the new workstream |
| `name` | string | Auto-generated workstream name |
**Error (limit reached):**
```json
{"error": "Maximum of 10 workstreams reached"}
```
Status code: `400`
---
### `POST /api/workstreams/close`
Closes and removes a workstream. The last remaining workstream cannot be
closed.
**Request body:**
```json
{"ws_id": "abc123"}
```
| Field | Type | Required | Description |
|---------|--------|----------|---------------------------|
| `ws_id` | string | yes | Workstream ID to close |
**Response (success):**
```json
{"status": "ok"}
```
**Error (last workstream):**
```json
{"error": "Cannot close last workstream"}
```
Status code: `400`
---
### `OPTIONS` (any path)
Handles CORS preflight requests.
**Response headers:**
```
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type
```
Status code: `200` with an empty body.
---
## Error Handling
| Condition | Behavior |
|------------------------------------|------------------------------------------------------------|
| Malformed or unparseable JSON body | Treated as an empty dict `{}`; missing fields use defaults |
| Unknown `ws_id` | `404` with `{"error": "Unknown workstream"}` |
| Unknown path (GET or POST) | `404` with plain-text body `Not found` |
| Empty `message` on `/api/send` | `400` with `{"error": "Empty message"}` |
| Empty `command` on `/api/command` | `400` with `{"error": "Empty command"}` |
---
## SSE Reconnection
The embedded JavaScript client implements exponential backoff for SSE
reconnection:
| Parameter | Value |
|--------------------|-------------------------------------------|
| Base delay | 1 second |
| Backoff multiplier | 2x on each consecutive failure |
| Maximum delay | 30 seconds |
| Reset | Delay resets to 1 second on first success |
On reconnect, the server replays the full conversation history via the
`history` event, so the client can rebuild its UI state without data loss. The
same reconnection strategy applies to both the per-workstream SSE stream
(`/api/events`) and the global state stream (`/api/events/global`).
---
## Observability
### `GET /health`
Returns server health status. Always returns `200 OK` while the server process
is running. Suitable for load-balancer health checks and Kubernetes liveness
probes.
**Response:** `application/json`
```json
{
"status": "ok",
"version": "0.2.0",
"uptime_seconds": 3614.72,
"model": "llama-3.1-70b-instruct",
"workstreams": {
"total": 2,
"idle": 1,
"thinking": 1,
"running": 0,
"attention": 0,
"error": 0
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `status` | string | Always `"ok"` while server is running |
| `version` | string | turnstone server version |
| `uptime_seconds` | number | Seconds since the server process started |
| `model` | string | Model name detected or configured at startup |
| `workstreams.total` | integer | Total active workstreams |
| `workstreams.idle` | integer | Workstreams waiting for user input |
| `workstreams.thinking` | integer | Workstreams with LLM currently streaming |
| `workstreams.running` | integer | Workstreams executing tools |
| `workstreams.attention` | integer | Workstreams blocked on approval or plan review |
| `workstreams.error` | integer | Workstreams in error state |
---
### `GET /metrics`
Returns operational metrics in **Prometheus text exposition format v0.0.4**.
Compatible with Prometheus `scrape_configs`, VictoriaMetrics, Grafana Agent,
and any other OpenMetrics-compatible collector.
**Response:** `text/plain; version=0.0.4; charset=utf-8`
#### Prometheus scrape config example
```yaml
scrape_configs:
- job_name: turnstone
static_configs:
- targets: ["localhost:8080"]
metrics_path: /metrics
```
#### Metrics reference
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `turnstone_build_info` | gauge | `version`, `model` | Always 1; carries version/model as labels |
| `turnstone_uptime_seconds` | gauge | — | Seconds since server start |
| `turnstone_workstreams_active_total` | gauge | — | Number of active workstreams |
| `turnstone_workstreams_by_state` | gauge | `state` | Workstream count per state (`idle`, `thinking`, `running`, `attention`, `error`) |
| `turnstone_http_requests_total` | counter | `method`, `endpoint`, `status_code` | Total HTTP requests handled |
| `turnstone_http_request_duration_seconds` | histogram | `method`, `endpoint` | Request latency distribution (11 buckets: 5ms–10s) |
| `turnstone_messages_sent_total` | counter | — | User messages dispatched to the AI |
| `turnstone_tokens_total` | counter | `type` | Tokens consumed (`type="prompt"` or `type="completion"`) |
| `turnstone_tool_calls_total` | counter | `tool` | Tool executions by name (e.g. `tool="bash"`) |
| `turnstone_errors_total` | counter | — | Errors reported by workstreams |
| `turnstone_context_window_used_ratio` | gauge | — | Last known fraction of context window in use (0.0–1.0) |
#### Example output
```
# HELP turnstone_build_info Server version and model info
# TYPE turnstone_build_info gauge
turnstone_build_info{version="0.2.0",model="llama-3.1-70b-instruct"} 1
# HELP turnstone_uptime_seconds Server uptime in seconds
# TYPE turnstone_uptime_seconds gauge
turnstone_uptime_seconds 3614.72
# HELP turnstone_workstreams_active_total Number of active workstreams
# TYPE turnstone_workstreams_active_total gauge
turnstone_workstreams_active_total 1
# HELP turnstone_http_requests_total Total HTTP requests handled
# TYPE turnstone_http_requests_total counter
turnstone_http_requests_total{method="GET",endpoint="/health",status_code="200"} 42
turnstone_http_requests_total{method="GET",endpoint="/metrics",status_code="200"} 7
turnstone_http_requests_total{method="POST",endpoint="/api/send",status_code="200"} 18
# HELP turnstone_tokens_total Total tokens consumed
# TYPE turnstone_tokens_total counter
turnstone_tokens_total{type="prompt"} 84320
turnstone_tokens_total{type="completion"} 12150
# HELP turnstone_tool_calls_total Total tool executions by name
# TYPE turnstone_tool_calls_total counter
turnstone_tool_calls_total{tool="bash"} 7
turnstone_tool_calls_total{tool="read_file"} 3
```
docs/architecture.md
+786
View File
@@ -0,0 +1,786 @@
# Turnstone Architecture
Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent
memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) and
gives the model 14 tools for reading, writing, searching, planning, and
executing code.
The core design principle is a **UI-agnostic engine with pluggable frontends**.
The engine (`ChatSession`) drives the conversation loop -- streaming, tool
dispatch, retry, compaction -- while every user-facing interaction is delegated
through the `SessionUI` protocol. Any frontend implements that protocol and
plugs in.
## Entry Points
| Command | Module | Frontend | Purpose |
|---------|--------|----------|---------|
| `turnstone` | `turnstone.cli` | `TerminalUI` | Interactive terminal REPL |
| `turnstone-server` | `turnstone.server` | `WebUI` | Browser-based chat (HTTP + SSE) |
| `turnstone-bridge` | `turnstone.mq.bridge` | Bridge | Message queue ↔ HTTP API bridge |
| `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
---
## Module Map
```
turnstone/
cli.py Terminal frontend (TerminalUI, WorkstreamTerminalUI, REPL)
server.py Web frontend (WebUI, HTTP handler, static-file serving)
eval.py Evaluation harness (HeadlessSession, scoring, prompt optimization)
core/
session.py ChatSession engine, SessionUI protocol, tool dispatch
workstream.py Parallel workstream manager (WorkstreamState, Workstream, WorkstreamManager)
tools.py Tool schema loader (JSON -> OpenAI function-calling format)
memory.py SQLite persistence (conversations, memories, FTS5 search)
metrics.py Prometheus-compatible metrics collector (MetricsCollector)
edit.py File edit utilities (find_occurrences, pick_nearest)
safety.py Command safety validation (blocked patterns, sanitization)
sandbox.py Math code sandboxing (AST validation, subprocess execution)
web.py Web utilities (HTML stripping, SSRF prevention)
mq/
protocol.py Inbound/outbound message dataclasses (JSON serialization)
broker.py Abstract MessageBroker protocol + RedisBroker
bridge.py Bridge service (queue ↔ turnstone-server HTTP API)
client.py TurnstoneClient library + TurnResult for external systems
console/
collector.py ClusterCollector — aggregates state from all nodes via Redis + HTTP
server.py Cluster dashboard HTTP server + SSE + CLI entry point
static/ Cluster dashboard web UI (HTML, CSS, JS)
ui/
colors.py ANSI color constants with NO_COLOR support
markdown.py Streaming terminal markdown renderer (line-buffered)
spinner.py Braille character spinner (daemon thread)
static/
index.html Single-page app shell (links to CSS and JS)
style.css All UI styles (dark/light themes, dashboard, approval blocks)
app.js All client-side JavaScript (SSE, workstreams, dashboard, markdown)
tools/
*.json 14 tool schemas (OpenAI function-calling format + turnstone metadata)
```
---
## Core Loop
A user message flows through the system as follows:
```
User input
|
v
ChatSession.send(user_input)
|
v
_full_messages() ------------> system_messages + self.messages
|
v
_emit_state("thinking")
|
v
_create_stream_with_retry() ----> client.chat.completions.create(stream=True)
| up to 3 retries (4 total attempts), exponential backoff
v
_stream_response(stream) --------> dispatch tokens to UI:
| on_reasoning_token() / on_content_token()
| accumulate tool_calls from deltas
| track finish_reason
v
finish_reason check:
+--- "length" --> warn, discard partial tool_calls
+--- "content_filter" --> warn
v
tool_calls present?
|
+--- No ---> _print_status_line() -> _emit_state("idle") -> return
|
+--- Yes --> _emit_state("running")
|
v
_execute_tools(tool_calls) <--- three-phase pipeline (see below)
|
v
append tool results to self.messages
|
v
loop back to _full_messages()
```
### Tool Execution Pipeline
Tool execution is a three-phase process:
```
Phase 1: PREPARE (serial)
For each tool_call:
_prepare_tool(tc)
-> parse JSON arguments (with regex fallback for malformed JSON)
-> dispatch to _prepare_{tool_name}(call_id, args)
-> validate inputs, build preview text
-> return item dict with: header, preview, needs_approval, execute fn
Phase 2: APPROVE (serial, blocking)
_emit_state("attention")
ui.approve_tools(items)
-> display all headers and previews
-> if any need approval and not auto_approve: prompt user
-> return (approved, feedback)
_emit_state("running")
Phase 3: EXECUTE (parallel)
if len(items) == 1:
run_one(items[0])
else:
ThreadPoolExecutor(max_workers=4).map(run_one, items)
For plan tool: post-execution gate via ui.on_plan_review()
```
### State Transitions
The engine emits state changes via `_emit_state()` which calls
`ui.on_state_change(state)`. Frontends use these to update indicators
(spinner, tab badges, status line).
```
send() called
|
v
"thinking" ---> streaming response
|
v
"running" ---> tool execution
|
v
"attention" ---> waiting for user approval / plan review
|
v
"running" ---> executing approved tools
|
v
"idle" ---> no more tool calls, turn complete
|
(or "error" ---> exception or KeyboardInterrupt)
```
---
## SessionUI Protocol
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 13
methods. Every frontend must implement all of them.
```python
class SessionUI(Protocol):
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
def on_content_token(self, text: str) -> None: ...
def on_stream_end(self) -> None: ...
def approve_tools(self, items: list[dict]) -> tuple[bool, str | None]: ...
def on_tool_result(self, name: str, output: str) -> None: ...
def on_status(self, usage: dict, context_window: int, effort: str) -> None: ...
def on_plan_review(self, content: str) -> str: ...
def on_info(self, message: str) -> None: ...
def on_error(self, message: str) -> None: ...
def on_state_change(self, state: str) -> None: ...
def on_rename(self, name: str) -> None: ... # propagate alias to tab/UI label
```
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
### Three Implementations
| Class | Module | Notes |
|-------|--------|-------|
| `TerminalUI` | `turnstone.cli` | ANSI colors, `MarkdownRenderer`, `Spinner`, readline-based `input()` for approval |
| `WebUI` | `turnstone.server` | SSE event queue per workstream, `threading.Event` for blocking on approval/plan |
| `NullUI` | `turnstone.eval` | Discards all output; `approve_tools` always returns `(True, None)` |
### WorkstreamTerminalUI
`WorkstreamTerminalUI` (in `turnstone.cli`) extends `TerminalUI` with workstream
awareness:
- **Output buffering**: When in background (`is_foreground` is False), tokens
are appended to `_output_buffer` instead of written to stdout. When the user
switches to this workstream, `flush_buffer()` replays them.
- **Approval blocking**: `approve_tools()` and `on_plan_review()` call
`_fg_event.wait()` when in background, blocking the worker thread until the
workstream is foregrounded. This ensures the user sees the approval prompt
in the correct context.
- **Foreground/background toggle**: `set_foreground(bool)` sets or clears
`_fg_event` (a `threading.Event`). The manager calls this during `/ws <N>`
switches.
---
## Workstream Architecture
Workstreams are parallel, independent chat sessions. Each has its own
`ChatSession`, `SessionUI`, message history, and worker thread.
### WorkstreamState
Defined in `turnstone.core.workstream.WorkstreamState` (5 states):
```
IDLE waiting for user input
THINKING LLM is streaming a response
RUNNING tools are executing
ATTENTION blocked on user approval or plan review
ERROR last operation failed
```
### Data Model
```python
@dataclass
class Workstream:
id: str # uuid hex, 8 chars
name: str # user-visible label
state: WorkstreamState # current state
session: ChatSession | None # the conversation engine
ui: SessionUI | None # frontend adapter
worker_thread: threading.Thread | None
error_message: str
last_active: float # time.monotonic() timestamp, updated on every state change
_lock: threading.Lock # per-workstream state lock
```
### WorkstreamManager
```python
class WorkstreamManager:
MAX_WORKSTREAMS = 10
def __init__(self, session_factory: Callable[[SessionUI], ChatSession]): ...
def create(self, name="", ui_factory=None) -> Workstream: ...
def close(self, ws_id: str) -> bool: ...
def close_idle(self, max_age_seconds: float) -> list[str]: ... # auto-close stale IDLE workstreams
def get(self, ws_id: str) -> Workstream | None: ...
def get_active(self) -> Workstream | None: ...
def list_all(self) -> list[Workstream]: ...
def switch(self, ws_id: str) -> Workstream | None: ...
def switch_by_index(self, index: int) -> Workstream | None: ...
def set_state(self, ws_id, state, error_msg=""): ... # updates last_active
```
The `session_factory` pattern decouples session creation from configuration.
The factory captures shared config (client, model, temperature, etc.) and
accepts only a `SessionUI`, so the manager can create sessions without knowing
API details.
### Idle Workstream Lifecycle
The web server runs a background `_idle_cleanup_thread` (daemon) that calls
`WorkstreamManager.close_idle()` periodically (every `timeout / 4`, max 5 min).
Any IDLE workstream whose `last_active` is older than the configured timeout is
closed; non-IDLE workstreams (THINKING, RUNNING, ATTENTION, ERROR) are never
touched. The last workstream is always preserved even if expired. On close, a
`ws_closed` event is broadcast on the global SSE channel so browser clients
remove the tab immediately. Controlled by `--workstream-idle-timeout` (default:
120 minutes, 0 = disable).
### CLI Workstreams
- `/ws list` -- show all workstreams with state indicators
- `/ws new [name]` -- create a new workstream and switch to it
- `/ws <N>` -- switch to workstream by 1-based index
- `/ws close [N]` -- close a workstream
- `/ws rename <name>` -- rename the active workstream
Background notifications: when a background workstream enters `ATTENTION`
state, `_bg_attention_notify` writes an ANSI escape sequence to stderr
(overwrites the line above the prompt) with the workstream name.
Status line: `_print_ws_status_line()` shows a compact status of all
non-idle background workstreams above the input prompt.
### Web Workstreams
- **Tab bar**: Each workstream renders as a tab with a colored state indicator
(CSS `@keyframes pulse` animation per state).
- **Per-tab SSE**: `connectContentSSE(wsId)` opens
`/api/events?ws_id=<id>` for the active tab's event stream.
- **Global SSE**: `connectGlobalSSE()` opens `/api/events/global` which
receives `ws_state` broadcasts from all workstreams, used to update tab
indicators without switching.
- **New tab / close**: POST `/api/workstreams/new`, POST `/api/workstreams/close`.
### Thread Safety
- `WorkstreamManager._lock`: guards `_workstreams` dict and `_order` list on
all create/close/switch/list operations.
- `Workstream._lock`: guards per-workstream state mutations in `set_state()`.
- `WorkstreamTerminalUI._print_lock`: guards `_output_buffer` access.
- `WorkstreamTerminalUI._fg_event`: `threading.Event` that blocks background
approval until the workstream is foregrounded.
---
## Tool System
### Schema Format
Each tool is a JSON file in `turnstone/tools/`. The file contains an OpenAI
function-calling schema (`name`, `description`, `parameters`) plus optional
turnstone metadata keys:
| Metadata Key | Type | Meaning |
|-------------|------|---------|
| `agent` | `bool` | Include this tool when running as a plan/task sub-agent |
| `task_agent` | `bool` | Include this tool when running as a task sub-agent |
| `auto_approve` | `bool` | Tool is read-only; skip user approval |
| `primary_key` | `str` | Fallback argument name for bare-string JSON recovery |
Example (`read_file.json`):
```json
{
"name": "read_file",
"description": "Read the contents of a file. ...",
"parameters": {
"type": "object",
"properties": {
"path": { "type": "string", "description": "..." },
"offset": { "type": "integer", "description": "..." },
"limit": { "type": "integer", "description": "..." }
},
"required": ["path"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "path"
}
```
At import time, `turnstone.core.tools._load_tools()` strips the metadata keys
from each schema and builds:
- `TOOLS` -- list of `{"type": "function", "function": {...}}` dicts for the API
- `AGENT_TOOLS` -- subset with `agent: true`
- `TASK_AGENT_TOOLS` -- subset with `task_agent: true`
- `AGENT_AUTO_TOOLS` / `TASK_AUTO_TOOLS` -- sets of tool names with `auto_approve: true`
- `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery
### 14 Tools by Category
**Read-only (auto-approve)**:
- `read_file` -- read file contents with optional offset/limit
- `search` -- ripgrep-based codebase search
- `man` -- read man pages
- `recall` -- retrieve stored memories
**Write (requires approval)**:
- `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`)
- `write_file` -- create or overwrite a file
- `edit_file` -- string replacement in an existing file (requires prior `read_file`)
- `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`)
- `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`)
- `web_search` -- search the web via Tavily API
**Agent (delegated sub-sessions)**:
- `task` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
- `plan` -- explore codebase and write a structured plan (`AGENT_TOOLS`)
**Memory (persistent key-value store)**:
- `remember` -- save a fact
- `forget` -- delete a fact
### Prepare / Execute Pattern
Every tool has a `_prepare_{name}` method and a corresponding `_exec_{name}`
method on `ChatSession`:
```
_prepare_bash(call_id, args) -> item dict with execute=self._exec_bash
_prepare_read_file(call_id, args) -> item dict with execute=self._exec_read_file
...
```
The prepare method validates inputs and builds the preview. The item dict
carries the validated data and a reference to the execute function. This
separation allows the UI to show previews before any side effects occur.
### Agent Tools
`task` and `plan` invoke `_run_agent()`, which runs a multi-turn loop with
a subset of tools and its own system prompt. The sub-agent runs
independently, then returns the final content as the tool result.
- **task**: uses `TASK_AGENT_TOOLS` (includes bash, read, write, edit, search)
- **plan**: uses `AGENT_TOOLS` (read-only subset for exploration). Writes output
to `.plan-<session_id>.md` — unique per `ChatSession` so concurrent workstreams
don't collide. On repeat invocations the prior `plan` tool call and its result
are forwarded from `self.messages` so the agent refines the existing plan rather
than starting over. Planning instructions are passed via `model_identity` in
`chat_template_kwargs` rather than as a developer message.
- **Turn limit**: controlled by `agent_max_turns` (default: `-1`, unlimited).
When a limit is set and reached, the agent is forced to synthesize a final
response without tools. When unlimited, the loop only exits when the model
stops calling tools or hits `finish_reason: "length"`.
- **Retry**: each API call in the agent loop uses the same retry+backoff logic
as the main `_create_stream_with_retry()`.
- **Finish reason handling**: `finish_reason: "length"` stops the agent early
and returns whatever content was generated. `finish_reason: "content_filter"`
returns a placeholder.
### Tool Output Truncation
Tool execution results (bash, read_file, search, math, man) are truncated by
`_truncate_output()` when they exceed `tool_truncation` characters. Truncation
preserves the first half and last half of the output, with a message in
between:
```
... [N chars truncated — output exceeded LIMIT char limit] ...
```
The default limit is 50% of the context window in characters (computed as
`context_window * chars_per_token * 0.5`). For a 131K context window this is
~262K characters. Override with `--tool-truncation <chars>`.
This truncation message is visible to the model, so it knows output was cut.
---
## Persistence
### Database
SQLite via `turnstone.core.memory`. Database file: `.turnstone.db` in the
current working directory (overridable via `memory.db_override` for eval
isolation).
### Tables
```sql
memories
key TEXT PRIMARY KEY
value TEXT NOT NULL
created TEXT NOT NULL
updated TEXT NOT NULL
sessions
session_id TEXT PRIMARY KEY
alias TEXT UNIQUE -- user-assigned short name (nullable)
title TEXT -- LLM-generated title (nullable)
created TEXT NOT NULL
updated TEXT NOT NULL -- bumped on every save_message()
conversations
id INTEGER PRIMARY KEY AUTOINCREMENT
session_id TEXT NOT NULL
timestamp TEXT NOT NULL
role TEXT NOT NULL -- user | assistant | tool_call | tool_result
content TEXT
tool_name TEXT
tool_args TEXT
tool_call_id TEXT -- links tool_call ↔ tool_result for resume
conversations_fts -- FTS5 virtual table
content (content=conversations, content_rowid=id)
```
The `tool_call_id` column was added via schema migration (`ALTER TABLE`) for
backwards compatibility with existing databases.
### Key Functions
| Function | Purpose |
|----------|---------|
| `open_db()` | Open/create database, run migrations, initialize tables |
| `load_memories()` | Return all `(key, value)` pairs sorted by key |
| `save_message(session_id, role, content, ...)` | Log a message to conversations (accepts `tool_call_id`) |
| `search_history(query, limit)` | Full-text search via FTS5 (falls back to LIKE) |
| `search_history_recent(limit)` | Return most recent messages |
| `register_session(session_id, title)` | Create a sessions row (no-op if exists) |
| `update_session_title(session_id, title)` | Set/update LLM-generated title |
| `set_session_alias(session_id, alias)` | Set user-friendly alias (returns False if taken) |
| `get_session_name(session_id)` | Return alias if set, else title, else None |
| `resolve_session(alias_or_id)` | Resolve alias, exact id, or id prefix to full session_id |
| `list_sessions(limit)` | List sessions with ≥1 message, ordered by updated DESC |
| `load_session_messages(session_id)` | Reconstruct OpenAI message format from DB rows |
| `delete_session(session_id)` | Delete session and all its messages |
| `prune_sessions(retention_days, log_fn)` | Remove empty sessions and old unnamed sessions; called at startup |
| `normalize_key(key)` | Normalize memory keys (`lower`, replace `-`/` ` with `_`) |
| `fts5_query(query)` | Convert plain text to safe FTS5 query (quoted terms) |
### Session Persistence and Resume
Each `ChatSession` generates a 12-char hex `_session_id` on creation and
registers it in the `sessions` table. Messages are saved to `conversations`
as they happen via `save_message()`.
**Auto-titling:** After the first complete exchange (user message + assistant
response), a background thread calls the LLM with a title-generation prompt
(`reasoning_effort: "low"`, `max_completion_tokens: 200`). The generated
title (3-8 words) is stored in `sessions.title`.
**Resume flow:** `ChatSession.resume_session(session_id)` calls
`load_session_messages()` which reconstructs the OpenAI message format from
database rows:
- `user` and `assistant` rows map directly
- Consecutive `tool_call` rows are grouped into one assistant message's
`tool_calls` array, paired with subsequent `tool_result` rows via
`tool_call_id` (or positional matching for legacy data)
- The session adopts the old `_session_id`, so new messages continue in
the same session
**`/clear` vs `/new`:** `/clear` wipes in-memory context but preserves
messages in the database for future resume. `/new` starts a fresh session
(new `_session_id`), leaving the old session resumable.
**Resolution:** `resolve_session()` accepts aliases, exact session IDs, or
session ID prefixes, enabling `turnstone --resume refactor` or `/resume abc12`.
**Session listing:** `list_sessions()` only returns sessions that have at
least one saved message (`WHERE EXISTS` on `conversations`). Sessions
registered but never used (e.g., from process startup) are invisible until
a message is sent.
**Session pruning:** `prune_sessions(retention_days, log_fn)` runs once at
startup (CLI and server). It removes:
- Sessions with no messages (orphaned registrations)
- Unnamed sessions (`alias IS NULL`) older than `retention_days` days (default 90)
Named (aliased) sessions are never age-pruned. Configure with
`--session-retention-days N` (0 = disable age pruning).
---
## Error Handling and Retry
### API Retry
`ChatSession._create_stream_with_retry()` (streaming path) and the agent
`_api_call()` (non-streaming) both use the same retry pattern:
- **Retries**: 4 total attempts (1 initial + 3 retries, `_MAX_RETRIES = 3`)
- **Backoff**: exponential, base 1 second (`delay = 1s * 2^attempt`)
- **Retryable errors**: `RateLimitError`, `APITimeoutError`,
`APIConnectionError`, `InternalServerError`, `ServiceUnavailableError`,
`APIError` (matched by class name to avoid importing backend-specific
exception hierarchies)
- On retry: `ui.on_info()` notification
- On final failure: exception propagates
`_compact_messages()` also wraps its non-streaming API call in the same
retry loop.
### Finish Reason Handling
`_stream_response()` tracks `finish_reason` from the final streaming chunk:
- **`"length"`**: warns via `ui.on_error()` that the response was truncated.
Any partial tool calls are discarded (their JSON would be malformed),
causing the `send()` loop to exit cleanly.
- **`"content_filter"`**: warns via `ui.on_error()` that the response was
blocked.
Agent sub-sessions (`_run_agent()`) check `finish_reason` on each
non-streaming response and stop the agent early on `"length"` or
`"content_filter"`.
`_compact_messages()` checks `finish_reason` on the compaction response and
warns if the summary was truncated.
### State Emission on Errors
- `send()` catches `KeyboardInterrupt` and generic `Exception`: calls
`_emit_state("error")` before re-raising
- On interrupt: partial tool results and the originating assistant message
are popped from `self.messages` to keep state consistent
### Web UI Resilience
- **SSE reconnect**: both `connectContentSSE()` and `connectGlobalSSE()` use
exponential backoff on `onerror` -- starting at 1 second, doubling on each
failure, capped at 30 seconds. On successful message, delay resets to 1s.
- **Disconnection indicator**: `#status-bar.disconnected` class turns the
status text red and shows "Reconnecting..."
- **Fetch error handling**: all `fetch()` calls use `.catch()` to prevent
unhandled promise rejections
- **Pending approval across tab switches**: `WebUI._pending_approval` stores
the `approve_request` event payload while the session is blocked waiting
for user response. On SSE reconnect (e.g., switching back to the tab),
the event is re-injected after history replay. `_build_history` marks the
pending tool call as `"pending": true` so `replayHistory` skips the
false `✓ approved` badge; the live approval UI is rendered by the
re-injected event instead.
- **Browser history integration**: `history.pushState` is called in
`switchTab()` with `{turnstone: 'workstream', wsId}`. The initial state is
seeded with `history.replaceState({turnstone: 'dashboard'})` on load. The
`popstate` listener restores the correct tab or shows the dashboard,
guarded by `_historyNavigation = true` to prevent re-entrant pushState.
### Eval Resilience
`_run_single_test()`: wraps `session.send_headless()` in a retry loop (3
attempts) to avoid transient API errors from poisoning evaluation scores.
---
## Threading Model
### CLI
```
Main thread Spinner thread (daemon) ThreadPoolExecutor
+--------------+ +------------------+ +-----------------+
| REPL loop | | Braille animation| | Tool execution |
| input() -> | | 80ms tick to | | max_workers=4 |
| send() -> | | stderr | | parallel tools |
| stream -> | | started/stopped | | run concurrently|
| tools -> | | by TerminalUI | | |
+--------------+ +------------------+ +-----------------+
| ^ ^
+-- on_thinking_start/stop -------------------------+
+-- _execute_tools ---------------------------------+
```
Key constraint: `input()` blocks the main thread. The spinner writes to
stderr so it does not interfere with readline. Tool execution may use a
`ThreadPoolExecutor` with up to 4 workers for parallel tool calls.
### Server
```
ThreadedHTTPServer (ThreadingMixIn + HTTPServer, daemon_threads=True)
|
+-- Thread per HTTP request
| POST /api/send -> worker thread per workstream
| POST /api/approve -> unblocks WebUI._approval_event
| POST /api/plan -> unblocks WebUI._plan_event
| POST /api/workstreams/new -> creates workstream + worker
| GET /api/events -> SSE long-poll (per workstream)
| GET /api/events/global -> SSE long-poll (fan-out)
|
+-- Worker thread per workstream
| Runs session.send() in a loop
| Blocks on WebUI._approval_event / _plan_event
|
+-- Global SSE fan-out
WebUI._global_queue shared across all WebUI instances
Global SSE endpoint drains this queue
```
`ThreadingMixIn` ensures each HTTP request (including long-lived SSE
connections) gets its own thread. This is necessary because SSE connections
block indefinitely, and POST requests must be handled concurrently.
Each workstream's `WebUI` has:
- `_event_queue` (per-workstream SSE events)
- `_approval_event` / `_plan_event` (`threading.Event` for blocking)
- `_global_queue` (class variable, shared, for state broadcasts)
### Workstream Threading (CLI)
```
Main thread Background workstream thread
+------------------+ +---------------------------+
| REPL input() | | session.send() |
| /ws commands | | streams response |
| active workstream| | executes tools |
| send() inline | | approve_tools() -> |
+------------------+ | _fg_event.wait() BLOCKS |
| +---------------------------+
| ^
+-- /ws <N> switch ------------->|
| old.set_foreground(False) |
| new.set_foreground(True) |
| new.flush_buffer() |
+-- _fg_event.set() unblocks --->+
```
When a background workstream needs approval, its `WorkstreamTerminalUI`
calls `_fg_event.wait()`, which blocks the worker thread until the user
switches to that workstream. The `_bg_attention_notify` callback writes a
bell + status line to stderr to alert the user.
### Message Queue Bridge
```
Main thread Global SSE thread Per-WS SSE threads (×N)
+------------------+ +------------------+ +-------------------+
| Inbound loop | | GET /events/glob | | GET /events?ws_id |
| BLPOP on Redis | | Parse SSE data | | Parse SSE data |
| Dispatch to | | Forward state | | Forward content, |
| handler | | changes | | tool results |
| POST to server | | Detect turn | | Handle approval |
| Publish ACK | | completion | | forwarding |
+------------------+ +------------------+ +-------------------+
| | |
+-- Redis inbound queue +-- Redis pub/sub +-- Redis pub/sub
(RPUSH/BLPOP) (PUBLISH) (PUBLISH)
+ response queue
(BLPOP on
approval)
```
**Approval flow:** When a per-WS SSE thread receives an `approve_request`, it checks
the workstream's `auto_approve_tools` set. If all requested tools are in the set, the
bridge auto-approves via `POST /api/approve`. Otherwise, it publishes an
`ApprovalRequestEvent` to the outbound channel with a `request_id`, then blocks on
`BLPOP` of a Redis response queue (`turnstone:resp:{request_id}`) until the client pushes
a response or the approval timeout (default 300s) expires.
**Completion detection:** The bridge tracks which `correlation_id` maps to which
`ws_id` for active sends. When the global SSE reports `ws_state → idle` for a tracked
workstream, the bridge emits a synthetic `TurnCompleteEvent` with the correlation ID.
**Multi-node routing:** Each bridge has a `node_id` (defaults to hostname) and BLPOPs
from both `turnstone:inbound:{node_id}` (directed, priority) and `turnstone:inbound` (shared).
Messages with `target_node` set are pushed to the target's per-node queue. Messages
for existing workstreams are auto-routed via `turnstone:ws:{ws_id}` ownership keys in Redis.
If a bridge picks up a shared-queue message for a workstream owned by another node, it
re-routes to that node's queue (1 extra hop). Bridges publish heartbeats to
`turnstone:node:{node_id}` with configurable TTL for node discovery.
### Cluster Console
```
Event subscriber Node discovery Poll loop
+------------------+ +------------------+ +-------------------+
| SUBSCRIBE on | | SCAN node:* keys | | For each node: |
| events:cluster | | every 15 seconds | | GET /api/dash |
| Apply state | | Add/remove nodes | | GET /health |
| changes to | | Emit join/lost | | ThreadPoolExecutor|
| in-memory model | | events | | (50 workers) |
+------------------+ +------------------+ +-------------------+
| | |
+-- Redis pub/sub +-- Redis SCAN +-- HTTP to each
(SUBSCRIBE) (every 15s) server (every 10s)
```
The console is read-only — it never writes to Redis queues or sends commands to servers.
Real-time events provide instant state transitions; periodic polling provides full data
consistency (tokens, context ratios, activity strings). See [docs/console.md](console.md)
for the full API reference.
---
## Conversation Compaction
When the prompt exceeds `auto_compact_pct` of the context window (default:
80%, configurable via `--auto-compact-pct`), `ChatSession` auto-compacts by
summarizing the entire conversation into a structured summary
(`_compact_messages`). The summary model call uses `compact_max_tokens`
(default: 32768, configurable via `--compact-max-tokens`). The summary
preserves:
- Decisions made (architecture, libraries, approaches)
- Files read, created, or modified
- Exact identifiers, paths, and code snippets
- Important tool results
- Open tasks
- User preferences
After compaction, `_read_files` is cleared to force re-reads before edits,
since file contents are no longer in the message history.
docs/console.md
+235
View File
@@ -0,0 +1,235 @@
# Cluster Dashboard (turnstone-console)
`turnstone-console` is a standalone monitoring service that provides cluster-wide visibility across all turnstone nodes. It connects to the shared Redis broker, discovers nodes via heartbeat keys, polls each node's HTTP API for workstream data, and subscribes to a cluster event channel for real-time state changes.
The console is read-only — it observes but does not own workstreams or drive LLM sessions.
## Architecture
```
turnstone-server ──→ turnstone-bridge ──→ Redis ──→ turnstone-console ──→ Browser
(per node) (per node) (shared) (one instance)
```
Each bridge publishes state changes to `{prefix}:events:cluster` on Redis pub/sub. The console subscribes once to that channel for real-time updates and periodically polls each node's `GET /api/dashboard` for full workstream snapshots.
### Data Sources
| Source | Method | Frequency | Data |
|--------|--------|-----------|------|
| Redis heartbeats | `SCAN turnstone:node:*` | Every 15s | Node discovery (node_id, server_url, started) |
| Redis pub/sub | `SUBSCRIBE turnstone:events:cluster` | Real-time | State changes, creates, closes, renames |
| Node HTTP API | `GET {server_url}/api/dashboard` | Every 10s | Full workstream list with tokens, context, activity |
| Node HTTP API | `GET {server_url}/health` | Every 10s | Node health status |
### Redis Key: Cluster Event Channel
Bridges publish to `{prefix}:events:cluster` whenever a workstream state change, creation, closure, or rename occurs. Events include `node_id` so the console can attribute them to the correct node.
Event types on the cluster channel:
| Event | Fields | Trigger |
|-------|--------|---------|
| `cluster_state` | ws_id, state, node_id, tokens, context_ratio, activity | Workstream state transition |
| `ws_created` | ws_id, name, node_id | New workstream created |
| `ws_closed` | ws_id | Workstream closed |
| `ws_rename` | ws_id, name | Workstream renamed |
---
## ClusterCollector
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Three daemon threads handle data acquisition:
1. **Event subscriber** — subscribes to `{prefix}:events:cluster` via `RedisBroker.subscribe_cluster()`. Applies state changes, creates, closes, and renames to the in-memory model immediately.
2. **Node discovery** — scans heartbeat keys every 15 seconds via `broker.list_nodes()`. Adds newly discovered nodes, removes expired ones, emits `node_joined` / `node_lost` events to SSE listeners.
3. **Poll loop** — fetches `GET /api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
### Thread Safety
All reads and writes to the node/workstream map are protected by a single `threading.Lock`. Query methods acquire the lock, copy data, and release before returning.
### Scale Considerations
- **10,000 workstreams** at ~500 bytes each = ~5 MB in memory
- **1,000 nodes** polled in parallel with 50 threads at ~100ms each = ~2 second poll cycle
- **Filtering and pagination** run in-memory on the full workstream list — sub-millisecond at this scale
- **SSE fan-out** uses the same per-client queue pattern as the per-node server — backed-up clients get events dropped, not blocking
---
## HTTP API
### `GET /api/cluster/overview`
Cluster-wide state counts and aggregate metrics.
```json
{
"nodes": 847,
"workstreams": 4219,
"states": {"running": 1847, "thinking": 312, "attention": 89, "idle": 1940, "error": 31},
"aggregate": {"total_tokens": 12400000, "total_tool_calls": 34200}
}
```
### `GET /api/cluster/nodes?sort=activity&limit=100&offset=0`
Paginated node list. Sort options: `activity` (default, by running+attention count), `tokens`, `name`.
```json
{
"nodes": [
{
"node_id": "db-west-04",
"server_url": "http://10.0.3.4:8080",
"ws_total": 6, "ws_running": 4, "ws_thinking": 0, "ws_attention": 1, "ws_idle": 1, "ws_error": 0,
"total_tokens": 48200,
"started": 1709294400.0,
"reachable": true,
"health": {}
}
],
"total": 847
}
```
### `GET /api/cluster/workstreams?state=running&node=db-west-04&search=perf&page=1&per_page=50`
Filtered, paginated workstream list. All query parameters are optional. `per_page` is capped at 200.
```json
{
"workstreams": [
{
"id": "a1b2c3d4", "name": "perf-db-west", "state": "running", "node": "db-west-04",
"title": "Query latency analysis", "tokens": 24100, "context_ratio": 0.18,
"activity": "bash: EXPLAIN ANALYZE...", "activity_state": "tool", "tool_calls": 42
}
],
"total": 1847, "page": 1, "per_page": 50, "pages": 37
}
```
### `GET /api/cluster/node/{node_id}`
Single node detail with all its workstreams.
```json
{
"node_id": "db-west-04",
"server_url": "http://10.0.3.4:8080",
"health": {"status": "ok", "version": "0.2.0", "model": "kappa_20b_131k"},
"workstreams": [...],
"aggregate": {"total_tokens": 48200, "total_tool_calls": 156}
}
```
### `GET /api/cluster/events`
Server-Sent Events stream for real-time cluster updates.
```
data: {"type":"cluster_state","ws_id":"a1b2","node_id":"db-west-04","state":"running"}
data: {"type":"ws_created","ws_id":"e5f6","node_id":"api-east-01","name":"new-task"}
data: {"type":"ws_closed","ws_id":"a1b2"}
data: {"type":"node_joined","node_id":"db-west-05"}
data: {"type":"node_lost","node_id":"db-west-03"}
```
Keepalive comments (`: keepalive\n\n`) are sent every 5 seconds. Clients should reconnect on error with exponential backoff.
### `GET /health`
```json
{"status": "ok", "service": "turnstone-console", "nodes": 847, "workstreams": 4219}
```
---
## Browser Dashboard
The web UI has three views, toggled client-side:
### 1. Cluster Overview (landing)
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, HEALTH. Sorted by activity. Clickable rows drill down to node detail.
### 2. Node Drill-down
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's own dashboard (`http://{server_url}/`).
### 3. Filtered Workstreams
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom.
All three views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
---
## CLI Commands
The `/cluster` command in the turnstone CLI queries the console's HTTP API. Requires `--console-url` or `[console] url` in config.toml.
| Command | Description |
|---------|-------------|
| `/cluster status` | Cluster overview — node/workstream counts, state breakdown, aggregate stats |
| `/cluster nodes` | Node table — WS, RUN, ATTN, TOKENS per node |
| `/cluster workstreams [state] [node=X]` | Filtered workstream list with state, name, node, tokens, context |
| `/cluster node <id>` | Single node's workstreams with activity details |
---
## Configuration
CLI flags for `turnstone-console`:
| Flag | Default | Description |
|------|---------|-------------|
| `--host` | `0.0.0.0` | Bind host |
| `--port` | `8090` | HTTP port |
| `--redis-host` | `localhost` | Redis host |
| `--redis-port` | `6379` | Redis port |
| `--redis-password` | `$REDIS_PASSWORD` | Redis password |
| `--redis-db` | `0` | Redis DB |
| `--poll-interval` | `10` | Node polling interval (seconds) |
| `--log-level` | `INFO` | Log level |
Config file (`~/.config/turnstone/config.toml`):
```toml
[console]
host = "0.0.0.0"
port = 8090
url = "http://localhost:8090" # used by CLI /cluster commands
poll_interval = 10
[redis]
host = "localhost"
port = 6379
password = "my-redis-password"
```
---
## Deployment
```bash
# Start Redis
redis-server
# Start turnstone servers (one per node)
turnstone-server --port 8080
# Start bridges (one per server)
turnstone-bridge --server-url http://localhost:8080 --node-id node-a
# Start cluster console (one instance)
turnstone-console --redis-host localhost --port 8090
```
Open `http://localhost:8090` for the cluster dashboard.
docs/docker.md
+139
View File
@@ -0,0 +1,139 @@
# Docker Deployment
Docker Compose stack for running the full turnstone platform or the simulator.
## Quick Start
```bash
# Copy and edit environment config
cp .env.example .env
# Full stack (needs an LLM API on the host)
docker compose up
# Simulator only (no LLM needed)
docker compose --profile sim up redis console sim
```
Console dashboard: http://localhost:8090
## Services
| Service | Port | Profile | Description |
|---------|------|---------|-------------|
| `redis` | 6379 | default | Message broker, pub/sub, node registry |
| `server` | 8080 | default | Web UI + chat workstreams + LLM |
| `bridge` | — | default | Redis-to-HTTP bridge (multi-node routing) |
| `console` | 8090 | default | Cluster dashboard |
| `sim` | — | sim | Multi-node cluster simulator |
## Profiles
**Default** (no flag) — starts `redis`, `server`, `bridge`, `console`. Requires an OpenAI-compatible LLM API running on the host (default: `http://localhost:8000/v1`).
```bash
docker compose up
```
**Sim** — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:
```bash
# Sim + console (no LLM needed)
docker compose --profile sim up redis console sim
# Everything including sim
docker compose --profile sim up
```
## Configuration
All configuration is via environment variables in `.env` (copy from `.env.example`):
### LLM Backend
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | OpenAI-compatible API URL |
| `OPENAI_API_KEY` | `dummy` | API key (`dummy` for local servers) |
| `TAVILY_API_KEY` | — | Web search API key (optional) |
### Redis
| Variable | Default | Description |
|----------|---------|-------------|
| `REDIS_PASSWORD` | — | Redis auth password (empty = no auth) |
| `REDIS_PORT` | `6379` | Host port mapping |
### Server
| Variable | Default | Description |
|----------|---------|-------------|
| `SERVER_PORT` | `8080` | Host port mapping |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tools |
### Console
| Variable | Default | Description |
|----------|---------|-------------|
| `CONSOLE_PORT` | `8090` | Host port mapping |
| `CONSOLE_POLL_INTERVAL` | `10` | Node polling interval (seconds) |
### Auth
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_AUTH_ENABLED` | — | Set to `1` to require Bearer token auth |
| `TURNSTONE_AUTH_TOKEN` | — | Shared auth token for server/bridge/console |
### Simulator
| Variable | Default | Description |
|----------|---------|-------------|
| `SIM_NODES` | `100` | Number of simulated nodes |
| `SIM_SCENARIO` | `steady` | Scenario: `steady`, `burst`, `node_failure`, `directed`, `lifecycle` |
| `SIM_DURATION` | `60` | Duration in seconds |
| `SIM_MPS` | `5.0` | Messages per second (steady scenario) |
| `SIM_LOG_LEVEL` | `INFO` | Log verbosity |
| `SIM_SEED` | — | Random seed for reproducibility |
| `SIM_METRICS_FILE` | — | Write JSON report to file |
## Scaling
Scale to multiple server/bridge pairs:
```bash
docker compose up --scale server=3 --scale bridge=3
```
Each bridge auto-generates a unique node ID from its container hostname. When scaling `server`, remove the host port mapping (or use a reverse proxy) to avoid port conflicts.
## Volumes
| Volume | Mount | Purpose |
|--------|-------|---------|
| `redis-data` | `/data` | Redis persistence |
| `turnstone-data` | `/data` | SQLite database (`.turnstone.db`) |
## Building
The image uses a multi-stage Dockerfile:
```bash
# Build all services
docker compose build
# Rebuild without cache
docker compose build --no-cache
```
All five entry points are installed in a single image: `turnstone-server`, `turnstone-bridge`, `turnstone-console`, `turnstone-sim`, `turnstone-eval`.
## Cleanup
```bash
# Stop and remove containers
docker compose down
# Stop, remove containers and volumes
docker compose down -v
```
docs/eval.md
+337
View File
@@ -0,0 +1,337 @@
# Evaluation and Prompt Optimization (turnstone-eval)
`turnstone-eval` is the evaluation and prompt optimization system for turnstone. It
runs test cases against the LLM, scores tool call sequences against expected
actions, and optionally uses the model to self-optimize the developer prompt.
Source: `turnstone/eval.py`
---
## Overview
The system works in an iterative loop:
1. Run each test case N times against the current developer prompt.
2. Score each run by comparing the actual tool call sequence to expected actions.
3. If not all tests pass, use the model to rewrite the prompt based on failures.
4. Repeat until all tests pass or max iterations are reached.
When optimization is disabled (`--no-optimize`), only step 1 and 2 execute
(a single iteration).
---
## Test Case Format
Test suites are JSON files with this structure:
```json
{
"defaults": {
"n_runs": 3
},
"cases": [
{
"id": "test_name",
"user_prompt": "the prompt to send to the model",
"setup": {
"files": {
"filename.py": "file content here",
"src/utils.py": "another file"
}
},
"expected_actions": [
{"tool": "read_file", "args": {"path": "filename.py"}},
{"tool": "bash", "args_pattern": {"command": "python.*test"}},
{"tool": "edit_file"}
],
"match_mode": "ordered_subset",
"max_turns": 10,
"n_runs": 5
}
]
}
```
### Fields
| Field | Required | Default | Description |
|--------------------|----------|--------------------|-------------|
| `id` | yes | -- | Unique test case identifier. |
| `user_prompt` | yes | -- | The message sent to the model. |
| `setup.files` | no | `{}` | Files to create in the temp directory before running. Keys are relative paths, values are file content. |
| `expected_actions` | no | `[]` | List of expected tool calls to match against. |
| `match_mode` | no | `"ordered_subset"` | How to match actual vs expected actions (see Scoring). |
| `max_turns` | no | `10` | Maximum conversation turns before stopping. |
| `n_runs` | no | suite default or 3 | Per-case override for number of runs. |
### Expected Action Specs
Each entry in `expected_actions` can contain:
- `tool` (required): The tool name to match (e.g. `"read_file"`, `"bash"`).
- `args`: Exact key-value matching. Each key in `args` must exist in the actual call with the same string value.
- `args_pattern`: Regex key-value matching. Each key's value is a regex pattern tested against the actual argument value.
- If neither `args` nor `args_pattern` is specified, only the tool name is matched.
---
## Scoring
Scoring is handled by `score_run()`, which compares a run's tool call log
against the expected actions.
### Match Modes
| Mode | Description |
|-------------------|-------------|
| `exact` | Tool calls must match expected actions in exact order and exact count. Extra or missing calls cause failure. |
| `ordered_subset` | Expected actions must appear in order within the actual tool log, but extra calls between them are allowed. This is the default. |
| `subset` | Expected actions must all appear somewhere in the tool log, in any order. Each actual call can only match one expected action. |
| `contains_any` | Passes if at least one expected action appears anywhere in the tool log. |
### Action Matching (`_match_action`)
A single actual tool call matches an expected action when:
1. The tool names are equal.
2. If `args` is specified: every key in `args` must exist in the actual call's
arguments with the same string value (partial key matching -- extra actual
args are ignored).
3. If `args_pattern` is specified: every key's regex pattern must match the
corresponding actual argument value via `re.search()`.
4. If the actual args contain only `_raw` (unparseable JSON fallback), the
action matches only when no `args` or `args_pattern` is expected.
### Score Calculation
- **Score** = number of matched expected actions / total expected actions.
- **Pass** = score equals 1.0 (all expected actions matched).
- The return dict includes: `pass`, `score`, `matched` (indices), `unmatched`
(indices), `extra_tools`, and `detail` (human-readable summary).
### JSON Dump Detection
When a run fails and the model's final text content contains JSON that looks
like a tool call (keys like `"tool"`, `"command"`, `"path"`), the run is
flagged with `json_dump: true`. This indicates the model tried to call a tool
but emitted JSON as text instead of using the function-calling interface.
---
## HeadlessSession
`HeadlessSession` extends `ChatSession` for headless evaluation. It provides
deterministic, non-interactive execution suitable for automated testing.
### Differences from ChatSession
| Aspect | ChatSession | HeadlessSession |
|-----------------|-------------------------|----------------------------|
| Streaming | Streaming API | Non-streaming (`stream=False`) |
| Tool approval | User confirmation | `auto_approve = True` |
| UI | Terminal/Web UI | `NullUI` (discards output) |
| Stdout | Normal | Suppressed during execution |
| Tool logging | Display only | Structured `tool_call_log` |
| System prompt | Built-in developer prompt | Overridable via constructor |
### NullUI
A minimal UI adapter that satisfies the `SessionUI` protocol by discarding
all output. `approve_tools()` always returns `(True, None)`.
### send_headless()
```python
def send_headless(
self,
user_input: str,
max_turns: int = 10,
verbose: bool = False,
log_prefix: str = "",
) -> list[dict]:
```
Runs a complete multi-turn conversation:
1. Appends the user message.
2. Calls the model API (non-streaming).
3. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
4. Repeats up to `max_turns` or until the model responds without tool calls.
5. Returns the tool call log: list of dicts with keys `tool`, `args`,
`result` (truncated to 500 chars), and `turn`.
Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
### Retry Logic
`send_headless()` is called inside `_run_single_test()` with retry logic:
3 attempts with exponential backoff (sleep `2^attempt` seconds) on any
exception. This prevents transient API errors from poisoning eval scores.
---
## Test Execution
Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A new `HeadlessSession` is created with the current developer prompt.
5. `send_headless()` runs the user prompt through the conversation loop.
6. The tool log is scored against expected actions.
7. The temp directory is cleaned up.
The memory database is also isolated per test (an ephemeral SQLite database
in the temp directory) so tests do not pollute each other or the user's
real memory store.
---
## Optimization Loop
`run_optimization()` is the main entry point for iterative prompt optimization.
### Flow
```
for iteration in 0..max_iterations:
1. Run all test cases n_runs times with current prompt
2. Score and aggregate results
3. Save intermediate results to JSON
4. If all tests pass -> stop
5. Every 3 iterations (at iteration 2, 5, 8, ...):
-> Observer reviews optimizer strategy
-> Reset prompt to best-performing iteration
6. Propose new prompt via optimizer model call
7. If prompt unchanged -> stop
8. Continue with new prompt
```
### Prompt Proposal (`_propose_prompt_modification`)
Uses the model to rewrite the developer prompt based on test results:
- **Input**: Current prompt, test case definitions, per-case results with
actual vs expected tool sequences, and a history of the last 3 iterations.
- **Optimizer system prompt** (`OPTIMIZER_SYSTEM`): Instructs the model to
act as a text rewriter. Key guidance includes:
- Address critical failure modes (text-only responses, write_file vs edit_file,
unnecessary search before create, missing plan calls).
- Preserve phrasing that drives 100% pass rate on passing tests.
- Use direct imperative style with concrete tool call examples.
- Stay within 130% of original prompt length.
- **Output**: The rewritten prompt text (stripped of reasoning tags and code fences).
### Observer System (`_observe_and_update_optimizer`)
Every 3 iterations, a meta-level "observer" reviews the optimizer's strategy:
- Analyzes the iteration history: score trends, regressions, prompt length changes,
and diffs between iterations.
- Summarizes the optimizer's behavioral patterns (list style, header usage, length).
- Uses `OBSERVER_SYSTEM` to rewrite the optimizer's own system prompt.
- Rejects degenerate outputs (over 200% of input length).
- After updating the optimizer prompt, resets the developer prompt to the
best-performing iteration so far.
This two-level optimization (optimizer + observer) helps the system escape
local minima and adjust its rewriting strategy.
### Result Persistence
After each iteration, results are written to the output JSON file. The
structure is:
```json
{
"meta": {
"model": "model-name",
"base_url": "http://localhost:8000/v1",
"started": "2025-01-01T00:00:00",
"test_suite": "tests.json",
"n_runs_default": 3
},
"iterations": [
{
"iteration": 0,
"prompt": "the developer prompt used",
"prompt_diff": null,
"optimizer_system": "the optimizer system prompt",
"timestamp": "2025-01-01T00:01:00",
"cases": {
"test_name": {
"runs": [
{
"pass": true,
"score": 1.0,
"matched": [0, 1],
"unmatched": [],
"extra_tools": [],
"detail": "Ordered subset: 2/2",
"tool_sequence": ["read_file", "edit_file"],
"tool_args": [{"read_file": {"path": "f.py"}}, ...],
"elapsed": 3.2
}
],
"pass_rate": 1.0,
"avg_score": 1.0
}
},
"aggregate": {
"total_cases": 5,
"total_runs": 15,
"overall_pass_rate": 0.8,
"overall_avg_score": 0.87,
"json_dumps": 0,
"per_case_pass_rates": {"test_name": 1.0, ...}
}
}
]
}
```
---
## CLI Usage
The entry point is `turnstone-eval` (installed as a console script) or
`python -m turnstone.eval`.
```
turnstone-eval tests.json # evaluate + optimize
turnstone-eval tests.json --no-optimize # evaluate only (single iteration)
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
turnstone-eval tests.json --prompt custom.txt # start from a custom prompt
turnstone-eval tests.json -v # verbose per-turn logging
```
### All Options
| Flag | Default | Description |
|---------------------|-------------------------------|-------------|
| `test_file` | (positional, required) | Path to test cases JSON file. |
| `--base-url` | `http://localhost:8000/v1` | API base URL. |
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
| `--max-iter` | 5 | Maximum optimization iterations. |
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
| `--temperature` | 0.7 | Sampling temperature. |
| `--max-tokens` | 32768 | Max completion tokens. |
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
| `--context-window` | 131072 | Context window size. |
| `--output` | `eval_results.json` | Output results file path. |
| `-v`, `--verbose` | false | Show detailed per-turn logging (API calls, tool args, results). |
### Precedence for n_runs
The number of runs per test case is resolved in this order:
1. Per-case `n_runs` field in the test case definition.
2. CLI `--n-runs` argument (if provided).
3. Suite-level `defaults.n_runs` in the test JSON file.
4. Code default: 3.
docs/simulator.md
+202
View File
@@ -0,0 +1,202 @@
# Cluster Simulator
The simulator (`turnstone-sim`) creates lightweight simulated nodes that talk to a real Redis instance using the standard turnstone protocol. External observers — `TurnstoneClient`, `turnstone-console`, real bridges — see identical behavior. No LLM backend is needed.
## Quick Start
```bash
pip install turnstone[sim]
# 10 nodes, steady load, 60 seconds
turnstone-sim --nodes 10 --scenario steady --duration 60 --mps 5
# 100 nodes via Docker
docker compose --profile sim up redis console sim
```
## How It Works
Each simulated node is an asyncio coroutine (not a thread or process), so 1000 nodes run efficiently on a single event loop. The simulator:
1. Registers nodes via Redis heartbeats (same keys as real bridges)
2. Accepts messages from per-node and shared inbound queues
3. Simulates LLM responses with configurable latency and token generation
4. Simulates tool execution with configurable latency and failure rates
5. Publishes real protocol events (`ContentEvent`, `StateChangeEvent`, `TurnCompleteEvent`, etc.)
6. Reports latency, throughput, and utilization metrics at completion
```
TurnstoneClient → Redis Queue → SimNode → Redis Pub/Sub → TurnstoneClient
↓
turnstone-console (cluster dashboard)
```
## Scenarios
| Scenario | Description |
|----------|-------------|
| `steady` | Inject messages at a constant rate (`--mps`) for `--duration` seconds |
| `burst` | Push `--burst-size` messages instantly, then wait for completion |
| `node_failure` | Steady load + periodically kill nodes to test redistribution |
| `directed` | Send messages to specific nodes via `target_node` routing |
| `lifecycle` | Create, use, and close workstreams across nodes |
## CLI Reference
```
turnstone-sim [options]
```
### Cluster
| Flag | Default | Description |
|------|---------|-------------|
| `--nodes` | `10` | Number of simulated nodes |
### Scenario
| Flag | Default | Description |
|------|---------|-------------|
| `--scenario` | `steady` | Scenario name |
| `--duration` | `60` | Duration in seconds |
| `--mps` | `5.0` | Messages per second (steady) |
| `--burst-size` | `100` | Messages to send (burst) |
| `--node-kill-interval` | `15` | Seconds between kills (node_failure) |
| `--node-kill-count` | `1` | Nodes per kill cycle |
### Simulation
| Flag | Default | Description |
|------|---------|-------------|
| `--llm-latency` | `2.0` | Mean LLM response latency (seconds) |
| `--tool-latency` | `0.5` | Mean tool execution latency (seconds) |
| `--tool-failure-rate` | `0.02` | Tool failure probability (0.0–1.0) |
| `--seed` | — | Random seed for reproducibility |
### Redis
| Flag | Default | Description |
|------|---------|-------------|
| `--redis-host` | `localhost` | Redis host |
| `--redis-port` | `6379` | Redis port |
| `--redis-password` | — | Redis password |
| `--prefix` | `turnstone` | Redis key prefix |
### Output
| Flag | Default | Description |
|------|---------|-------------|
| `--metrics-file` | — | Write JSON report to file |
| `--log-level` | `INFO` | Log verbosity |
## Example: Load Testing
```bash
# 100 nodes, high throughput, 2 minutes
turnstone-sim --nodes 100 --scenario steady --duration 120 --mps 50
# Burst of 500 messages across 50 nodes
turnstone-sim --nodes 50 --scenario burst --burst-size 500 --duration 60
# Node failure resilience (kill 2 nodes every 10 seconds)
turnstone-sim --nodes 20 --scenario node_failure --duration 120 \
--node-kill-interval 10 --node-kill-count 2
# Fast simulation (low latency, no failures)
turnstone-sim --nodes 10 --scenario steady --duration 30 \
--llm-latency 0.1 --tool-latency 0.05 --tool-failure-rate 0 --mps 10
```
## Metrics Report
The simulator prints a summary at completion:
```
============================================================
SIMULATION REPORT
============================================================
Scenario: steady
Nodes: 100
Duration: 60.2s
Total turns: 295
Total errors: 5
Node kills: 0
------------------------------------------------------------
THROUGHPUT
Messages/sec: 4.97
Turns/sec: 4.89
------------------------------------------------------------
LATENCY (seconds)
p50: 3.21
p90: 5.44
p99: 8.12
mean: 3.56
max: 12.1
------------------------------------------------------------
UTILIZATION
Mean ws/node: 2.3
Max ws/node: 8
Idle nodes: 12
============================================================
```
Use `--metrics-file report.json` to write the full report as JSON.
## Console Integration
The simulator's nodes appear in `turnstone-console` exactly like real nodes. Run them together to see the dashboard populate with simulated workstreams:
```bash
# Terminal 1: start Redis and console
docker compose up redis console
# Terminal 2: run simulator
docker compose --profile sim up sim
```
Or all at once:
```bash
SIM_NODES=50 SIM_DURATION=120 docker compose --profile sim up redis console sim
```
Open http://localhost:8090 to see simulated nodes, workstream states, token counts, and load bars updating in real time.
## Architecture
```
turnstone/sim/
├── __init__.py # Public API: SimCluster, SimConfig
├── config.py # SimConfig — all simulation parameters
├── engine.py # SimEngine — LLM + tool execution simulation
├── node.py # SimNode + SimWorkstream — protocol-compatible node
├── cluster.py # SimCluster + InboundDispatcher + PooledBroker
├── scenario.py # 5 scenario classes
├── metrics.py # MetricsCollector — latency, throughput, utilization
└── cli.py # CLI entry point
```
**Key design:** The `InboundDispatcher` batches ~50 node queues into a single Redis `BLPOP` call, keeping connection count bounded at ~20 regardless of node count. All nodes share a single `ConnectionPool(max_connections=64)`.
## Programmatic Use
```python
import asyncio
from turnstone.sim import SimCluster, SimConfig
async def main():
config = SimConfig(
num_nodes=10,
scenario="steady",
duration=30,
messages_per_second=2.0,
llm_latency_mean=0.5,
)
cluster = SimCluster(config)
await cluster.start()
await cluster.run_scenario()
print(cluster.report())
await cluster.stop()
asyncio.run(main())
```
docs/tools.md
+394
View File
@@ -0,0 +1,394 @@
# Tools Reference
turnstone exposes 14 tools to the LLM via the OpenAI function-calling interface.
Each tool is defined as a JSON file under `turnstone/tools/` and loaded at startup
by `turnstone/core/tools.py`.
---
## Tool Schema Format
Each JSON file in `turnstone/tools/` contains a standard OpenAI function-calling
schema plus turnstone-specific metadata keys:
```json
{
"name": "tool_name",
"description": "What the tool does.",
"parameters": {
"type": "object",
"properties": { ... },
"required": ["param1"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "param1"
}
```
**Metadata keys** (stripped before sending the schema to the model):
| Key | Type | Meaning |
|----------------|------|---------|
| `agent` | bool | Tool is available to plan/task sub-agents (read-only subset). |
| `task_agent` | bool | Tool is available to task sub-agents (broader subset). |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
---
## Derived Tool Sets
`turnstone/core/tools.py` loads all JSON files and derives these collections:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 14 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
## Execution Pipeline
Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools()`:
### Phase 1: Prepare
`_prepare_tool(tc)` is called for each tool call returned by the model.
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Validates arguments and builds a preview dict containing:
- `call_id`, `func_name`, `header`, `preview` (for display)
- `needs_approval` (bool)
- `execute` (callable to run the tool)
- `error` (set if validation fails; tool will not execute)
### Phase 2: Approve
All prepared items are sent to the UI via `ui.approve_tools(items)`.
- The UI displays each tool's header and preview to the user.
- Items where `needs_approval` is `False` (auto-approved tools) are shown
but do not block execution.
- Items where `needs_approval` is `True` require the user to accept or deny.
- The user can provide feedback alongside their approval (e.g. "y, use full path").
- If `auto_approve` is `True` on the session (headless mode), all tools are
approved automatically.
### Phase 3: Execute
Each item's `execute` callable is invoked:
- Single tool calls run directly on the current thread.
- Multiple tool calls run in parallel via `ThreadPoolExecutor(max_workers=4)`.
- Errored or denied items return their error/denial message without executing.
- Special post-execution gate for `plan`: the plan output is shown to the user
for review, and the user can reject or annotate it.
---
## Tool Approval Flow
**Auto-approved** (no user confirmation needed at runtime):
- `read_file` -- reads files, no side effects
- `search` -- grep-style search, no side effects
- `man` -- reads man pages, no side effects
- `remember` -- writes to persistent memory database (lightweight, always auto-approved)
- `recall` -- reads from persistent memory database
- `forget` -- deletes from persistent memory database (lightweight, always auto-approved)
**Requires user confirmation** (write operations, network access, side effects):
- `bash` -- arbitrary command execution
- `write_file` -- creates or overwrites files
- `edit_file` -- modifies file content
- `math` -- sandboxed computation (confirmation required despite being sandboxed)
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
- `web_search` -- web search via Tavily API (makes network requests)
- `task` -- spawns an autonomous sub-agent
- `plan` -- spawns a planning sub-agent, plus post-execution review gate
Note: The JSON schema metadata key `auto_approve` controls membership in
`AGENT_AUTO_TOOLS`/`TASK_AUTO_TOOLS` (used for agent sub-sessions). The actual
runtime approval behavior is determined by the `needs_approval` field set in
each `_prepare_*` method on `ChatSession`. These two mechanisms can differ.
---
## Primary Key Fallback
When the model sends a bare string instead of a JSON object as tool arguments
(common with smaller models), the `primary_key` mapping rescues the call:
```
Model sends: bash("ls -la")
raw_args = "ls -la" (not valid JSON)
PRIMARY_KEY_MAP["bash"] = "command"
Result: args = {"command": "ls -la"}
```
Every tool defines a `primary_key`. The mapping is:
| Tool | primary_key |
|--------------|-------------|
| `bash` | `command` |
| `read_file` | `path` |
| `write_file` | `content` |
| `edit_file` | `old_string`|
| `search` | `query` |
| `math` | `code` |
| `man` | `page` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `task` | `prompt` |
| `plan` | `prompt` |
| `remember` | `key` |
| `recall` | `query` |
| `forget` | `key` |
---
## File Operations
### bash
Execute a bash command and return stdout + stderr.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `command` | string | yes | The bash command to execute. |
- **What it does**: Runs the command in a subprocess with a configurable timeout. Commands are sanitized and checked against a blocklist (e.g. `rm -rf /`).
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only (not available to plan sub-agents).
---
### read_file
Read the contents of a file, returning numbered lines.
| Parameter | Type | Required | Description |
|-----------|---------|----------|-------------|
| `path` | string | yes | Absolute or relative file path. |
| `offset` | integer | no | Line number to start from (1-based, default: 1). |
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. |
- **What it does**: Reads the file and returns content with line numbers. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
### write_file
Write content to a file, creating it if needed.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `path` | string | yes | Absolute or relative file path. |
| `content` | string | yes | The full file content to write. |
- **What it does**: Creates or overwrites the file at the given path. Parent directories are created as needed.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only.
---
### edit_file
Replace an exact string in a file with new content.
| Parameter | Type | Required | Description |
|--------------|---------|----------|-------------|
| `path` | string | yes | Absolute or relative file path. |
| `old_string` | string | yes | The exact text to find and replace. |
| `new_string` | string | yes | The replacement text. |
| `near_line` | integer | no | Disambiguate when `old_string` matches multiple locations. |
- **What it does**: Finds `old_string` in the file and replaces it with `new_string`. Fails if the string is not found or matches multiple locations (unless `near_line` is provided to pick the nearest match). Requires a prior `read_file` call on the same path.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only.
---
### search
Search file contents for a regex pattern.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `query` | string | yes | Regex pattern (extended regex). |
| `path` | string | no | File or directory to search in (default: current directory). |
- **What it does**: Recursively searches for the pattern using `grep -rn`. Returns matching lines with file paths and line numbers.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
## Computation
### math
Execute Python code for math and computation in a sandbox.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `code` | string | yes | Python code to execute. Must use `print()` for output. |
- **What it does**: Runs Python code in a sandboxed environment with pre-imported libraries: `sympy`, `numpy`, `scipy`, `math`, `fractions`, `itertools`, `functools`, `collections`, `decimal`, `operator`, `random`, `re`, `string`. Common sympy names (`symbols`, `solve`, `simplify`, `sqrt`, `Matrix`, etc.) are pre-imported.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `agent` and `task_agent`.
---
## Information
### man
Read a man page.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `page` | string | yes | The man page name (e.g. `grep`, `socket`, `printf`). |
| `section` | string | no | Manual section (e.g. `1` commands, `2` syscalls, `3` library). |
- **What it does**: Returns the full formatted manual entry. Preferred over `bash('man ...')` or `web_search` for command/API documentation.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
### web_fetch
Fetch a URL and extract specific information from it.
| Parameter | Type | Required | Description |
|------------|--------|----------|-------------|
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
| `question` | string | yes | What to extract or answer from the page content. |
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `agent` and `task_agent`.
---
### web_search
Search the web using a text query.
| Parameter | Type | Required | Description |
|---------------|---------|----------|-------------|
| `query` | string | yes | The search query. |
| `max_results` | integer | no | Max results to return (default 5, max 20). |
| `topic` | string | no | Search topic: `general`, `news`, or `finance` (default `general`). |
- **What it does**: Searches the web via the Tavily API and returns ranked results with titles, URLs, and content snippets.
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `agent` and `task_agent`.
---
## Agent
### task
Delegate a general-purpose task to an autonomous sub-agent.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, math, man, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Not available to sub-agents (top-level only).
---
### plan
Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `prompt` | string | yes | What to plan -- the goal, constraints, and scope. |
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<session_id>.md` (unique per session, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **Auto-approve**: No -- requires user confirmation, plus post-execution review gate.
- **Agent availability**: Not available to sub-agents (top-level only).
---
## Memory
### remember
Save a persistent memory that persists across sessions.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `key` | string | yes | Short identifier (e.g. `user_name`). |
| `value` | string | yes | Content to remember. |
- **What it does**: Stores a key-value pair in the SQLite memory database. Memories persist across sessions and are included in the system prompt on startup.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
---
### recall
Search memories and past conversations.
| Parameter | Type | Required | Description |
|-----------|---------|----------|-------------|
| `query` | string | no | Search term or phrase. Omit to list all memories. |
| `limit` | integer | no | Max conversation results to return (default 20). |
- **What it does**: With no query, lists all saved memories. With a query, searches both the memory store and conversation history using FTS5 full-text search.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
---
### forget
Remove a persistent memory by key.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `key` | string | yes | The memory key to remove (e.g. `user_name`). |
- **What it does**: Deletes the memory entry with the given key from the SQLite database.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
---
## Summary Table
| Tool | Category | Auto-approve | agent | task_agent | primary_key |
|--------------|------------|--------------|-------|------------|-------------|
| `bash` | File Ops | No | No | Yes | `command` |
| `read_file` | File Ops | Yes | Yes | Yes | `path` |
| `write_file` | File Ops | No | No | Yes | `content` |
| `edit_file` | File Ops | No | No | Yes | `old_string`|
| `search` | File Ops | Yes | Yes | Yes | `query` |
| `math` | Compute | No | Yes | Yes | `code` |
| `man` | Info | Yes | Yes | Yes | `page` |
| `web_fetch` | Info | No | Yes | Yes | `url` |
| `web_search` | Info | No | Yes | Yes | `query` |
| `task` | Agent | No | No | No | `prompt` |
| `plan` | Agent | No | No | No | `prompt` |
| `remember` | Memory | Yes | No | No | `key` |
| `recall` | Memory | Yes | No | No | `query` |
| `forget` | Memory | Yes | No | No | `key` |
pyproject.toml
+41
View File
@@ -0,0 +1,41 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "0.2.0"
description = "AI chat client with tool use, agent tools, and persistent memory."
readme = "README.md"
license = "BUSL-1.1"
requires-python = ">=3.11"
dependencies = ["openai>=1.0", "httpx>=0.24"]
[project.optional-dependencies]
test = ["pytest>=7.0"]
mq = ["redis>=5.0"]
console = ["redis>=5.0"]
sim = ["redis>=5.0"]
[project.scripts]
turnstone = "turnstone.cli:main"
turnstone-eval = "turnstone.eval:main"
turnstone-server = "turnstone.server:main"
turnstone-bridge = "turnstone.mq.bridge:main"
turnstone-console = "turnstone.console.server:main"
turnstone-sim = "turnstone.sim.cli:main"
[tool.hatch.build.targets.wheel]
include = [
"turnstone/**/*.py",
"turnstone/tools/*.json",
"turnstone/ui/static/*.html",
"turnstone/ui/static/*.css",
"turnstone/ui/static/*.js",
"turnstone/console/static/*.html",
"turnstone/console/static/*.css",
"turnstone/console/static/*.js",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
tests.json
+180
View File
@@ -0,0 +1,180 @@
{
"description": "pcode behavior tests — tool selection, sequencing, and multi-step reasoning",
"defaults": {
"n_runs": 5,
"max_turns": 15
},
"cases": [
{
"id": "read-before-edit",
"description": "Must read_file before edit_file on the same path",
"user_prompt": "Fix the typo in config.py — change 'recieve' to 'receive'",
"setup": {
"files": {
"config.py": "# Config module\ndef recieve_data(source):\n \"\"\"Recieve data from source.\"\"\"\n return source.read()\n"
}
},
"expected_actions": [
{ "tool": "read_file", "args": { "path": "config.py" } },
{ "tool": "edit_file", "args": { "path": "config.py" } }
],
"match_mode": "ordered_subset"
},
{
"id": "write-file-not-bash",
"description": "Use write_file for file creation, not bash echo/cat",
"user_prompt": "Create a file called hello.py that prints hello world",
"expected_actions": [
{ "tool": "write_file", "args_pattern": { "path": "hello\\.py" } }
],
"match_mode": "subset"
},
{
"id": "bash-for-commands",
"description": "Use bash for running system commands",
"user_prompt": "What Python version is installed?",
"expected_actions": [
{ "tool": "bash", "args_pattern": { "command": "python" } }
],
"match_mode": "subset"
},
{
"id": "search-for-patterns",
"description": "Use search tool for finding code patterns across files",
"user_prompt": "Find all functions that start with 'test_' in the project",
"setup": {
"files": {
"tests.py": "def test_one():\n assert True\n\ndef test_two():\n assert 1 + 1 == 2\n",
"main.py": "def main():\n print('hello')\n"
}
},
"expected_actions": [
{ "tool": "search", "args_pattern": { "query": "test_" } }
],
"match_mode": "subset"
},
{
"id": "multi-file-edit",
"description": "Read and edit multiple files — must read before editing each, and edit both",
"user_prompt": "Change the default port from 8000 to 9000 in both server.py and config.py",
"setup": {
"files": {
"server.py": "from config import PORT\n\ndef run():\n print(f'Listening on port {PORT}')\n",
"config.py": "PORT = 8000\nHOST = 'localhost'\n"
}
},
"expected_actions": [
{ "tool": "read_file" },
{ "tool": "read_file" },
{ "tool": "edit_file" },
{ "tool": "edit_file" }
],
"match_mode": "ordered_subset"
},
{
"id": "search-then-edit",
"description": "Search to find where something is defined, then read and edit",
"user_prompt": "Find where MAX_RETRIES is defined and change it from 3 to 5",
"setup": {
"files": {
"app.py": "import os\n\ndef start():\n pass\n",
"utils.py": "import time\n\ndef helper():\n pass\n",
"config/settings.py": "# Settings\nDEBUG = True\nMAX_RETRIES = 3\nTIMEOUT = 30\n"
}
},
"expected_actions": [
{ "tool": "search", "args_pattern": { "query": "MAX_RETRIES" } },
{ "tool": "read_file" },
{ "tool": "edit_file", "args_pattern": { "old_string": "3" } }
],
"match_mode": "ordered_subset"
},
{
"id": "bash-git-log",
"description": "Use bash for git commands, not other tools",
"user_prompt": "Show me the git log for the last 5 commits",
"expected_actions": [
{ "tool": "bash", "args_pattern": { "command": "git\\s+log" } }
],
"match_mode": "subset"
},
{
"id": "write-then-run",
"description": "Create a script and run it to verify it works",
"user_prompt": "Create a Python script called fib.py that prints the first 10 Fibonacci numbers, then run it to verify",
"expected_actions": [
{ "tool": "write_file", "args_pattern": { "path": "fib\\.py" } },
{ "tool": "bash", "args_pattern": { "command": "python" } }
],
"match_mode": "ordered_subset"
},
{
"id": "no-bash-for-file-write",
"description": "Should NOT use bash (echo/cat/heredoc) to create files — only write_file",
"user_prompt": "Create a new file called README.md with a title and description of this project",
"expected_actions": [
{ "tool": "write_file", "args_pattern": { "path": "README" } }
],
"match_mode": "subset"
},
{
"id": "plan-before-refactor",
"description": "Use the plan tool before a large refactoring task",
"user_prompt": "I need to refactor this codebase to separate the database layer from the API layer. Use the plan tool to think through the approach before making any changes.",
"setup": {
"files": {
"app.py": "import sqlite3\nfrom flask import Flask, jsonify\n\napp = Flask(__name__)\nDB = 'data.db'\n\ndef get_db():\n return sqlite3.connect(DB)\n\n@app.route('/users')\ndef list_users():\n db = get_db()\n users = db.execute('SELECT * FROM users').fetchall()\n db.close()\n return jsonify(users)\n\n@app.route('/users/<int:uid>')\ndef get_user(uid):\n db = get_db()\n user = db.execute('SELECT * FROM users WHERE id=?', (uid,)).fetchone()\n db.close()\n return jsonify(user)\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
}
},
"expected_actions": [{ "tool": "plan" }],
"match_mode": "subset"
},
{
"id": "edit-not-rewrite",
"description": "Use edit_file for small changes, not write_file to rewrite the entire file",
"user_prompt": "Add a docstring to the process_data function in utils.py",
"setup": {
"files": {
"utils.py": "import os\nimport sys\n\n\ndef helper():\n \"\"\"A helper function.\"\"\"\n return 42\n\n\ndef process_data(items):\n results = []\n for item in items:\n if item > 0:\n results.append(item * 2)\n return results\n\n\ndef cleanup():\n \"\"\"Clean up resources.\"\"\"\n pass\n"
}
},
"expected_actions": [
{ "tool": "read_file", "args": { "path": "utils.py" } },
{ "tool": "edit_file", "args": { "path": "utils.py" } }
],
"match_mode": "ordered_subset"
},
{
"id": "bash-run-tests",
"description": "Use bash to run a test suite",
"user_prompt": "Run the tests",
"setup": {
"files": {
"test_math.py": "def test_add():\n assert 1 + 1 == 2\n\ndef test_mul():\n assert 2 * 3 == 6\n"
}
},
"expected_actions": [
{ "tool": "bash", "args_pattern": { "command": "pytest|python.*test" } }
],
"match_mode": "subset"
},
{
"id": "web-fetch-url",
"description": "Use web_fetch when asked to retrieve content from a URL",
"user_prompt": "Fetch the contents of https://example.com and summarize what's on the page",
"expected_actions": [
{ "tool": "web_fetch", "args_pattern": { "url": "example\\.com" } }
],
"match_mode": "subset"
},
{
"id": "man-page-lookup",
"description": "Use man tool to look up command documentation",
"user_prompt": "Look up the man page for tar and tell me what the --xattrs flag does",
"expected_actions": [
{ "tool": "man", "args_pattern": { "page": "tar" } }
],
"match_mode": "subset"
}
]
}
tests/__init__.py
View File
tests/conftest.py
+22
View File
@@ -0,0 +1,22 @@
import pytest
from unittest.mock import MagicMock
@pytest.fixture
def tmp_db(tmp_path, monkeypatch):
"""Provide a temporary SQLite database."""
import turnstone.core.memory as memory
db_path = str(tmp_path / "test.db")
monkeypatch.setattr(memory, "db_override", db_path)
memory.db_initialized.discard(db_path)
yield db_path
memory.db_initialized.discard(db_path)
@pytest.fixture
def mock_openai_client():
"""Return a minimal mock OpenAI client."""
client = MagicMock()
client.models.list.return_value.data = [MagicMock(id="test-model")]
return client
tests/test_auth.py
+1010
View File
File diff suppressed because it is too large Load Diff
tests/test_config.py
+185
View File
@@ -0,0 +1,185 @@
"""Tests for turnstone.core.config — unified TOML config loading."""
import argparse
import turnstone.core.config as config_mod
from turnstone.core.config import apply_config, load_config
def _reset_cache():
"""Clear the module-level config cache between tests."""
config_mod._cache = None
def test_load_config_missing_file(tmp_path, monkeypatch):
_reset_cache()
monkeypatch.setattr(config_mod, "CONFIG_PATH", tmp_path / "nope.toml")
assert load_config() == {}
def test_load_config_valid_toml(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[redis]\nhost = "10.0.0.1"\nport = 6380\npassword = "secret"\n')
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
result = load_config()
assert result["redis"]["host"] == "10.0.0.1"
assert result["redis"]["port"] == 6380
assert result["redis"]["password"] == "secret"
def test_load_config_section(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[api]\nbase_url = "http://x:8000/v1"\n[redis]\nhost = "y"\n')
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
assert load_config("redis") == {"host": "y"}
assert load_config("api") == {"base_url": "http://x:8000/v1"}
assert load_config("nonexistent") == {}
def test_load_config_invalid_toml(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text("this is not valid toml [[[")
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
assert load_config() == {}
def test_load_config_caches(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[api]\nbase_url = "http://first"\n')
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
first = load_config()
assert first["api"]["base_url"] == "http://first"
# Change file — should NOT be re-read (cached)
cfg.write_text('[api]\nbase_url = "http://second"\n')
second = load_config()
assert second["api"]["base_url"] == "http://first"
def test_apply_config_sets_defaults(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text(
'[redis]\nhost = "redis.local"\nport = 7777\npassword = "pw"\n'
'[bridge]\nserver_url = "http://bridge:9090"\n'
)
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
parser = argparse.ArgumentParser()
parser.add_argument("--redis-host", default="localhost")
parser.add_argument("--redis-port", type=int, default=6379)
parser.add_argument("--redis-password", default=None)
parser.add_argument("--server-url", default="http://localhost:8080")
apply_config(parser, ["redis", "bridge"])
args = parser.parse_args([])
assert args.redis_host == "redis.local"
assert args.redis_port == 7777
assert args.redis_password == "pw"
assert args.server_url == "http://bridge:9090"
def test_apply_config_cli_overrides(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[redis]\nhost = "config-host"\nport = 7777\n')
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
parser = argparse.ArgumentParser()
parser.add_argument("--redis-host", default="localhost")
parser.add_argument("--redis-port", type=int, default=6379)
apply_config(parser, ["redis"])
# CLI flag overrides config
args = parser.parse_args(["--redis-host", "cli-host"])
assert args.redis_host == "cli-host" # CLI wins
assert args.redis_port == 7777 # config wins (no CLI override)
def test_apply_config_missing_keys_keep_defaults(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[redis]\nhost = "only-host"\n') # no port, no password
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
parser = argparse.ArgumentParser()
parser.add_argument("--redis-host", default="localhost")
parser.add_argument("--redis-port", type=int, default=6379)
parser.add_argument("--redis-password", default=None)
apply_config(parser, ["redis"])
args = parser.parse_args([])
assert args.redis_host == "only-host"
assert args.redis_port == 6379 # original default kept
assert args.redis_password is None # original default kept
def test_apply_config_no_file(tmp_path, monkeypatch):
_reset_cache()
monkeypatch.setattr(config_mod, "CONFIG_PATH", tmp_path / "nope.toml")
parser = argparse.ArgumentParser()
parser.add_argument("--redis-host", default="localhost")
apply_config(parser, ["redis"])
args = parser.parse_args([])
assert args.redis_host == "localhost"
def test_apply_config_model_section(tmp_path, monkeypatch):
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[model]\nname = "qwen-72b"\ntemperature = 0.3\n')
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
parser = argparse.ArgumentParser()
parser.add_argument("--model", default=None)
parser.add_argument("--temperature", type=float, default=0.5)
apply_config(parser, ["model"])
args = parser.parse_args([])
assert args.model == "qwen-72b"
assert args.temperature == 0.3
def test_tavily_key_from_config(tmp_path, monkeypatch):
"""get_tavily_key() reads from config.toml [api] tavily_key."""
_reset_cache()
import turnstone.core.memory as mem
mem._tavily_key = None
mem._tavily_key_loaded = False
cfg = tmp_path / "config.toml"
cfg.write_text('[api]\ntavily_key = "tvly-from-config"\n')
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
monkeypatch.delenv("TAVILY_API_KEY", raising=False)
key = mem.get_tavily_key()
assert key == "tvly-from-config"
def test_tavily_key_fallback_to_env(tmp_path, monkeypatch):
"""get_tavily_key() falls back to $TAVILY_API_KEY env var."""
_reset_cache()
import turnstone.core.memory as mem
mem._tavily_key = None
mem._tavily_key_loaded = False
# Config exists but no tavily_key in it
cfg = tmp_path / "config.toml"
cfg.write_text("[api]\n")
monkeypatch.setattr(config_mod, "CONFIG_PATH", cfg)
monkeypatch.setenv("TAVILY_API_KEY", "tvly-from-env")
key = mem.get_tavily_key()
assert key == "tvly-from-env"
tests/test_console.py
+661
View File
@@ -0,0 +1,661 @@
"""Tests for turnstone.console — collector and HTTP server."""
import json
import queue
import threading
import time
from http.server import HTTPServer, BaseHTTPRequestHandler
from socketserver import ThreadingMixIn
from unittest.mock import MagicMock, patch
import httpx
import pytest
from turnstone.console.collector import ClusterCollector, NodeSnapshot
from turnstone.mq.protocol import (
ClusterStateEvent,
WorkstreamClosedEvent,
WorkstreamCreatedEvent,
WorkstreamRenameEvent,
)
# ---------------------------------------------------------------------------
# Mock broker for collector tests
# ---------------------------------------------------------------------------
class MockBroker:
"""Minimal broker mock that records calls and stores nodes."""
def __init__(self):
self.nodes: list[dict] = []
self._subscriptions: dict[str, list] = {}
def list_nodes(self) -> list[dict]:
return list(self.nodes)
def subscribe_outbound(self, channel, callback):
self._subscriptions.setdefault(channel, []).append(callback)
def publish_outbound(self, channel, event):
for cb in self._subscriptions.get(channel, []):
cb(event)
def subscribe_cluster(self, callback):
channel = "turnstone:events:cluster"
self.subscribe_outbound(channel, callback)
def close(self):
pass
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_collector(broker=None, poll_interval=999, discovery_interval=999):
"""Create a collector with long intervals so threads don't auto-fire."""
b = broker or MockBroker()
return ClusterCollector(
broker=b,
poll_interval=poll_interval,
discovery_interval=discovery_interval,
)
def _dashboard_response(workstreams=None, aggregate=None):
"""Build a /api/dashboard-style response dict."""
return {
"workstreams": workstreams or [],
"aggregate": aggregate
or {
"total_tokens": 0,
"total_tool_calls": 0,
"active_count": 0,
"total_count": 0,
"uptime_seconds": 0,
"node": "local",
},
}
# ---------------------------------------------------------------------------
# ClusterCollector — unit tests
# ---------------------------------------------------------------------------
class TestCollectorDiscovery:
"""Node discovery from heartbeat keys."""
def test_discover_new_nodes(self):
broker = MockBroker()
broker.nodes = [
{"node_id": "node-a", "server_url": "http://a:8080"},
{"node_id": "node-b", "server_url": "http://b:8080"},
]
c = _make_collector(broker)
c._discover_nodes()
overview = c.get_overview()
assert overview["nodes"] == 2
def test_discover_removes_lost_nodes(self):
broker = MockBroker()
broker.nodes = [{"node_id": "node-a", "server_url": "http://a:8080"}]
c = _make_collector(broker)
c._discover_nodes()
assert c.get_overview()["nodes"] == 1
# Node disappears
broker.nodes = []
c._discover_nodes()
assert c.get_overview()["nodes"] == 0
def test_discover_updates_server_url(self):
broker = MockBroker()
broker.nodes = [{"node_id": "node-a", "server_url": "http://a:8080"}]
c = _make_collector(broker)
c._discover_nodes()
broker.nodes = [{"node_id": "node-a", "server_url": "http://a:9090"}]
c._discover_nodes()
detail = c.get_node_detail("node-a")
assert detail["server_url"] == "http://a:9090"
def test_discover_emits_node_joined_event(self):
broker = MockBroker()
c = _make_collector(broker)
events = []
q = queue.Queue()
c.register_listener(q)
broker.nodes = [{"node_id": "node-a", "server_url": "http://a:8080"}]
c._discover_nodes()
event = q.get_nowait()
assert event["type"] == "node_joined"
assert event["node_id"] == "node-a"
def test_discover_emits_node_lost_event(self):
broker = MockBroker()
broker.nodes = [{"node_id": "node-a", "server_url": "http://a:8080"}]
c = _make_collector(broker)
c._discover_nodes()
q = queue.Queue()
c.register_listener(q)
broker.nodes = []
c._discover_nodes()
event = q.get_nowait()
assert event["type"] == "node_lost"
assert event["node_id"] == "node-a"
class TestCollectorPolling:
"""Polling /api/dashboard from nodes."""
def test_apply_poll_populates_workstreams(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
dashboard = _dashboard_response(
workstreams=[
{
"id": "ws1",
"name": "test",
"state": "running",
"tokens": 1000,
"context_ratio": 0.15,
"activity": "bash: ls",
"activity_state": "tool",
"tool_calls": 3,
"title": "My task",
},
],
aggregate={"total_tokens": 1000, "total_tool_calls": 3},
)
c._apply_poll("node-a", dashboard, {"status": "ok"})
detail = c.get_node_detail("node-a")
assert len(detail["workstreams"]) == 1
assert detail["workstreams"][0]["name"] == "test"
assert detail["workstreams"][0]["node"] == "node-a"
assert detail["health"]["status"] == "ok"
def test_apply_poll_replaces_stale_workstreams(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={"old-ws": {"id": "old-ws", "name": "old", "state": "idle"}},
)
dashboard = _dashboard_response(
workstreams=[{"id": "new-ws", "name": "new", "state": "running"}]
)
c._apply_poll("node-a", dashboard, {})
detail = c.get_node_detail("node-a")
assert len(detail["workstreams"]) == 1
assert detail["workstreams"][0]["id"] == "new-ws"
def test_apply_poll_ignores_unknown_node(self):
c = _make_collector()
# Should not raise
c._apply_poll("unknown", _dashboard_response(), {})
class TestCollectorEvents:
"""Real-time event handling from cluster channel."""
def test_cluster_state_event_updates_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={
"ws1": {"id": "ws1", "name": "test", "state": "idle", "node": "node-a"}
},
)
event = ClusterStateEvent(
ws_id="ws1",
state="running",
node_id="node-a",
tokens=5000,
context_ratio=0.25,
activity="bash: echo hi",
activity_state="tool",
)
c._on_cluster_event(event.to_json())
ws = c._nodes["node-a"].workstreams["ws1"]
assert ws["state"] == "running"
assert ws["tokens"] == 5000
assert ws["activity"] == "bash: echo hi"
def test_ws_created_event_adds_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a", server_url="http://a:8080")
event_json = json.dumps(
{
"type": "ws_created",
"ws_id": "ws-new",
"name": "new-task",
"node_id": "node-a",
"correlation_id": "abc",
}
)
c._on_cluster_event(event_json)
assert "ws-new" in c._nodes["node-a"].workstreams
assert c._nodes["node-a"].workstreams["ws-new"]["name"] == "new-task"
def test_ws_closed_event_removes_workstream(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
workstreams={"ws1": {"id": "ws1", "state": "idle"}},
)
event_json = json.dumps({"type": "ws_closed", "ws_id": "ws1"})
c._on_cluster_event(event_json)
assert "ws1" not in c._nodes["node-a"].workstreams
def test_ws_rename_event_updates_name(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
workstreams={"ws1": {"id": "ws1", "name": "old-name", "state": "idle"}},
)
event_json = json.dumps(
{"type": "ws_rename", "ws_id": "ws1", "name": "new-name"}
)
c._on_cluster_event(event_json)
assert c._nodes["node-a"].workstreams["ws1"]["name"] == "new-name"
def test_event_fans_out_to_listeners(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
workstreams={"ws1": {"id": "ws1", "state": "idle", "node": "node-a"}},
)
q = queue.Queue()
c.register_listener(q)
event = ClusterStateEvent(ws_id="ws1", state="running", node_id="node-a")
c._on_cluster_event(event.to_json())
fan_event = q.get_nowait()
assert fan_event["type"] == "cluster_state"
assert fan_event["ws_id"] == "ws1"
def test_unregister_listener_stops_fanout(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(node_id="node-a")
q = queue.Queue()
c.register_listener(q)
c.unregister_listener(q)
c._fanout({"type": "test"})
assert q.empty()
def test_invalid_json_event_ignored(self):
c = _make_collector()
# Should not raise
c._on_cluster_event("not valid json {{{")
c._on_cluster_event("")
class TestCollectorQueries:
"""Query methods: get_overview, get_nodes, get_workstreams, get_node_detail."""
@pytest.fixture()
def populated_collector(self):
c = _make_collector()
c._nodes["node-a"] = NodeSnapshot(
node_id="node-a",
server_url="http://a:8080",
workstreams={
"ws1": {
"id": "ws1",
"name": "alpha",
"state": "running",
"node": "node-a",
"title": "Task A",
"tokens": 5000,
"context_ratio": 0.2,
"activity": "",
"activity_state": "",
"tool_calls": 10,
},
"ws2": {
"id": "ws2",
"name": "beta",
"state": "idle",
"node": "node-a",
"title": "Task B",
"tokens": 2000,
"context_ratio": 0.1,
"activity": "",
"activity_state": "",
"tool_calls": 5,
},
},
aggregate={"total_tokens": 7000, "total_tool_calls": 15},
)
c._nodes["node-b"] = NodeSnapshot(
node_id="node-b",
server_url="http://b:8080",
workstreams={
"ws3": {
"id": "ws3",
"name": "gamma",
"state": "attention",
"node": "node-b",
"title": "Task C",
"tokens": 10000,
"context_ratio": 0.5,
"activity": "awaiting approval",
"activity_state": "approval",
"tool_calls": 20,
},
},
aggregate={"total_tokens": 10000, "total_tool_calls": 20},
)
return c
def test_get_overview(self, populated_collector):
o = populated_collector.get_overview()
assert o["nodes"] == 2
assert o["workstreams"] == 3
assert o["states"]["running"] == 1
assert o["states"]["idle"] == 1
assert o["states"]["attention"] == 1
assert o["aggregate"]["total_tokens"] == 17000
assert o["aggregate"]["total_tool_calls"] == 35
def test_get_nodes_sorted_by_activity(self, populated_collector):
nodes, total = populated_collector.get_nodes(sort_by="activity")
assert total == 2
# node-a has 1 running, node-b has 1 attention — both have activity=1
# order depends on tie-breaking but both should be present
ids = [n["node_id"] for n in nodes]
assert "node-a" in ids
assert "node-b" in ids
def test_get_nodes_pagination(self, populated_collector):
nodes, total = populated_collector.get_nodes(limit=1, offset=0)
assert len(nodes) == 1
assert total == 2
nodes2, _ = populated_collector.get_nodes(limit=1, offset=1)
assert len(nodes2) == 1
assert nodes2[0]["node_id"] != nodes[0]["node_id"]
def test_get_workstreams_no_filter(self, populated_collector):
ws, total = populated_collector.get_workstreams()
assert total == 3
assert len(ws) == 3
def test_get_workstreams_filter_by_state(self, populated_collector):
ws, total = populated_collector.get_workstreams(state="running")
assert total == 1
assert ws[0]["name"] == "alpha"
def test_get_workstreams_filter_by_node(self, populated_collector):
ws, total = populated_collector.get_workstreams(node="node-b")
assert total == 1
assert ws[0]["name"] == "gamma"
def test_get_workstreams_filter_by_search(self, populated_collector):
ws, total = populated_collector.get_workstreams(search="Task C")
assert total == 1
assert ws[0]["name"] == "gamma"
def test_get_workstreams_search_case_insensitive(self, populated_collector):
ws, total = populated_collector.get_workstreams(search="task c")
assert total == 1
def test_get_workstreams_pagination(self, populated_collector):
ws, total = populated_collector.get_workstreams(page=1, per_page=2)
assert len(ws) == 2
assert total == 3
ws2, _ = populated_collector.get_workstreams(page=2, per_page=2)
assert len(ws2) == 1
def test_get_workstreams_sorted_by_state(self, populated_collector):
ws, _ = populated_collector.get_workstreams(sort_by="state")
states = [w["state"] for w in ws]
# running before attention before idle
assert (
states.index("running") < states.index("attention") < states.index("idle")
)
def test_get_workstreams_combined_filters(self, populated_collector):
ws, total = populated_collector.get_workstreams(state="idle", node="node-a")
assert total == 1
assert ws[0]["name"] == "beta"
def test_get_node_detail_found(self, populated_collector):
detail = populated_collector.get_node_detail("node-a")
assert detail is not None
assert detail["node_id"] == "node-a"
assert len(detail["workstreams"]) == 2
def test_get_node_detail_not_found(self, populated_collector):
assert populated_collector.get_node_detail("nonexistent") is None
# ---------------------------------------------------------------------------
# ClusterStateEvent protocol tests
# ---------------------------------------------------------------------------
class TestClusterStateEventProtocol:
"""Ensure ClusterStateEvent round-trips through JSON correctly."""
def test_round_trip(self):
event = ClusterStateEvent(
ws_id="ws1",
state="running",
node_id="node-a",
tokens=5000,
context_ratio=0.25,
activity="bash: ls",
activity_state="tool",
)
raw = event.to_json()
data = json.loads(raw)
assert data["type"] == "cluster_state"
assert data["ws_id"] == "ws1"
assert data["node_id"] == "node-a"
assert data["tokens"] == 5000
assert data["context_ratio"] == 0.25
def test_from_json(self):
from turnstone.mq.protocol import OutboundEvent
raw = json.dumps(
{
"type": "cluster_state",
"ws_id": "ws1",
"state": "running",
"node_id": "node-a",
"tokens": 5000,
}
)
event = OutboundEvent.from_json(raw)
assert isinstance(event, ClusterStateEvent)
assert event.node_id == "node-a"
assert event.tokens == 5000
# ---------------------------------------------------------------------------
# Console HTTP server tests
# ---------------------------------------------------------------------------
class TestConsoleHTTPEndpoints:
"""Test console HTTP API endpoints with a mock collector."""
@pytest.fixture()
def mock_collector(self):
collector = MagicMock(spec=ClusterCollector)
collector.get_overview.return_value = {
"nodes": 3,
"workstreams": 15,
"states": {
"running": 5,
"thinking": 2,
"attention": 1,
"idle": 6,
"error": 1,
},
"aggregate": {"total_tokens": 50000, "total_tool_calls": 200},
}
collector.get_nodes.return_value = (
[
{
"node_id": "node-a",
"ws_total": 5,
"ws_running": 3,
"total_tokens": 20000,
}
],
1,
)
collector.get_workstreams.return_value = (
[{"id": "ws1", "name": "test", "state": "running", "node": "node-a"}],
1,
)
collector.get_node_detail.return_value = {
"node_id": "node-a",
"server_url": "http://a:8080",
"health": {},
"workstreams": [],
"aggregate": {},
}
return collector
@pytest.fixture()
def server(self, mock_collector):
from turnstone.console.server import (
ConsoleHTTPHandler,
ThreadedHTTPServer,
_load_static,
)
_load_static()
from turnstone.core.auth import AuthConfig
httpd = ThreadedHTTPServer(("127.0.0.1", 0), ConsoleHTTPHandler)
httpd.collector = mock_collector
httpd.auth_config = AuthConfig() # auth disabled by default
port = httpd.server_address[1]
thread = threading.Thread(target=httpd.serve_forever, daemon=True)
thread.start()
yield f"http://127.0.0.1:{port}"
httpd.shutdown()
def _get(self, server, path):
resp = httpx.get(f"{server}{path}", timeout=5)
return resp.status_code, resp.json()
def _get_raw(self, server, path):
resp = httpx.get(f"{server}{path}", timeout=5)
return resp.status_code, resp.text, resp.headers.get("content-type")
def test_get_overview(self, server, mock_collector):
status, data = self._get(server, "/api/cluster/overview")
assert status == 200
assert data["nodes"] == 3
assert data["workstreams"] == 15
assert data["states"]["running"] == 5
mock_collector.get_overview.assert_called_once()
def test_get_nodes(self, server, mock_collector):
status, data = self._get(
server, "/api/cluster/nodes?sort=activity&limit=10&offset=0"
)
assert status == 200
assert len(data["nodes"]) == 1
assert data["total"] == 1
mock_collector.get_nodes.assert_called_once_with(
sort_by="activity", limit=10, offset=0
)
def test_get_workstreams(self, server, mock_collector):
status, data = self._get(
server, "/api/cluster/workstreams?state=running&page=1&per_page=25"
)
assert status == 200
assert len(data["workstreams"]) == 1
assert data["total"] == 1
assert data["page"] == 1
assert data["pages"] == 1
mock_collector.get_workstreams.assert_called_once_with(
state="running",
node=None,
search=None,
sort_by="state",
page=1,
per_page=25,
)
def test_get_workstreams_per_page_capped(self, server, mock_collector):
self._get(server, "/api/cluster/workstreams?per_page=999")
call_kwargs = mock_collector.get_workstreams.call_args
assert call_kwargs.kwargs["per_page"] == 200
def test_get_node_detail(self, server, mock_collector):
status, data = self._get(server, "/api/cluster/node/node-a")
assert status == 200
assert data["node_id"] == "node-a"
mock_collector.get_node_detail.assert_called_once_with("node-a")
def test_get_node_detail_not_found(self, server, mock_collector):
mock_collector.get_node_detail.return_value = None
status, data = self._get(server, "/api/cluster/node/nonexistent")
assert status == 404
assert "error" in data
def test_health_endpoint(self, server, mock_collector):
status, data = self._get(server, "/health")
assert status == 200
assert data["status"] == "ok"
assert data["service"] == "turnstone-console"
assert data["nodes"] == 3
def test_index_html(self, server):
status, body, ct = self._get_raw(server, "/")
assert status == 200
assert "text/html" in ct
assert "turnstone console" in body
def test_static_css(self, server):
status, body, ct = self._get_raw(server, "/static/style.css")
assert status == 200
assert "text/css" in ct
def test_static_js(self, server):
status, body, ct = self._get_raw(server, "/static/app.js")
assert status == 200
assert "javascript" in ct
def test_404(self, server):
resp = httpx.get(f"{server}/nonexistent", timeout=5)
assert resp.status_code == 404
tests/test_db.py
+87
View File
@@ -0,0 +1,87 @@
"""Tests for turnstone.core.memory — database operations."""
import turnstone.core.memory as memory
from turnstone.core.memory import (
open_db,
save_message,
search_history,
search_history_recent,
normalize_key,
)
class TestOpenDb:
def test_creates_tables(self, tmp_db):
conn = open_db()
try:
# Check memories table exists
rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='memories'"
).fetchall()
assert len(rows) == 1
# Check conversations table exists
rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='conversations'"
).fetchall()
assert len(rows) == 1
finally:
conn.close()
def test_idempotent_open(self, tmp_db):
# Opening twice should not raise
conn1 = open_db()
conn1.close()
conn2 = open_db()
conn2.close()
class TestSaveAndSearchHistory:
def test_save_and_search_roundtrip(self, tmp_db):
save_message("sess1", "user", "hello world test message")
results = search_history("hello")
assert len(results) >= 1
# Result tuple: (timestamp, session_id, role, content, tool_name)
found = any(r[3] == "hello world test message" for r in results)
assert found
def test_search_empty_query_returns_empty(self, tmp_db):
save_message("sess1", "user", "something")
assert search_history("") == []
assert search_history(" ") == []
def test_search_no_match(self, tmp_db):
save_message("sess1", "user", "hello world")
results = search_history("zzzznotfound")
assert results == []
class TestSearchHistoryRecent:
def test_returns_recent_messages(self, tmp_db):
save_message("sess1", "user", "first message")
save_message("sess1", "assistant", "second message")
results = search_history_recent(limit=10)
assert len(results) == 2
def test_respects_limit(self, tmp_db):
for i in range(5):
save_message("sess1", "user", f"message {i}")
results = search_history_recent(limit=3)
assert len(results) == 3
class TestNormalizeKey:
def test_lowercase(self):
assert normalize_key("Hello") == "hello"
def test_hyphens_to_underscores(self):
assert normalize_key("my-key") == "my_key"
def test_spaces_to_underscores(self):
assert normalize_key("my key") == "my_key"
def test_combined(self):
assert normalize_key("My-Key Name") == "my_key_name"
def test_already_normalized(self):
assert normalize_key("my_key") == "my_key"
tests/test_edit.py
+60
View File
@@ -0,0 +1,60 @@
"""Tests for turnstone.core.edit — find_occurrences and pick_nearest."""
from turnstone.core.edit import find_occurrences, pick_nearest
class TestFindOccurrences:
def test_empty_old_string_returns_empty(self):
assert find_occurrences("hello world", "") == []
def test_no_match_returns_empty(self):
assert find_occurrences("hello world", "xyz") == []
def test_single_occurrence(self):
content = "line one\nline two\nline three"
assert find_occurrences(content, "two") == [2]
def test_multiple_occurrences(self):
content = "aaa\nbbb\naaa\nccc\naaa"
assert find_occurrences(content, "aaa") == [1, 3, 5]
def test_occurrence_on_first_line(self):
content = "hello\nworld"
assert find_occurrences(content, "hello") == [1]
def test_multiline_match(self):
content = "start\nfoo\nbar\nend"
assert find_occurrences(content, "foo\nbar") == [2]
def test_overlapping_positions(self):
content = "aaa"
# "aa" occurs at index 0 (line 1) and index 1 (line 1)
assert find_occurrences(content, "aa") == [1, 1]
def test_empty_content(self):
assert find_occurrences("", "hello") == []
class TestPickNearest:
def test_returns_char_index(self):
content = "aaa\nbbb\nccc"
idx = pick_nearest(content, "bbb", 2)
assert idx == 4 # "bbb" starts at char index 4
def test_picks_nearest_to_target_line(self):
content = "xxx\naaa\nbbb\nccc\naaa\nddd"
# "aaa" on line 2 (idx=4) and line 5 (idx=16)
# Near line 5 should pick second occurrence
idx = pick_nearest(content, "aaa", 5)
assert content[idx : idx + 3] == "aaa"
assert idx == 16
def test_picks_nearest_first_occurrence(self):
content = "xxx\naaa\nbbb\nccc\naaa\nddd"
# Near line 1 should pick first occurrence
idx = pick_nearest(content, "aaa", 1)
assert idx == 4
def test_no_match_returns_negative(self):
content = "hello world"
assert pick_nearest(content, "xyz", 1) == -1
tests/test_fts5.py
+50
View File
@@ -0,0 +1,50 @@
"""Tests for turnstone.core.memory — fts5_query and escape_like."""
from turnstone.core.memory import fts5_query, escape_like
class TestFts5Query:
def test_single_word(self):
result = fts5_query("hello")
assert result == '"hello"'
def test_multiple_words_joined_with_and(self):
result = fts5_query("hello world")
# Each word is quoted; space between = implicit AND
assert result == '"hello" "world"'
def test_special_chars_safely_quoted(self):
result = fts5_query("test*")
assert result == '"test*"'
def test_dash_safely_quoted(self):
result = fts5_query("-negative")
assert result == '"-negative"'
def test_embedded_double_quotes(self):
# Double quotes inside a term are doubled per FTS5 convention
result = fts5_query('say"hello')
assert result == '"say""hello"'
def test_empty_query(self):
assert fts5_query("") == ""
def test_whitespace_only(self):
assert fts5_query(" ") == ""
class TestEscapeLike:
def test_percent_escaped(self):
assert escape_like("100%") == "100\\%"
def test_underscore_escaped(self):
assert escape_like("a_b") == "a\\_b"
def test_backslash_escaped(self):
assert escape_like("a\\b") == "a\\\\b"
def test_no_metacharacters(self):
assert escape_like("hello") == "hello"
def test_combined(self):
assert escape_like("50%_off\\sale") == "50\\%\\_off\\\\sale"
tests/test_html.py
+39
View File
@@ -0,0 +1,39 @@
"""Tests for turnstone.core.web — strip_html."""
from turnstone.core.web import strip_html
class TestStripHtml:
def test_removes_tags(self):
assert strip_html("<b>hello</b>") == "hello"
def test_removes_nested_tags(self):
assert strip_html("<div><p>text</p></div>") == "text"
def test_decodes_entities(self):
assert strip_html("&amp; &lt; &gt;") == "& < >"
def test_collapses_whitespace(self):
result = strip_html("hello world")
assert result == "hello world"
def test_collapses_blank_lines(self):
result = strip_html("a\n\n\n\n\nb")
assert result == "a\n\nb"
def test_empty_string(self):
assert strip_html("") == ""
def test_strips_leading_trailing_whitespace(self):
assert strip_html(" hello ") == "hello"
def test_complex_html(self):
html = "<html><body><h1>Title</h1><p>Some &amp; text</p></body></html>"
result = strip_html(html)
assert "Title" in result
assert "Some & text" in result
assert "<" not in result
def test_self_closing_tags(self):
result = strip_html("hello<br/>world")
assert result == "helloworld"
tests/test_markdown.py
+80
View File
@@ -0,0 +1,80 @@
"""Tests for turnstone.ui.markdown — MarkdownRenderer."""
from turnstone.ui.markdown import MarkdownRenderer
from turnstone.ui.colors import BOLD, MAGENTA, CYAN, DIM, ITALIC, RESET
class TestMarkdownRenderer:
def setup_method(self):
self.r = MarkdownRenderer()
def test_header_rendering(self):
result = self.r.feed("# Hello\n")
assert BOLD in result
assert MAGENTA in result
assert "Hello" in result
def test_h2_header(self):
result = self.r.feed("## Sub\n")
assert BOLD in result
assert MAGENTA in result
assert "Sub" in result
def test_bold_text(self):
result = self.r.feed("some **bold** text\n")
assert BOLD in result
assert "bold" in result
def test_underscore_bold(self):
result = self.r.feed("some __bold__ text\n")
assert BOLD in result
assert "bold" in result
def test_inline_code(self):
result = self.r.feed("use `code` here\n")
assert CYAN in result
assert "code" in result
def test_code_block_toggle(self):
# Opening fence
result = self.r.feed("```python\n")
assert DIM in result
assert self.r.in_code_block is True
# Content inside code block
result = self.r.feed("x = 1\n")
assert CYAN in result
# Closing fence
result = self.r.feed("```\n")
assert DIM in result
assert self.r.in_code_block is False
def test_bullet_list_cyan(self):
result = self.r.feed("- item one\n")
assert CYAN in result
def test_asterisk_bullet_list(self):
result = self.r.feed("* item one\n")
assert CYAN in result
def test_numbered_list_cyan(self):
result = self.r.feed("1. first\n")
assert CYAN in result
def test_flush_returns_remaining_buffer(self):
# Feed text without a newline
result = self.r.feed("no newline yet")
assert result == "" # No complete line yet
# Flush should return the buffered content
result = self.r.flush()
assert "no newline yet" in result
def test_flush_empty_buffer(self):
assert self.r.flush() == ""
def test_italic_text(self):
result = self.r.feed("some *italic* text\n")
assert ITALIC in result
assert "italic" in result
tests/test_protocol.py
+224
View File
@@ -0,0 +1,224 @@
"""Tests for turnstone.mq.protocol message serialization."""
import json
import pytest
from turnstone.mq.protocol import (
AckEvent,
ApprovalRequestEvent,
ApproveMessage,
CloseWorkstreamMessage,
CommandMessage,
ContentEvent,
CreateWorkstreamMessage,
ErrorEvent,
HealthMessage,
HealthResponseEvent,
InboundMessage,
InfoEvent,
ListNodesMessage,
ListWorkstreamsMessage,
NodeListEvent,
OutboundEvent,
PlanFeedbackMessage,
PlanReviewEvent,
ReasoningEvent,
SendMessage,
StateChangeEvent,
StatusEvent,
StreamEndEvent,
ToolInfoEvent,
ToolResultEvent,
TurnCompleteEvent,
WorkstreamClosedEvent,
WorkstreamCreatedEvent,
WorkstreamListEvent,
WorkstreamRenameEvent,
)
# ---------------------------------------------------------------------------
# Inbound message round-trip tests
# ---------------------------------------------------------------------------
INBOUND_TYPES = [
(
SendMessage,
{
"message": "hello",
"ws_id": "abc",
"auto_approve": True,
"auto_approve_tools": ["bash"],
},
),
(
ApproveMessage,
{"ws_id": "abc", "request_id": "r1", "approved": True, "feedback": "ok"},
),
(
PlanFeedbackMessage,
{"ws_id": "abc", "request_id": "r2", "feedback": "looks good"},
),
(CommandMessage, {"ws_id": "abc", "command": "/clear"}),
(
CreateWorkstreamMessage,
{"name": "test-ws", "auto_approve": False, "auto_approve_tools": ["read_file"]},
),
(CloseWorkstreamMessage, {"ws_id": "abc"}),
(ListWorkstreamsMessage, {}),
(HealthMessage, {}),
(ListNodesMessage, {}),
]
@pytest.mark.parametrize("cls,kwargs", INBOUND_TYPES)
def test_inbound_round_trip(cls, kwargs):
msg = cls(**kwargs)
raw = msg.to_json()
parsed = json.loads(raw)
# type field matches
assert parsed["type"] == msg.type
# correlation_id auto-generated
assert len(msg.correlation_id) == 12
assert parsed["correlation_id"] == msg.correlation_id
# timestamp present
assert msg.timestamp > 0
# Deserialize back
restored = InboundMessage.from_json(raw)
assert type(restored) is cls
assert restored.type == msg.type
assert restored.correlation_id == msg.correlation_id
# Check custom fields
for k, v in kwargs.items():
assert getattr(restored, k) == v
def test_inbound_unknown_type():
with pytest.raises(ValueError, match="Unknown inbound"):
InboundMessage.from_json('{"type": "nonexistent"}')
def test_inbound_extra_fields_ignored():
raw = json.dumps({"type": "send", "message": "hi", "extra_field": 42})
msg = InboundMessage.from_json(raw)
assert isinstance(msg, SendMessage)
assert msg.message == "hi"
assert not hasattr(msg, "extra_field")
# ---------------------------------------------------------------------------
# Outbound event round-trip tests
# ---------------------------------------------------------------------------
OUTBOUND_TYPES = [
(AckEvent, {"status": "ok", "detail": "done"}),
(ContentEvent, {"text": "hello world"}),
(ReasoningEvent, {"text": "thinking..."}),
(ToolInfoEvent, {"items": [{"name": "bash", "preview": "ls"}]}),
(ApprovalRequestEvent, {"items": [{"name": "bash", "needs_approval": True}]}),
(ToolResultEvent, {"name": "bash", "output": "file.txt"}),
(PlanReviewEvent, {"content": "# Plan\n\nStep 1: ..."}),
(StatusEvent, {"prompt_tokens": 100, "completion_tokens": 50, "pct": 0.42}),
(StateChangeEvent, {"state": "thinking"}),
(TurnCompleteEvent, {}),
(StreamEndEvent, {}),
(WorkstreamCreatedEvent, {"name": "test-ws"}),
(WorkstreamClosedEvent, {}),
(WorkstreamListEvent, {"workstreams": [{"id": "abc", "name": "ws"}]}),
(WorkstreamRenameEvent, {"name": "renamed"}),
(HealthResponseEvent, {"data": {"status": "ok"}}),
(ErrorEvent, {"message": "something broke"}),
(InfoEvent, {"message": "heads up"}),
(
NodeListEvent,
{"nodes": [{"node_id": "server-12", "server_url": "http://x:8080"}]},
),
]
@pytest.mark.parametrize("cls,kwargs", OUTBOUND_TYPES)
def test_outbound_round_trip(cls, kwargs):
event = cls(ws_id="ws1", correlation_id="c1", **kwargs)
raw = event.to_json()
parsed = json.loads(raw)
assert parsed["type"] == event.type
assert parsed["ws_id"] == "ws1"
assert parsed["correlation_id"] == "c1"
restored = OutboundEvent.from_json(raw)
assert type(restored) is cls
assert restored.ws_id == "ws1"
assert restored.correlation_id == "c1"
for k, v in kwargs.items():
assert getattr(restored, k) == v
def test_outbound_unknown_type_falls_back():
raw = json.dumps({"type": "future_event", "ws_id": "x"})
event = OutboundEvent.from_json(raw)
assert isinstance(event, OutboundEvent)
assert event.ws_id == "x"
def test_send_message_defaults():
msg = SendMessage(message="hello")
assert msg.ws_id == ""
assert msg.auto_approve is False
assert msg.auto_approve_tools == []
assert msg.name == ""
assert msg.target_node == ""
assert len(msg.correlation_id) == 12
def test_create_workstream_with_tools():
msg = CreateWorkstreamMessage(
name="ci-runner",
auto_approve=False,
auto_approve_tools=["bash", "read_file", "search"],
)
raw = msg.to_json()
restored = InboundMessage.from_json(raw)
assert restored.auto_approve_tools == ["bash", "read_file", "search"]
assert restored.name == "ci-runner"
def test_send_message_target_node():
msg = SendMessage(message="check disk", target_node="server-12")
raw = msg.to_json()
restored = InboundMessage.from_json(raw)
assert isinstance(restored, SendMessage)
assert restored.target_node == "server-12"
assert restored.message == "check disk"
def test_create_workstream_target_node():
msg = CreateWorkstreamMessage(name="debug-ws", target_node="gpu-node-3")
raw = msg.to_json()
restored = InboundMessage.from_json(raw)
assert isinstance(restored, CreateWorkstreamMessage)
assert restored.target_node == "gpu-node-3"
assert restored.name == "debug-ws"
def test_list_nodes_round_trip():
msg = ListNodesMessage()
raw = msg.to_json()
restored = InboundMessage.from_json(raw)
assert isinstance(restored, ListNodesMessage)
def test_node_list_event_round_trip():
nodes = [{"node_id": "a", "server_url": "http://a:8080"}]
event = NodeListEvent(nodes=nodes, correlation_id="c1")
raw = event.to_json()
restored = OutboundEvent.from_json(raw)
assert isinstance(restored, NodeListEvent)
assert restored.nodes == nodes
tests/test_safety.py
+69
View File
@@ -0,0 +1,69 @@
"""Tests for turnstone.core.safety — is_command_blocked and sanitize_command."""
from turnstone.core.safety import is_command_blocked, sanitize_command
class TestIsCommandBlocked:
def test_rm_rf_root_blocked(self):
result = is_command_blocked("rm -rf /")
assert result is not None
assert "Blocked" in result
def test_rm_rf_star_blocked(self):
result = is_command_blocked("rm -rf /*")
assert result is not None
def test_mkfs_blocked(self):
result = is_command_blocked("mkfs /dev/sda")
assert result is not None
def test_shutdown_blocked(self):
result = is_command_blocked("shutdown -h now")
assert result is not None
def test_fork_bomb_blocked(self):
result = is_command_blocked(":(){ :|:& };:")
assert result is not None
def test_dd_if_blocked(self):
result = is_command_blocked("dd if=/dev/zero of=/dev/sda")
assert result is not None
def test_safe_ls_returns_none(self):
assert is_command_blocked("ls -la") is None
def test_safe_git_returns_none(self):
assert is_command_blocked("git status") is None
def test_safe_python_returns_none(self):
assert is_command_blocked("python script.py") is None
def test_safe_rm_specific_file(self):
assert is_command_blocked("rm file.txt") is None
def test_whitespace_preserved(self):
# Leading/trailing whitespace is stripped before checking
result = is_command_blocked(" rm -rf / ")
assert result is not None
class TestSanitizeCommand:
def test_left_single_curly_quote_replaced(self):
assert sanitize_command("\u2018hello\u2019") == "'hello'"
def test_double_curly_quotes_replaced(self):
assert sanitize_command("\u201chello\u201d") == '"hello"'
def test_en_dash_replaced(self):
assert sanitize_command("ls \u2013la") == "ls -la"
def test_em_dash_replaced(self):
assert sanitize_command("cmd \u2014flag") == "cmd -flag"
def test_plain_ascii_unchanged(self):
cmd = "git commit -m 'test'"
assert sanitize_command(cmd) == cmd
def test_mixed_replacements(self):
cmd = "\u201ctest\u201d \u2013flag"
assert sanitize_command(cmd) == '"test" -flag'
tests/test_sandbox.py
+99
View File
@@ -0,0 +1,99 @@
"""Tests for turnstone.core.sandbox — validate_math_code and auto_print_wrap."""
from turnstone.core.sandbox import validate_math_code, auto_print_wrap
class TestValidateMathCode:
def test_safe_code_no_errors(self):
assert validate_math_code("x = 1 + 2\nprint(x)") == []
def test_safe_math_import(self):
assert validate_math_code("import math\nprint(math.pi)") == []
def test_blocked_import_os(self):
errors = validate_math_code("import os")
assert len(errors) == 1
assert "os" in errors[0]
def test_blocked_import_sys(self):
errors = validate_math_code("import sys")
assert len(errors) == 1
assert "sys" in errors[0]
def test_blocked_import_subprocess(self):
errors = validate_math_code("import subprocess")
assert len(errors) == 1
assert "subprocess" in errors[0]
def test_blocked_from_import(self):
errors = validate_math_code("from os.path import join")
assert len(errors) == 1
assert "os" in errors[0]
def test_blocked_builtin_exec(self):
errors = validate_math_code("exec('print(1)')")
assert len(errors) == 1
assert "exec" in errors[0]
def test_blocked_builtin_eval(self):
errors = validate_math_code("eval('1+1')")
assert len(errors) == 1
assert "eval" in errors[0]
def test_blocked_builtin_open(self):
errors = validate_math_code("open('file.txt')")
assert len(errors) == 1
assert "open" in errors[0]
def test_blocked_dunder_access(self):
errors = validate_math_code("x.__dict__")
assert len(errors) == 1
assert "__dict__" in errors[0]
def test_allowed_dunder_name(self):
# __name__, __doc__, __class__ are allowed
assert validate_math_code("print(int.__name__)") == []
def test_syntax_error_caught(self):
errors = validate_math_code("def f(\n")
assert len(errors) == 1
assert "Syntax error" in errors[0]
def test_multiple_violations(self):
code = "import os\nimport sys\nexec('x')"
errors = validate_math_code(code)
assert len(errors) == 3
class TestAutoPrintWrap:
def test_bare_expression_wrapped(self):
result = auto_print_wrap("1 + 2")
assert "print(" in result
assert "1 + 2" in result
def test_assignment_not_wrapped(self):
code = "x = 1 + 2"
assert auto_print_wrap(code) == code
def test_code_with_print_not_wrapped(self):
code = "x = 1\nprint(x)"
assert auto_print_wrap(code) == code
def test_code_with_result_assignment_not_wrapped(self):
code = "result = 42"
assert auto_print_wrap(code) == code
def test_multiline_with_bare_expression_last(self):
code = "x = 2\ny = 3\nx + y"
result = auto_print_wrap(code)
assert "print(" in result
# The assignments should still be there
assert "x = 2" in result
assert "y = 3" in result
def test_empty_code(self):
assert auto_print_wrap("") == ""
def test_syntax_error_returns_original(self):
code = "def f(\n"
assert auto_print_wrap(code) == code
tests/test_scoring.py
+166
View File
@@ -0,0 +1,166 @@
"""Tests for turnstone.eval — score_run and _match_action."""
from turnstone.eval import score_run, _match_action
class TestMatchAction:
def test_tool_name_match(self):
actual = {"tool": "bash", "args": {"command": "ls"}}
expected = {"tool": "bash"}
assert _match_action(actual, expected) is True
def test_tool_name_mismatch(self):
actual = {"tool": "bash", "args": {"command": "ls"}}
expected = {"tool": "read_file"}
assert _match_action(actual, expected) is False
def test_exact_args_match(self):
actual = {"tool": "bash", "args": {"command": "ls -la"}}
expected = {"tool": "bash", "args": {"command": "ls -la"}}
assert _match_action(actual, expected) is True
def test_partial_key_matching(self):
# Expected only specifies a subset of actual args
actual = {"tool": "bash", "args": {"command": "ls", "extra": "val"}}
expected = {"tool": "bash", "args": {"command": "ls"}}
assert _match_action(actual, expected) is True
def test_args_value_mismatch(self):
actual = {"tool": "bash", "args": {"command": "ls"}}
expected = {"tool": "bash", "args": {"command": "pwd"}}
assert _match_action(actual, expected) is False
def test_args_missing_key(self):
actual = {"tool": "bash", "args": {"command": "ls"}}
expected = {"tool": "bash", "args": {"path": "/tmp"}}
assert _match_action(actual, expected) is False
def test_args_pattern_regex_match(self):
actual = {"tool": "bash", "args": {"command": "git log -5"}}
expected = {"tool": "bash", "args_pattern": {"command": r"git\s+log"}}
assert _match_action(actual, expected) is True
def test_args_pattern_regex_mismatch(self):
actual = {"tool": "bash", "args": {"command": "ls -la"}}
expected = {"tool": "bash", "args_pattern": {"command": r"^git"}}
assert _match_action(actual, expected) is False
def test_raw_fallback_no_expected_args(self):
actual = {"tool": "bash", "args": {"_raw": "something"}}
expected = {"tool": "bash"}
assert _match_action(actual, expected) is True
def test_raw_fallback_with_expected_args(self):
actual = {"tool": "bash", "args": {"_raw": "something"}}
expected = {"tool": "bash", "args": {"command": "ls"}}
assert _match_action(actual, expected) is False
class TestScoreRun:
def test_empty_expected_actions_passes(self):
result = score_run([{"tool": "bash", "args": {}}], [])
assert result["pass"] is True
assert result["score"] == 1.0
def test_ordered_subset_all_match(self):
tool_log = [
{"tool": "read_file", "args": {"path": "a.py"}},
{"tool": "bash", "args": {"command": "ls"}},
{"tool": "edit_file", "args": {"path": "a.py"}},
]
expected = [
{"tool": "read_file"},
{"tool": "edit_file"},
]
result = score_run(tool_log, expected, match_mode="ordered_subset")
assert result["pass"] is True
assert result["score"] == 1.0
def test_ordered_subset_wrong_order(self):
tool_log = [
{"tool": "edit_file", "args": {"path": "a.py"}},
{"tool": "read_file", "args": {"path": "a.py"}},
]
expected = [
{"tool": "read_file"},
{"tool": "edit_file"},
]
result = score_run(tool_log, expected, match_mode="ordered_subset")
# edit_file comes before read_file, so only one can match
assert result["pass"] is False
assert result["score"] == 0.5
def test_exact_mode_pass(self):
tool_log = [
{"tool": "bash", "args": {"command": "ls"}},
{"tool": "read_file", "args": {"path": "a.py"}},
]
expected = [
{"tool": "bash"},
{"tool": "read_file"},
]
result = score_run(tool_log, expected, match_mode="exact")
assert result["pass"] is True
assert result["score"] == 1.0
def test_exact_mode_length_mismatch(self):
tool_log = [
{"tool": "bash", "args": {"command": "ls"}},
{"tool": "read_file", "args": {"path": "a.py"}},
{"tool": "edit_file", "args": {"path": "a.py"}},
]
expected = [
{"tool": "bash"},
{"tool": "read_file"},
]
result = score_run(tool_log, expected, match_mode="exact")
# Length mismatch: 3 vs 2, so pass=False even though first 2 match
assert result["pass"] is False
def test_subset_mode_unordered(self):
tool_log = [
{"tool": "edit_file", "args": {"path": "a.py"}},
{"tool": "read_file", "args": {"path": "a.py"}},
]
expected = [
{"tool": "read_file"},
{"tool": "edit_file"},
]
result = score_run(tool_log, expected, match_mode="subset")
assert result["pass"] is True
assert result["score"] == 1.0
def test_contains_any_mode_pass(self):
tool_log = [
{"tool": "bash", "args": {"command": "ls"}},
{"tool": "read_file", "args": {"path": "a.py"}},
]
expected = [
{"tool": "read_file"},
]
result = score_run(tool_log, expected, match_mode="contains_any")
assert result["pass"] is True
assert result["score"] == 1.0
def test_contains_any_mode_fail(self):
tool_log = [
{"tool": "bash", "args": {"command": "ls"}},
]
expected = [
{"tool": "read_file"},
]
result = score_run(tool_log, expected, match_mode="contains_any")
assert result["pass"] is False
assert result["score"] == 0.0
def test_score_partial(self):
tool_log = [
{"tool": "bash", "args": {"command": "ls"}},
]
expected = [
{"tool": "bash"},
{"tool": "read_file"},
]
result = score_run(tool_log, expected, match_mode="ordered_subset")
assert result["score"] == 0.5
assert len(result["unmatched"]) == 1
tests/test_server_live.py
+521
View File
@@ -0,0 +1,521 @@
"""Integration tests against a live llama.cpp backend on port 8000.
These tests use turnstone's HeadlessSession to run actual LLM inference
and tool execution against the backend. They verify end-to-end behavior:
model connectivity, tool calling, response quality, and session mechanics.
Requires: llama-server (or compatible OpenAI API) running on localhost:8000.
Run with: pytest tests/test_server_live.py -v --timeout=120
The TestServerHealthMetrics class does NOT require a live LLM and can be run
independently: pytest tests/test_server_live.py::TestServerHealthMetrics -v
"""
import json
import os
import queue
import tempfile
import threading
import time
import httpx
import pytest
from openai import OpenAI
from turnstone.core.session import ChatSession
from turnstone.core.tools import TOOLS
import turnstone.core.memory as _memory_module
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
BASE_URL = os.environ.get("TURNSTONE_TEST_BASE_URL", "http://localhost:8000/v1")
@pytest.fixture(scope="module")
def client():
"""Create an OpenAI client pointed at the local backend."""
return OpenAI(
base_url=BASE_URL,
api_key=os.environ.get("TURNSTONE_TEST_API_KEY", "not-needed"),
)
@pytest.fixture(scope="module")
def model_id(client):
"""Auto-detect the model name from the backend."""
models = client.models.list()
ids = [m.id for m in models.data]
assert len(ids) > 0, "No models found on the backend"
return ids[0]
class RecordingUI:
"""Minimal SessionUI that captures events for assertions."""
def __init__(self):
self.events: list[tuple[str, ...]] = []
self.content_tokens: list[str] = []
self.reasoning_tokens: list[str] = []
self.tool_results: list[tuple[str, str]] = []
self.errors: list[str] = []
self.infos: list[str] = []
def on_thinking_start(self):
self.events.append(("thinking_start",))
def on_thinking_stop(self):
self.events.append(("thinking_stop",))
def on_reasoning_token(self, text):
self.reasoning_tokens.append(text)
def on_content_token(self, text):
self.content_tokens.append(text)
def on_stream_end(self):
self.events.append(("stream_end",))
def approve_tools(self, items):
return True, None # auto-approve everything
def on_tool_result(self, name, output):
self.tool_results.append((name, output))
def on_status(self, usage, context_window, effort):
self.events.append(("status",))
def on_plan_review(self, content):
return ""
def on_info(self, message):
self.infos.append(message)
def on_error(self, message):
self.errors.append(message)
def on_state_change(self, state):
self.events.append(("state_change", state))
def on_rename(self, name: str):
self.events.append(("rename", name))
@property
def full_content(self) -> str:
return "".join(self.content_tokens)
@property
def full_reasoning(self) -> str:
return "".join(self.reasoning_tokens)
@pytest.fixture
def tmp_db():
"""Temp DB to avoid polluting real conversation history."""
with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as f:
path = f.name
old = _memory_module.db_override
_memory_module.db_override = path
_memory_module.db_initialized.discard(path)
yield path
_memory_module.db_override = old
_memory_module.db_initialized.discard(path)
os.unlink(path)
def _make_session(
client, model_id, tmp_db, **kwargs
) -> tuple[ChatSession, RecordingUI]:
"""Create a ChatSession with RecordingUI and sensible test defaults."""
ui = RecordingUI()
defaults = dict(
client=client,
model=model_id,
ui=ui,
persona=None,
instructions=None,
temperature=0.3,
max_tokens=2048,
tool_timeout=30,
reasoning_effort="low",
)
defaults.update(kwargs)
session = ChatSession(**defaults)
session.auto_approve = True
return session, ui
# ---------------------------------------------------------------------------
# Tests — Backend connectivity
# ---------------------------------------------------------------------------
class TestBackendConnectivity:
"""Verify the LLM backend is reachable and returns valid responses."""
def test_models_endpoint(self, client):
models = client.models.list()
assert len(models.data) > 0
def test_model_id_detected(self, model_id):
assert isinstance(model_id, str)
assert len(model_id) > 0
def test_basic_completion(self, client, model_id):
"""Raw API call — no turnstone involved."""
resp = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": "Say 'hello'"}],
max_completion_tokens=200,
temperature=0.0,
stream=False,
)
assert (
resp.choices[0].message.content or resp.choices[0].message.reasoning_content
)
assert resp.usage.total_tokens > 0
# ---------------------------------------------------------------------------
# Tests — Streaming session
# ---------------------------------------------------------------------------
class TestStreamingSession:
"""Test ChatSession.send() with streaming against the live backend."""
def test_simple_response(self, client, model_id, tmp_db):
"""Model responds to a basic prompt via streaming."""
session, ui = _make_session(client, model_id, tmp_db)
session.send("Reply with exactly: PONG")
# Should have gotten some content or reasoning
total = ui.full_content + ui.full_reasoning
assert len(total) > 0, "No output from model"
def test_reasoning_tokens_appear(self, client, model_id, tmp_db):
"""Model produces reasoning tokens (extended thinking)."""
session, ui = _make_session(client, model_id, tmp_db)
session.send("What is 7 * 8?")
# This model uses reasoning_content, so we expect reasoning tokens
assert len(ui.reasoning_tokens) > 0, "No reasoning tokens received"
def test_stream_end_event(self, client, model_id, tmp_db):
"""stream_end event is emitted after response."""
session, ui = _make_session(client, model_id, tmp_db)
session.send("Say hi")
event_types = [e[0] for e in ui.events]
assert "stream_end" in event_types
def test_thinking_lifecycle(self, client, model_id, tmp_db):
"""thinking_start and thinking_stop bracket the response."""
session, ui = _make_session(client, model_id, tmp_db)
session.send("Say hi")
event_types = [e[0] for e in ui.events]
assert "thinking_start" in event_types
assert "thinking_stop" in event_types
# thinking_start should come before thinking_stop
start_idx = event_types.index("thinking_start")
stop_idx = event_types.index("thinking_stop")
assert start_idx < stop_idx
# ---------------------------------------------------------------------------
# Tests — Tool calling
# ---------------------------------------------------------------------------
class TestToolCalling:
"""Test that the model can invoke tools and turnstone executes them."""
def test_math_tool(self, client, model_id, tmp_db):
"""Model uses the math tool for computation."""
session, ui = _make_session(
client,
model_id,
tmp_db,
instructions="You have tools. Use the math tool to compute results. Always use tools when asked to calculate.",
)
session.send("Use the math tool to calculate: 17 * 23. Report the result.")
# Check if math tool was invoked
math_results = [r for r in ui.tool_results if r[0] == "math"]
if math_results:
# Verify the result contains 391
assert "391" in math_results[0][1], (
f"Expected 391, got: {math_results[0][1]}"
)
else:
# Model may have answered directly — check content
total = ui.full_content + ui.full_reasoning
assert "391" in total, f"Expected 391 somewhere in output"
def test_bash_tool(self, client, model_id, tmp_db):
"""Model uses bash to answer a system question."""
session, ui = _make_session(
client,
model_id,
tmp_db,
instructions="You have tools. Use the bash tool to run commands. Always use bash when asked about system info.",
)
session.send(
"Use the bash tool to run 'echo hello_from_test' and report what it prints."
)
bash_results = [r for r in ui.tool_results if r[0] == "bash"]
if bash_results:
assert "hello_from_test" in bash_results[0][1]
else:
total = ui.full_content + ui.full_reasoning
assert "hello_from_test" in total, "Expected bash output in response"
def test_read_file_tool(self, client, model_id, tmp_db):
"""Model uses read_file to read a known file."""
# Create a temp file for the model to read
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
f.write("SECRET_CONTENT_42\n")
path = f.name
try:
session, ui = _make_session(
client,
model_id,
tmp_db,
instructions="You have tools. Use the read_file tool to read files. Always use read_file when asked to read a file.",
)
session.send(
f"Use the read_file tool to read {path} and tell me what it says."
)
# read_file was invoked (UI gets a summary like "1 lines")
read_results = [r for r in ui.tool_results if r[0] == "read_file"]
assert len(read_results) > 0, "read_file tool was not called"
# The model sees the actual file content and should relay it
total = ui.full_content + ui.full_reasoning
assert "SECRET_CONTENT_42" in total, (
f"Model didn't relay file content. Got: {total[:500]}"
)
finally:
os.unlink(path)
# ---------------------------------------------------------------------------
# Tests — Multi-turn conversation
# ---------------------------------------------------------------------------
class TestMultiTurn:
"""Test multi-turn conversation state."""
def test_context_retained(self, client, model_id, tmp_db):
"""Second message can reference the first."""
session, ui = _make_session(client, model_id, tmp_db, max_tokens=1024)
session.send("My name is Zephyr. Remember it.")
# Reset UI tracking for second turn
ui.content_tokens.clear()
ui.reasoning_tokens.clear()
session.send("What is my name?")
total = ui.full_content + ui.full_reasoning
assert "zephyr" in total.lower(), f"Model forgot the name. Got: {total[:300]}"
def test_message_list_grows(self, client, model_id, tmp_db):
"""Each send adds user + assistant messages."""
session, ui = _make_session(client, model_id, tmp_db, max_tokens=512)
initial_count = len(session.messages)
session.send("Hello")
# Should have at least user + assistant
assert len(session.messages) >= initial_count + 2
# ---------------------------------------------------------------------------
# Tests — Session configuration
# ---------------------------------------------------------------------------
class TestSessionConfig:
"""Test session construction and configuration."""
def test_creative_mode_no_tools(self, client, model_id, tmp_db):
"""In creative mode, tools are not sent to the API."""
session, ui = _make_session(client, model_id, tmp_db, max_tokens=256)
session.creative_mode = True
session.send("Write a haiku about code.")
# Should get content back without tool calls
total = ui.full_content + ui.full_reasoning
assert len(total) > 0
assert len(ui.tool_results) == 0
def test_custom_instructions(self, client, model_id, tmp_db):
"""Custom instructions are included in the session."""
session, ui = _make_session(
client,
model_id,
tmp_db,
instructions="Always end your response with ENDMARKER.",
max_tokens=512,
)
session.send("Say hello briefly.")
total = ui.full_content
# We can't strictly guarantee the model follows instructions,
# but we verify the session didn't error out
assert len(ui.errors) == 0
# ---------------------------------------------------------------------------
# Tests — /health and /metrics endpoints (no live LLM required)
# ---------------------------------------------------------------------------
class TestServerHealthMetrics:
"""Verify /health and /metrics endpoints using an in-process HTTP server.
These tests spin up a real ThreadedHTTPServer with a mock WorkstreamManager
so no live LLM backend is required. Run them independently with:
pytest tests/test_server_live.py::TestServerHealthMetrics -v
"""
@classmethod
def setup_class(cls):
from unittest.mock import MagicMock
import turnstone.server as srv_mod
from turnstone.core.workstream import WorkstreamState
# Reset module-level metrics so each test run starts fresh
srv_mod._metrics = srv_mod.MetricsCollector()
srv_mod._metrics.model = "test-model"
# Mock WorkstreamManager.list_all() to return one idle workstream
mock_ws = MagicMock()
mock_ws.state = WorkstreamState.IDLE
mock_mgr = MagicMock()
mock_mgr.list_all.return_value = [mock_ws]
# Start a server on a random port (port 0 → OS assigns free port)
cls.server = srv_mod.ThreadedHTTPServer(
("127.0.0.1", 0), srv_mod.TurnstoneHTTPHandler
)
from turnstone.core.auth import AuthConfig
cls.server.workstreams = mock_mgr
cls.server.skip_permissions = False
cls.server.global_listeners = []
cls.server.global_queue = queue.Queue()
cls.server.global_listeners_lock = threading.Lock()
cls.server.auth_config = AuthConfig() # auth disabled by default
port = cls.server.server_address[1]
cls.base = f"http://127.0.0.1:{port}"
cls._thread = threading.Thread(target=cls.server.serve_forever, daemon=True)
cls._thread.start()
@classmethod
def teardown_class(cls):
cls.server.shutdown()
cls._thread.join(timeout=5)
def _get(self, path) -> tuple[int, str, dict]:
"""Make a GET request; return (status, content_type, body_str)."""
url = self.base + path
resp = httpx.get(url, timeout=5)
ct = resp.headers.get("content-type", "")
return resp.status_code, ct, resp.text
def test_health_returns_200(self):
status, _, _ = self._get("/health")
assert status == 200
def test_health_content_type_json(self):
_, ct, _ = self._get("/health")
assert "application/json" in ct
def test_health_response_structure(self):
_, _, body = self._get("/health")
data = json.loads(body)
assert data["status"] == "ok"
assert "version" in data
assert "uptime_seconds" in data
assert "model" in data
assert "workstreams" in data
def test_health_model_field(self):
_, _, body = self._get("/health")
data = json.loads(body)
assert data["model"] == "test-model"
def test_health_workstream_counts(self):
_, _, body = self._get("/health")
data = json.loads(body)
wss = data["workstreams"]
assert wss["total"] == 1
assert wss["idle"] == 1
def test_health_uptime_positive(self):
_, _, body = self._get("/health")
data = json.loads(body)
assert data["uptime_seconds"] >= 0
def test_metrics_returns_200(self):
status, _, _ = self._get("/metrics")
assert status == 200
def test_metrics_content_type_prometheus(self):
_, ct, _ = self._get("/metrics")
assert "text/plain" in ct
assert "version=0.0.4" in ct
def test_metrics_contains_uptime(self):
_, _, body = self._get("/metrics")
assert "turnstone_uptime_seconds" in body
def test_metrics_contains_build_info(self):
_, _, body = self._get("/metrics")
assert "turnstone_build_info" in body
assert 'model="test-model"' in body
def test_metrics_contains_workstreams(self):
_, _, body = self._get("/metrics")
assert "turnstone_workstreams_active_total" in body
assert "turnstone_workstreams_by_state" in body
def test_metrics_contains_token_counters(self):
_, _, body = self._get("/metrics")
assert "turnstone_tokens_total" in body
assert 'type="prompt"' in body
assert 'type="completion"' in body
def test_metrics_contains_http_requests(self):
_, _, body = self._get("/metrics")
assert "turnstone_http_requests_total" in body
def test_metrics_request_counter_increments(self):
"""Hitting /health increments the HTTP request counter."""
# Make a known request to /health
self._get("/health")
_, _, body = self._get("/metrics")
# Counter should mention /health endpoint
assert 'endpoint="/health"' in body
def test_metrics_histogram_present(self):
_, _, body = self._get("/metrics")
assert "turnstone_http_request_duration_seconds" in body
assert 'le="' in body
assert 'le="+Inf"' in body
def test_unknown_endpoint_returns_404(self):
status, _, _ = self._get("/does-not-exist")
assert status == 404
tests/test_session.py
+275
View File
@@ -0,0 +1,275 @@
"""Tests for turnstone.core.session — ChatSession construction."""
import json
import os
from unittest.mock import MagicMock, patch
from turnstone.core.session import ChatSession
class NullUI:
"""UI adapter that discards all output. Used for testing."""
def on_thinking_start(self):
pass
def on_thinking_stop(self):
pass
def on_reasoning_token(self, text):
pass
def on_content_token(self, text):
pass
def on_stream_end(self):
pass
def approve_tools(self, items):
return True, None
def on_tool_result(self, name, output):
pass
def on_status(self, usage, context_window, effort):
pass
def on_plan_review(self, content):
return ""
def on_info(self, message):
pass
def on_error(self, message):
pass
def on_state_change(self, state):
pass
def _make_session(
mock_openai_client=None,
persona=None,
instructions=None,
**kwargs,
):
"""Helper to construct a ChatSession with minimal setup."""
client = mock_openai_client or MagicMock()
defaults = dict(
client=client,
model="test-model",
ui=NullUI(),
persona=persona,
instructions=instructions,
temperature=0.5,
max_tokens=4096,
tool_timeout=30,
)
defaults.update(kwargs)
return ChatSession(**defaults)
class TestChatSessionConstruction:
def test_system_messages_created(self, tmp_db):
session = _make_session()
assert len(session.system_messages) >= 1
# At least one developer message
roles = [m["role"] for m in session.system_messages]
assert "developer" in roles
def test_persona_injected_into_chat_template_kwargs(self, tmp_db):
session = _make_session(persona="Helpful assistant")
assert "model_identity" in session._chat_template_kwargs
assert "Helpful assistant" in session._chat_template_kwargs["model_identity"]
def test_no_persona_no_model_identity(self, tmp_db):
session = _make_session(persona=None)
assert "model_identity" not in session._chat_template_kwargs
def test_instructions_appended_to_developer_message(self, tmp_db):
session = _make_session(instructions="Always be concise.")
dev_msgs = [m for m in session.system_messages if m["role"] == "developer"]
assert len(dev_msgs) >= 1
assert "Always be concise." in dev_msgs[0]["content"]
def test_full_messages_returns_system_plus_conversation(self, tmp_db):
session = _make_session()
# Initially no conversation messages
full = session._full_messages()
assert len(full) == len(session.system_messages)
# Add a user message
session.messages.append({"role": "user", "content": "hello"})
full = session._full_messages()
assert len(full) == len(session.system_messages) + 1
assert full[-1]["role"] == "user"
def test_msg_char_count_content_only(self, tmp_db):
session = _make_session()
msg = {"role": "assistant", "content": "hello world"}
assert session._msg_char_count(msg) == 11
def test_msg_char_count_with_tool_calls(self, tmp_db):
session = _make_session()
msg = {
"role": "assistant",
"content": "hi",
"tool_calls": [
{
"id": "tc_1",
"function": {
"name": "bash",
"arguments": '{"command": "ls"}',
},
}
],
}
# "hi" (2) + "bash" (4) + '{"command": "ls"}' (17) = 23
assert session._msg_char_count(msg) == 23
def test_msg_char_count_none_content(self, tmp_db):
session = _make_session()
msg = {"role": "assistant", "content": None}
assert session._msg_char_count(msg) == 0
def test_reasoning_effort_stored(self, tmp_db):
session = _make_session(reasoning_effort="high")
assert session.reasoning_effort == "high"
def test_default_reasoning_effort(self, tmp_db):
session = _make_session()
assert session.reasoning_effort == "medium"
# ---------------------------------------------------------------------------
# Tests — _exec_plan (session-scoped plan files + existing-plan re-read)
# ---------------------------------------------------------------------------
class TestPlanExec:
"""Tests for _exec_plan: unique session-scoped plan file and existing-plan injection."""
def _run_plan(self, session, prompt, agent_return="# Plan\n\nDo the thing."):
"""Invoke _exec_plan with _run_agent patched to avoid LLM calls.
Returns (call_id_returned, content_returned, captured_messages) where
captured_messages is the agent_messages list passed to _run_agent.
"""
captured = {}
def fake_run_agent(messages, **kwargs):
captured["messages"] = list(messages)
return agent_return
item = {"call_id": "test-call-1", "prompt": prompt}
with patch.object(session, "_run_agent", side_effect=fake_run_agent):
call_id, content = session._exec_plan(item)
return call_id, content, captured.get("messages", [])
def test_plan_file_uses_session_id(self, tmp_db, tmp_path, monkeypatch):
"""Plan file is named .plan-<session_id>.md, not .plan.md."""
monkeypatch.chdir(tmp_path)
session = _make_session()
self._run_plan(session, "add feature")
expected = tmp_path / f".plan-{session._session_id}.md"
assert expected.exists(), f"Expected {expected} to be created"
assert not (tmp_path / ".plan.md").exists()
def test_plan_file_contains_agent_output(self, tmp_db, tmp_path, monkeypatch):
"""Written plan file contains the agent's output verbatim."""
monkeypatch.chdir(tmp_path)
session = _make_session()
plan_content = "## Goal\n\nAdd a new endpoint."
self._run_plan(session, "add endpoint", agent_return=plan_content)
plan_file = tmp_path / f".plan-{session._session_id}.md"
assert plan_file.read_text() == plan_content
def test_two_sessions_produce_different_files(self, tmp_db, tmp_path, monkeypatch):
"""Two ChatSession instances never collide on the same plan file."""
monkeypatch.chdir(tmp_path)
s1 = _make_session()
s2 = _make_session()
assert s1._session_id != s2._session_id
self._run_plan(s1, "feature A")
self._run_plan(s2, "feature B")
files = list(tmp_path.glob(".plan-*.md"))
assert len(files) == 2
def _seed_prior_plan(self, session, prior_prompt, prior_content):
"""Simulate a completed plan tool call in session.messages."""
tc_id = "call_prior_plan"
session.messages.append(
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": tc_id,
"type": "function",
"function": {
"name": "plan",
"arguments": json.dumps({"prompt": prior_prompt}),
},
}
],
}
)
session.messages.append(
{
"role": "tool",
"tool_call_id": tc_id,
"content": prior_content,
}
)
def test_no_prior_plan_no_extra_messages(self, tmp_db, tmp_path, monkeypatch):
"""First invocation: no prior plan in history, agent gets no tool pair."""
monkeypatch.chdir(tmp_path)
session = _make_session()
_, _, messages = self._run_plan(session, "build something")
roles = [m["role"] for m in messages]
assert "tool" not in roles
def test_prior_plan_from_messages_injected(self, tmp_db, tmp_path, monkeypatch):
"""Second invocation: prior plan from session.messages arrives as real tool result."""
monkeypatch.chdir(tmp_path)
session = _make_session()
self._seed_prior_plan(session, "build feature X", "## Goal\n\nOriginal plan.")
_, _, messages = self._run_plan(session, "also handle edge case Y")
# The real assistant tool_calls message is forwarded
assistant_with_tc = [
m for m in messages if m["role"] == "assistant" and m.get("tool_calls")
]
assert len(assistant_with_tc) == 1
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "plan"
# The real tool result is forwarded with its original content
tool_msgs = [m for m in messages if m["role"] == "tool"]
assert len(tool_msgs) == 1
assert "Original plan." in tool_msgs[0]["content"]
def test_prior_plan_appears_before_user_prompt(self, tmp_db, tmp_path, monkeypatch):
"""The prior plan tool pair appears before the new user prompt."""
monkeypatch.chdir(tmp_path)
session = _make_session()
self._seed_prior_plan(session, "original", "Old plan.")
_, _, messages = self._run_plan(session, "refinement prompt")
tool_idx = next(i for i, m in enumerate(messages) if m["role"] == "tool")
user_idx = next(i for i, m in enumerate(messages) if m["role"] == "user")
assert tool_idx < user_idx
def test_exec_plan_returns_content(self, tmp_db, tmp_path, monkeypatch):
"""_exec_plan returns (call_id, agent_output)."""
monkeypatch.chdir(tmp_path)
session = _make_session()
agent_output = "## Goal\n\nBuild it."
call_id, content, _ = self._run_plan(
session, "do stuff", agent_return=agent_output
)
assert call_id == "test-call-1"
assert content == agent_output
tests/test_sessions.py
+361
View File
@@ -0,0 +1,361 @@
"""Tests for session persistence and resume functionality."""
from unittest.mock import MagicMock, patch
import turnstone.core.memory as memory
from turnstone.core.memory import (
register_session,
update_session_title,
set_session_alias,
resolve_session,
list_sessions,
load_session_messages,
delete_session,
save_message,
open_db,
)
from turnstone.core.session import ChatSession
# ── Session registration ──────────────────────────────────────────────
class TestRegisterSession:
def test_register_creates_row(self, tmp_db):
register_session("abc123")
# Session exists in DB (resolve works) even without messages
assert resolve_session("abc123") == "abc123"
def test_register_with_title(self, tmp_db):
register_session("abc123", title="My Session")
save_message("abc123", "user", "hello")
rows = list_sessions()
assert rows[0][2] == "My Session" # title
def test_register_idempotent(self, tmp_db):
register_session("abc123", title="First")
register_session("abc123", title="Second") # should be ignored
save_message("abc123", "user", "hello")
rows = list_sessions()
assert len(rows) == 1
assert rows[0][2] == "First" # original title preserved
def test_update_title(self, tmp_db):
register_session("abc123")
update_session_title("abc123", "New Title")
save_message("abc123", "user", "hello")
rows = list_sessions()
assert rows[0][2] == "New Title"
# ── Session alias ─────────────────────────────────────────────────────
class TestSessionAlias:
def test_set_alias(self, tmp_db):
register_session("abc123")
assert set_session_alias("abc123", "my-session") is True
save_message("abc123", "user", "hello")
rows = list_sessions()
assert rows[0][1] == "my-session" # alias
def test_alias_conflict(self, tmp_db):
register_session("abc123")
register_session("def456")
set_session_alias("abc123", "taken")
assert set_session_alias("def456", "taken") is False
def test_alias_same_session_ok(self, tmp_db):
register_session("abc123")
set_session_alias("abc123", "mine")
assert set_session_alias("abc123", "mine") is True # no-op, same session
# ── Session resolution ────────────────────────────────────────────────
class TestResolveSession:
def test_resolve_by_alias(self, tmp_db):
register_session("abc123")
set_session_alias("abc123", "my-alias")
assert resolve_session("my-alias") == "abc123"
def test_resolve_by_exact_id(self, tmp_db):
register_session("abc123def456")
assert resolve_session("abc123def456") == "abc123def456"
def test_resolve_by_prefix(self, tmp_db):
register_session("abc123def456")
assert resolve_session("abc123") == "abc123def456"
def test_resolve_prefix_ambiguous(self, tmp_db):
register_session("abc123aaaaaa")
register_session("abc123bbbbbb")
# Ambiguous prefix should return None
assert resolve_session("abc123") is None
def test_resolve_not_found(self, tmp_db):
assert resolve_session("nonexistent") is None
def test_resolve_legacy_session(self, tmp_db):
"""Sessions that exist only in conversations (pre-migration) should auto-register."""
save_message("legacy123456", "user", "old message")
result = resolve_session("legacy123456")
assert result == "legacy123456"
# Should now appear in sessions list
rows = list_sessions()
assert any(r[0] == "legacy123456" for r in rows)
# ── List sessions ─────────────────────────────────────────────────────
class TestListSessions:
def test_empty(self, tmp_db):
assert list_sessions() == []
def test_ordered_by_updated(self, tmp_db):
register_session("first")
save_message("first", "user", "hello")
register_session("second")
save_message("second", "user", "hello")
# second is more recent
rows = list_sessions()
assert rows[0][0] == "second"
assert rows[1][0] == "first"
def test_includes_message_count(self, tmp_db):
register_session("sess1")
save_message("sess1", "user", "hello")
save_message("sess1", "assistant", "hi")
rows = list_sessions()
assert rows[0][5] == 2 # msg_count
def test_respects_limit(self, tmp_db):
for i in range(5):
register_session(f"sess{i}")
save_message(f"sess{i}", "user", "hello")
rows = list_sessions(limit=3)
assert len(rows) == 3
# ── Load session messages ─────────────────────────────────────────────
class TestLoadSessionMessages:
def test_simple_user_assistant(self, tmp_db):
save_message("s1", "user", "hello")
save_message("s1", "assistant", "hi there")
msgs = load_session_messages("s1")
assert len(msgs) == 2
assert msgs[0] == {"role": "user", "content": "hello"}
assert msgs[1] == {"role": "assistant", "content": "hi there"}
def test_tool_calls_with_ids(self, tmp_db):
save_message("s1", "user", "run ls")
save_message("s1", "assistant", "Let me check.")
save_message(
"s1", "tool_call", None, "bash", '{"command":"ls"}', tool_call_id="call_abc"
)
save_message(
"s1", "tool_result", "file1.txt\nfile2.txt", "bash", tool_call_id="call_abc"
)
msgs = load_session_messages("s1")
assert len(msgs) == 3 # user, assistant+tool_calls, tool
# Assistant should have content merged with tool_calls
assert msgs[1]["role"] == "assistant"
assert msgs[1]["content"] == "Let me check."
assert len(msgs[1]["tool_calls"]) == 1
assert msgs[1]["tool_calls"][0]["id"] == "call_abc"
assert msgs[1]["tool_calls"][0]["function"]["name"] == "bash"
# Tool result
assert msgs[2]["role"] == "tool"
assert msgs[2]["tool_call_id"] == "call_abc"
assert msgs[2]["content"] == "file1.txt\nfile2.txt"
def test_tool_calls_without_ids_positional(self, tmp_db):
"""Legacy data without tool_call_id uses positional matching."""
save_message("s1", "user", "do stuff")
save_message("s1", "tool_call", None, "bash", '{"command":"ls"}')
save_message("s1", "tool_result", "output", "bash")
msgs = load_session_messages("s1")
assert len(msgs) == 3
# Synthetic IDs should match
tc_id = msgs[1]["tool_calls"][0]["id"]
assert msgs[2]["tool_call_id"] == tc_id
def test_parallel_tool_calls(self, tmp_db):
save_message("s1", "user", "search two things")
save_message(
"s1", "tool_call", None, "search", '{"query":"a"}', tool_call_id="call_1"
)
save_message(
"s1", "tool_call", None, "search", '{"query":"b"}', tool_call_id="call_2"
)
save_message("s1", "tool_result", "result a", "search", tool_call_id="call_1")
save_message("s1", "tool_result", "result b", "search", tool_call_id="call_2")
msgs = load_session_messages("s1")
assert len(msgs) == 4 # user, assistant+2 tool_calls, 2 tool results
assert len(msgs[1]["tool_calls"]) == 2
assert msgs[2]["tool_call_id"] == "call_1"
assert msgs[3]["tool_call_id"] == "call_2"
def test_empty_session(self, tmp_db):
assert load_session_messages("nonexistent") == []
def test_orphaned_tool_result_skipped(self, tmp_db):
save_message("s1", "user", "hello")
save_message("s1", "tool_result", "orphan", "bash")
msgs = load_session_messages("s1")
assert len(msgs) == 1 # only the user message
# ── Delete session ────────────────────────────────────────────────────
class TestDeleteSession:
def test_delete_removes_session_and_messages(self, tmp_db):
register_session("abc123")
save_message("abc123", "user", "hello")
save_message("abc123", "assistant", "hi")
assert delete_session("abc123") is True
assert list_sessions() == []
assert load_session_messages("abc123") == []
def test_delete_nonexistent(self, tmp_db):
assert delete_session("nonexistent") is True # no-op, still returns True
# ── save_message with tool_call_id ────────────────────────────────────
class TestSaveMessageToolCallId:
def test_tool_call_id_stored(self, tmp_db):
save_message(
"s1", "tool_call", None, "bash", '{"cmd":"ls"}', tool_call_id="call_xyz"
)
conn = open_db()
try:
row = conn.execute(
"SELECT tool_call_id FROM conversations WHERE session_id = 's1'"
).fetchone()
assert row[0] == "call_xyz"
finally:
conn.close()
def test_tool_call_id_none_by_default(self, tmp_db):
save_message("s1", "user", "hello")
conn = open_db()
try:
row = conn.execute(
"SELECT tool_call_id FROM conversations WHERE session_id = 's1'"
).fetchone()
assert row[0] is None
finally:
conn.close()
# ── Sessions table creation ───────────────────────────────────────────
class TestSessionsTable:
def test_sessions_table_exists(self, tmp_db):
conn = open_db()
try:
rows = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='sessions'"
).fetchall()
assert len(rows) == 1
finally:
conn.close()
def test_tool_call_id_column_exists(self, tmp_db):
conn = open_db()
try:
# Should not raise
conn.execute("SELECT tool_call_id FROM conversations LIMIT 0")
finally:
conn.close()
# ── ChatSession.resume_session ────────────────────────────────────────
class TestResumeSession:
def test_resume_loads_messages(self, tmp_db, mock_openai_client):
# Set up a session with messages in DB
register_session("old_sess_123")
save_message("old_sess_123", "user", "hello world")
save_message("old_sess_123", "assistant", "hi there")
# Create a new session and resume
session = ChatSession(
client=mock_openai_client,
model="test-model",
ui=MagicMock(),
persona=None,
instructions=None,
temperature=0.5,
max_tokens=1000,
tool_timeout=10,
)
original_id = session._session_id
assert original_id != "old_sess_123"
result = session.resume_session("old_sess_123")
assert result is True
assert session._session_id == "old_sess_123"
assert len(session.messages) == 2
assert session.messages[0]["content"] == "hello world"
assert session._title_generated is True
def test_resume_nonexistent_returns_false(self, tmp_db, mock_openai_client):
session = ChatSession(
client=mock_openai_client,
model="test-model",
ui=MagicMock(),
persona=None,
instructions=None,
temperature=0.5,
max_tokens=1000,
tool_timeout=10,
)
assert session.resume_session("nonexistent") is False
def test_session_registered_on_init(self, tmp_db, mock_openai_client):
session = ChatSession(
client=mock_openai_client,
model="test-model",
ui=MagicMock(),
persona=None,
instructions=None,
temperature=0.5,
max_tokens=1000,
tool_timeout=10,
)
# Session is registered in DB (resolvable) even before any messages
assert resolve_session(session._session_id) == session._session_id
# But does not appear in list_sessions until a message is saved
assert not any(r[0] == session._session_id for r in list_sessions())
# ── save_message updates sessions.updated ─────────────────────────────
class TestSaveMessageUpdatesSession:
def test_updated_timestamp_bumped(self, tmp_db):
register_session("s1")
save_message("s1", "user", "first")
rows = list_sessions()
original_updated = rows[0][4]
import time
time.sleep(0.01) # ensure different timestamp
save_message("s1", "user", "hello")
rows = list_sessions()
new_updated = rows[0][4]
# updated should be same or later (sqlite datetime resolution is seconds,
# so they may be equal in fast tests — just verify no error)
assert new_updated is not None
tests/test_sim.py
+355
View File
@@ -0,0 +1,355 @@
"""Tests for the turnstone cluster simulator."""
from __future__ import annotations
import asyncio
import random
from unittest.mock import MagicMock, call
import pytest
from turnstone.mq.protocol import (
ContentEvent,
InboundMessage,
OutboundEvent,
SendMessage,
StateChangeEvent,
TurnCompleteEvent,
WorkstreamCreatedEvent,
)
from turnstone.sim.config import SimConfig
from turnstone.sim.engine import SimEngine, ToolSimulationError
from turnstone.sim.metrics import MetricsCollector
from turnstone.sim.node import SimNode, SimWorkstream
def _run(coro):
"""Run an async coroutine synchronously."""
return asyncio.run(coro)
# ---------------------------------------------------------------------------
# SimConfig
# ---------------------------------------------------------------------------
class TestSimConfig:
def test_defaults(self):
cfg = SimConfig()
assert cfg.num_nodes == 10
assert cfg.scenario == "steady"
assert cfg.llm_latency_mean == 2.0
assert cfg.tool_failure_rate == 0.02
def test_frozen(self):
cfg = SimConfig()
with pytest.raises(AttributeError):
cfg.num_nodes = 5 # type: ignore[misc]
def test_custom_values(self):
cfg = SimConfig(num_nodes=100, scenario="burst", seed=42)
assert cfg.num_nodes == 100
assert cfg.scenario == "burst"
assert cfg.seed == 42
# ---------------------------------------------------------------------------
# SimEngine
# ---------------------------------------------------------------------------
class TestSimEngine:
@pytest.fixture
def fast_config(self):
return SimConfig(
llm_latency_mean=0.01,
llm_latency_stddev=0.001,
llm_tokens_mean=20,
llm_tokens_stddev=5,
tool_latency_mean=0.01,
tool_latency_stddev=0.001,
tool_failure_rate=0.0,
seed=42,
)
@pytest.fixture
def engine(self, fast_config):
return SimEngine(fast_config)
def test_llm_response_returns_content(self, engine):
async def _test():
content, tool_calls = await engine.simulate_llm_response(True, 1)
assert isinstance(content, str)
assert len(content) > 0
assert isinstance(tool_calls, list)
_run(_test())
def test_llm_response_reproducible_with_seed(self, fast_config):
async def _test():
e1 = SimEngine(fast_config, rng=random.Random(123))
e2 = SimEngine(fast_config, rng=random.Random(123))
c1, t1 = await e1.simulate_llm_response(True, 1)
c2, t2 = await e2.simulate_llm_response(True, 1)
assert c1 == c2
assert len(t1) == len(t2)
_run(_test())
def test_tool_execution_success(self, engine):
async def _test():
result = await engine.simulate_tool_execution("bash")
assert "bash" in result
assert "completed" in result
_run(_test())
def test_tool_execution_failure(self, fast_config):
cfg = SimConfig(
llm_latency_mean=0.01,
tool_latency_mean=0.01,
tool_latency_stddev=0.001,
tool_failure_rate=1.0, # always fail
seed=42,
)
engine = SimEngine(cfg)
async def _test():
with pytest.raises(ToolSimulationError, match="Simulated bash failure"):
await engine.simulate_tool_execution("bash")
_run(_test())
def test_generate_content(self, engine):
content = engine._generate_content(10)
words = content.split()
assert len(words) == 10
# ---------------------------------------------------------------------------
# MetricsCollector
# ---------------------------------------------------------------------------
class TestMetricsCollector:
def test_record_and_summary(self):
m = MetricsCollector()
m.record_inject()
m.record_turn("ws1", "node-0", 1.5)
m.record_turn("ws2", "node-0", 2.5)
m.record_turn("ws3", "node-1", 3.0)
m.record_error("node-0", "test error")
report = m.summary()
assert report["total_turns"] == 3
assert report["total_errors"] == 1
assert report["latency"]["p50"] == 2.5
assert report["latency"]["max"] == 3.0
assert report["turns_per_node"]["node-0"] == 2
assert report["turns_per_node"]["node-1"] == 1
def test_empty_summary(self):
m = MetricsCollector()
report = m.summary()
assert report["total_turns"] == 0
assert report["latency"]["p50"] == 0
def test_node_kill_tracking(self):
m = MetricsCollector()
m.record_node_kill("node-0")
m.record_node_kill("node-1")
report = m.summary()
assert report["node_kills"] == 2
def test_utilization_snapshot(self):
m = MetricsCollector()
m.snapshot_utilization({"node-0": 3, "node-1": 5, "node-2": 0})
report = m.summary()
assert report["utilization"]["mean_ws_per_node"] == pytest.approx(8 / 3)
assert report["utilization"]["max_ws_per_node"] == 5
assert report["utilization"]["nodes_with_zero_ws"] == 1
# ---------------------------------------------------------------------------
# SimNode — message dispatch
# ---------------------------------------------------------------------------
class TestSimNode:
@pytest.fixture
def fast_config(self):
return SimConfig(
llm_latency_mean=0.01,
llm_latency_stddev=0.001,
llm_tokens_mean=10,
llm_tokens_stddev=2,
llm_token_rate=1000,
tool_latency_mean=0.01,
tool_latency_stddev=0.001,
tool_failure_rate=0.0,
max_tool_rounds=0, # no tool calls — fast turn
seed=42,
)
@pytest.fixture
def mock_broker(self):
broker = MagicMock()
broker.list_nodes.return_value = []
return broker
@pytest.fixture
def node(self, fast_config, mock_broker):
metrics = MetricsCollector()
return SimNode("test-node", mock_broker, fast_config, metrics)
def test_handle_send_creates_workstream(self, node, mock_broker):
async def _test():
msg = SendMessage(message="hello", auto_approve=True)
await node.handle_message(msg.to_json())
assert node.workstream_count == 1
mock_broker.set_ws_owner.assert_called_once()
assert mock_broker.publish_outbound.call_count > 0
_run(_test())
def test_handle_send_reuses_existing_ws(self, node, mock_broker):
async def _test():
msg1 = SendMessage(message="hello", auto_approve=True)
await node.handle_message(msg1.to_json())
assert node.workstream_count == 1
ws_id = list(node._workstreams.keys())[0]
msg2 = SendMessage(message="world", ws_id=ws_id, auto_approve=True)
await node.handle_message(msg2.to_json())
assert node.workstream_count == 1
_run(_test())
def test_published_events_are_valid_protocol(self, node, mock_broker):
async def _test():
msg = SendMessage(message="test", auto_approve=True)
await node.handle_message(msg.to_json())
for c in mock_broker.publish_outbound.call_args_list:
_channel, event_json = c[0]
event = OutboundEvent.from_json(event_json)
assert event.type != ""
_run(_test())
def test_state_transitions(self, node, mock_broker):
async def _test():
msg = SendMessage(message="test", auto_approve=True)
await node.handle_message(msg.to_json())
states = []
for c in mock_broker.publish_outbound.call_args_list:
channel, event_json = c[0]
event = OutboundEvent.from_json(event_json)
if isinstance(event, StateChangeEvent):
states.append(event.state)
assert "thinking" in states
assert "idle" in states
assert states.index("thinking") < states.index("idle")
_run(_test())
def test_turn_complete_published(self, node, mock_broker):
async def _test():
msg = SendMessage(message="test", auto_approve=True)
await node.handle_message(msg.to_json())
turn_completes = [
OutboundEvent.from_json(c[0][1])
for c in mock_broker.publish_outbound.call_args_list
if '"turn_complete"' in c[0][1]
]
assert len(turn_completes) >= 1
_run(_test())
def test_close_workstream(self, node, mock_broker):
async def _test():
msg = SendMessage(message="hello", auto_approve=True)
await node.handle_message(msg.to_json())
ws_id = list(node._workstreams.keys())[0]
from turnstone.mq.protocol import CloseWorkstreamMessage
close_msg = CloseWorkstreamMessage(ws_id=ws_id)
await node.handle_message(close_msg.to_json())
assert node.workstream_count == 0
mock_broker.del_ws_owner.assert_called_with(ws_id)
_run(_test())
def test_stop_cleans_up(self, node, mock_broker):
# Add a fake workstream
node._workstreams["fake"] = MagicMock()
mock_broker.set_ws_owner("fake", "test-node")
node.stop()
assert not node._running
assert node.workstream_count == 0
mock_broker.del_ws_owner.assert_called()
def test_heartbeat_once(self, node, mock_broker):
node.heartbeat_once()
mock_broker.register_node.assert_called_once()
args = mock_broker.register_node.call_args
assert args[0][0] == "test-node"
assert args[0][1]["sim"] is True
# ---------------------------------------------------------------------------
# SimWorkstream — state machine
# ---------------------------------------------------------------------------
class TestSimWorkstream:
@pytest.fixture
def fast_config(self):
return SimConfig(
llm_latency_mean=0.01,
llm_latency_stddev=0.001,
llm_tokens_mean=10,
llm_tokens_stddev=2,
llm_token_rate=1000,
tool_latency_mean=0.01,
tool_latency_stddev=0.001,
tool_failure_rate=0.0,
max_tool_rounds=0,
seed=42,
)
def test_turn_ends_in_idle(self, fast_config):
async def _test():
broker = MagicMock()
metrics = MetricsCollector()
node = SimNode("test", broker, fast_config, metrics)
engine = SimEngine(fast_config)
ws = SimWorkstream("ws1", "test-ws", node, engine, fast_config)
await ws.process_turn("hello", "cid-123")
assert ws.state == "idle"
_run(_test())
def test_turn_records_metrics(self, fast_config):
async def _test():
broker = MagicMock()
metrics = MetricsCollector()
node = SimNode("test", broker, fast_config, metrics)
engine = SimEngine(fast_config)
ws = SimWorkstream("ws1", "test-ws", node, engine, fast_config)
await ws.process_turn("hello", "cid-123")
report = metrics.summary()
assert report["total_turns"] == 1
assert report["turns_per_node"]["test"] == 1
_run(_test())
tests/test_tools_schema.py
+121
View File
@@ -0,0 +1,121 @@
"""Tests for turnstone.core.tools — JSON auto-loading and schema validation."""
from turnstone.core.tools import (
TOOLS,
AGENT_TOOLS,
TASK_AGENT_TOOLS,
AGENT_AUTO_TOOLS,
TASK_AUTO_TOOLS,
PRIMARY_KEY_MAP,
_META,
)
class TestToolsSchema:
def test_all_tools_have_function_type(self):
for tool in TOOLS:
assert tool["type"] == "function", f"Tool missing type='function': {tool}"
def test_all_tools_have_name(self):
for tool in TOOLS:
assert "name" in tool["function"], f"Tool missing name: {tool}"
assert isinstance(tool["function"]["name"], str)
def test_all_tools_have_description(self):
for tool in TOOLS:
assert "description" in tool["function"], (
f"Tool missing description: {tool}"
)
assert len(tool["function"]["description"]) > 0
def test_all_tools_have_parameters(self):
for tool in TOOLS:
params = tool["function"]["parameters"]
assert params["type"] == "object"
assert "properties" in params
def test_required_fields_exist_in_properties(self):
for tool in TOOLS:
func = tool["function"]
params = func["parameters"]
required = params.get("required", [])
properties = params["properties"]
for field in required:
assert field in properties, (
f"Tool '{func['name']}': required field '{field}' not in properties"
)
def test_tool_names_unique(self):
names = [t["function"]["name"] for t in TOOLS]
assert len(names) == len(set(names)), f"Duplicate tool names: {names}"
def test_agent_tools_subset(self):
tool_names = {t["function"]["name"] for t in TOOLS}
agent_names = {t["function"]["name"] for t in AGENT_TOOLS}
assert agent_names.issubset(tool_names), (
f"AGENT_TOOLS has names not in TOOLS: {agent_names - tool_names}"
)
def test_task_agent_tools_subset(self):
tool_names = {t["function"]["name"] for t in TOOLS}
task_names = {t["function"]["name"] for t in TASK_AGENT_TOOLS}
assert task_names.issubset(tool_names), (
f"TASK_AGENT_TOOLS has names not in TOOLS: {task_names - tool_names}"
)
def test_agent_tools_not_empty(self):
assert len(AGENT_TOOLS) > 0
def test_task_agent_tools_not_empty(self):
assert len(TASK_AGENT_TOOLS) > 0
class TestToolsMetadata:
"""Validate the metadata extracted from JSON files."""
def test_tool_count(self):
assert len(TOOLS) == 14
def test_agent_tools_count(self):
assert len(AGENT_TOOLS) == 6
def test_task_agent_tools_count(self):
assert len(TASK_AGENT_TOOLS) == 9
def test_auto_approve_sets_match(self):
expected = {"read_file", "search", "math", "man", "web_fetch", "web_search"}
assert AGENT_AUTO_TOOLS == expected
assert TASK_AUTO_TOOLS == expected
def test_primary_key_map(self):
expected = {
"bash": "command",
"math": "code",
"read_file": "path",
"search": "query",
"write_file": "content",
"edit_file": "old_string",
"man": "page",
"web_fetch": "url",
"web_search": "query",
"task": "prompt",
"plan": "prompt",
"remember": "key",
"recall": "query",
"forget": "key",
}
assert PRIMARY_KEY_MAP == expected
def test_no_metadata_in_function_dicts(self):
"""Ensure turnstone metadata keys are stripped from the OpenAI schema."""
meta_keys = {"agent", "task_agent", "auto_approve", "primary_key"}
for tool in TOOLS:
func = tool["function"]
leaked = meta_keys & set(func)
assert not leaked, (
f"Tool '{func['name']}' leaks metadata into function dict: {leaked}"
)
def test_meta_has_all_tools(self):
tool_names = {t["function"]["name"] for t in TOOLS}
assert set(_META.keys()) == tool_names
tests/test_workstream.py
+760
View File
@@ -0,0 +1,760 @@
"""Tests for turnstone.core.workstream — WorkstreamManager, state management, and UI adapters."""
import threading
import time
import pytest
from unittest.mock import MagicMock
from turnstone.core.workstream import WorkstreamManager, WorkstreamState, Workstream
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
class FakeSession:
"""Minimal stand-in for ChatSession in workstream tests."""
def __init__(self):
self.model = "test-model"
self.messages = []
def _fake_factory(ui):
return FakeSession()
class FakeUI:
"""Minimal SessionUI that tracks state changes."""
def __init__(self, ws_id=""):
self.ws_id = ws_id
self.state_changes = []
self.auto_approve = False
def on_state_change(self, state):
self.state_changes.append(state)
# Stubs for the rest of the protocol
def on_thinking_start(self):
pass
def on_thinking_stop(self):
pass
def on_reasoning_token(self, text):
pass
def on_content_token(self, text):
pass
def on_stream_end(self):
pass
def approve_tools(self, items):
return True, None
def on_tool_result(self, name, output):
pass
def on_status(self, usage, context_window, effort):
pass
def on_plan_review(self, content):
return ""
def on_info(self, message):
pass
def on_error(self, message):
pass
# ---------------------------------------------------------------------------
# WorkstreamState enum
# ---------------------------------------------------------------------------
class TestWorkstreamState:
def test_values(self):
assert WorkstreamState.IDLE.value == "idle"
assert WorkstreamState.THINKING.value == "thinking"
assert WorkstreamState.RUNNING.value == "running"
assert WorkstreamState.ATTENTION.value == "attention"
assert WorkstreamState.ERROR.value == "error"
def test_from_string(self):
assert WorkstreamState("idle") == WorkstreamState.IDLE
assert WorkstreamState("attention") == WorkstreamState.ATTENTION
# ---------------------------------------------------------------------------
# Workstream dataclass
# ---------------------------------------------------------------------------
class TestWorkstream:
def test_default_name(self):
ws = Workstream()
assert ws.name.startswith("ws-")
assert len(ws.name) == 7 # "ws-" + 4 hex chars
def test_custom_name(self):
ws = Workstream(name="my-stream")
assert ws.name == "my-stream"
def test_default_state(self):
ws = Workstream()
assert ws.state == WorkstreamState.IDLE
def test_id_uniqueness(self):
ws1 = Workstream()
ws2 = Workstream()
assert ws1.id != ws2.id
# ---------------------------------------------------------------------------
# WorkstreamManager — creation and lookup
# ---------------------------------------------------------------------------
class TestManagerCreation:
def test_create_first_sets_active(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.active_id == ws.id
assert mgr.get_active() is ws
def test_create_second_does_not_change_active(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.active_id == ws1.id
def test_create_assigns_session(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert isinstance(ws.session, FakeSession)
def test_create_assigns_ui(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert isinstance(ws.ui, FakeUI)
assert ws.ui.ws_id == ws.id
def test_create_custom_name(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(name="research", ui_factory=lambda wid: FakeUI(wid))
assert ws.name == "research"
def test_create_default_name(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert ws.name.startswith("ws-")
def test_create_max_workstreams(self):
mgr = WorkstreamManager(_fake_factory)
mgr.MAX_WORKSTREAMS = 3
mgr.create(ui_factory=lambda wid: FakeUI(wid))
mgr.create(ui_factory=lambda wid: FakeUI(wid))
mgr.create(ui_factory=lambda wid: FakeUI(wid))
with pytest.raises(RuntimeError, match="Maximum"):
mgr.create(ui_factory=lambda wid: FakeUI(wid))
class TestManagerLookup:
def test_get_existing(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.get(ws.id) is ws
def test_get_nonexistent(self):
mgr = WorkstreamManager(_fake_factory)
assert mgr.get("no-such-id") is None
def test_list_all_creation_order(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(name="a", ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(name="b", ui_factory=lambda wid: FakeUI(wid))
ws3 = mgr.create(name="c", ui_factory=lambda wid: FakeUI(wid))
result = mgr.list_all()
assert [w.name for w in result] == ["a", "b", "c"]
def test_index_of(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.index_of(ws1.id) == 1
assert mgr.index_of(ws2.id) == 2
assert mgr.index_of("nonexistent") == 0
def test_count(self):
mgr = WorkstreamManager(_fake_factory)
assert mgr.count == 0
mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.count == 1
mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.count == 2
# ---------------------------------------------------------------------------
# WorkstreamManager — switching
# ---------------------------------------------------------------------------
class TestManagerSwitching:
def test_switch_by_id(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.active_id == ws1.id
result = mgr.switch(ws2.id)
assert result is ws2
assert mgr.active_id == ws2.id
def test_switch_nonexistent_returns_none(self):
mgr = WorkstreamManager(_fake_factory)
mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.switch("bad-id") is None
def test_switch_by_index(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
result = mgr.switch_by_index(2)
assert result is ws2
assert mgr.active_id == ws2.id
def test_switch_by_index_out_of_range(self):
mgr = WorkstreamManager(_fake_factory)
mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.switch_by_index(0) is None
assert mgr.switch_by_index(5) is None
# ---------------------------------------------------------------------------
# WorkstreamManager — closing
# ---------------------------------------------------------------------------
class TestManagerClose:
def test_close_removes_workstream(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.close(ws2.id) is True
assert mgr.count == 1
assert mgr.get(ws2.id) is None
def test_close_last_returns_false(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.close(ws.id) is False
assert mgr.count == 1
def test_close_nonexistent_returns_false(self):
mgr = WorkstreamManager(_fake_factory)
mgr.create(ui_factory=lambda wid: FakeUI(wid))
mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert mgr.close("nonexistent") is False
def test_close_active_switches_to_first(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
mgr.switch(ws2.id)
mgr.close(ws2.id)
assert mgr.active_id == ws1.id
def test_close_updates_order(self):
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(name="a", ui_factory=lambda wid: FakeUI(wid))
ws2 = mgr.create(name="b", ui_factory=lambda wid: FakeUI(wid))
ws3 = mgr.create(name="c", ui_factory=lambda wid: FakeUI(wid))
mgr.close(ws2.id)
names = [w.name for w in mgr.list_all()]
assert names == ["a", "c"]
def test_close_unblocks_approval_event(self):
"""Closing a workstream whose UI has a pending approval should unblock it."""
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
# Create a workstream with a WebUI-like approval mechanism
from turnstone.server import WebUI
ws2 = mgr.create(ui_factory=lambda wid: WebUI(ws_id=wid))
ws2.ui._approval_event.clear() # simulate pending approval
mgr.close(ws2.id)
# The approval event should be set (unblocked)
assert ws2.ui._approval_event.is_set()
def test_close_unblocks_plan_event(self):
"""Closing a workstream with pending plan review should unblock it."""
mgr = WorkstreamManager(_fake_factory)
ws1 = mgr.create(ui_factory=lambda wid: FakeUI(wid))
from turnstone.server import WebUI
ws2 = mgr.create(ui_factory=lambda wid: WebUI(ws_id=wid))
ws2.ui._plan_event.clear()
mgr.close(ws2.id)
assert ws2.ui._plan_event.is_set()
assert ws2.ui._plan_result == "reject"
# ---------------------------------------------------------------------------
# WorkstreamManager — state management
# ---------------------------------------------------------------------------
class TestManagerState:
def test_set_state(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
assert ws.state == WorkstreamState.IDLE
mgr.set_state(ws.id, WorkstreamState.THINKING)
assert ws.state == WorkstreamState.THINKING
def test_set_state_with_error(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
mgr.set_state(ws.id, WorkstreamState.ERROR, error_msg="API timeout")
assert ws.state == WorkstreamState.ERROR
assert ws.error_message == "API timeout"
def test_set_state_nonexistent_is_noop(self):
mgr = WorkstreamManager(_fake_factory)
mgr.set_state("no-such-id", WorkstreamState.THINKING) # should not raise
def test_on_state_change_callback(self):
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
changes = []
mgr._on_state_change = lambda wid, state: changes.append((wid, state))
mgr.set_state(ws.id, WorkstreamState.RUNNING)
assert changes == [(ws.id, WorkstreamState.RUNNING)]
# ---------------------------------------------------------------------------
# WorkstreamManager — thread safety
# ---------------------------------------------------------------------------
class TestManagerThreadSafety:
def test_concurrent_create_respects_max(self):
"""Multiple threads creating workstreams should not exceed MAX."""
mgr = WorkstreamManager(_fake_factory)
mgr.MAX_WORKSTREAMS = 5
errors = []
created = []
def do_create():
try:
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
created.append(ws.id)
except RuntimeError:
errors.append(True)
threads = [threading.Thread(target=do_create) for _ in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
assert mgr.count == 5
assert len(errors) == 5
def test_concurrent_switch(self):
"""Concurrent switches should not corrupt state."""
mgr = WorkstreamManager(_fake_factory)
ids = []
for _ in range(5):
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
ids.append(ws.id)
def do_switch(wid):
for _ in range(20):
mgr.switch(wid)
threads = [threading.Thread(target=do_switch, args=(wid,)) for wid in ids]
for t in threads:
t.start()
for t in threads:
t.join()
# active_id should be one of the valid ids
assert mgr.active_id in ids
def test_concurrent_close_and_list(self):
"""close() and list_all() running concurrently should not crash."""
mgr = WorkstreamManager(_fake_factory)
# Keep one alive to prevent closing the last
anchor = mgr.create(ui_factory=lambda wid: FakeUI(wid))
targets = []
for _ in range(5):
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
targets.append(ws.id)
def do_close():
for wid in targets:
mgr.close(wid)
def do_list():
for _ in range(50):
mgr.list_all()
t1 = threading.Thread(target=do_close)
t2 = threading.Thread(target=do_list)
t1.start()
t2.start()
t1.join()
t2.join()
assert mgr.count == 1
assert mgr.get(anchor.id) is not None
# ---------------------------------------------------------------------------
# WorkstreamTerminalUI
# ---------------------------------------------------------------------------
class TestWorkstreamTerminalUI:
def test_foreground_detection(self):
from turnstone.cli import WorkstreamTerminalUI
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
assert ws.ui.is_foreground is True
ws2 = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
# ws1 is still active, so ws2 is not foreground
assert ws2.ui.is_foreground is False
def test_state_change_updates_manager(self):
from turnstone.cli import WorkstreamTerminalUI
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
ws.ui.on_state_change("thinking")
assert ws.state == WorkstreamState.THINKING
ws.ui.on_state_change("idle")
assert ws.state == WorkstreamState.IDLE
def test_invalid_state_change_ignored(self):
from turnstone.cli import WorkstreamTerminalUI
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
ws.ui.on_state_change("not_a_real_state") # should not raise
assert ws.state == WorkstreamState.IDLE # unchanged
def _make_background_ws(self):
"""Create a manager with two workstreams; switch to the second so the first is background."""
from turnstone.cli import WorkstreamTerminalUI
mgr = WorkstreamManager(_fake_factory)
bg = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
fg = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
mgr.switch(fg.id)
bg.ui.set_foreground(False)
fg.ui.set_foreground(True)
return mgr, bg, fg
def test_background_buffers_content(self):
mgr, bg, fg = self._make_background_ws()
assert bg.ui.is_foreground is False
bg.ui.on_content_token("hello ")
bg.ui.on_content_token("world")
bg.ui.on_stream_end()
assert len(bg.ui._output_buffer) == 3
assert bg.ui._output_buffer[0] == ("content", "hello ")
assert bg.ui._output_buffer[1] == ("content", "world")
assert bg.ui._output_buffer[2] == ("stream_end", "")
def test_flush_buffer_clears(self):
mgr, bg, fg = self._make_background_ws()
bg.ui.on_content_token("test")
bg.ui.on_stream_end()
assert len(bg.ui._output_buffer) == 2
mgr.switch(bg.id)
bg.ui.set_foreground(True)
bg.ui.flush_buffer()
assert len(bg.ui._output_buffer) == 0
def test_background_buffers_info_and_error(self):
mgr, bg, fg = self._make_background_ws()
bg.ui.on_info("info msg")
bg.ui.on_error("error msg")
assert ("info", "info msg") in bg.ui._output_buffer
assert ("error", "error msg") in bg.ui._output_buffer
def test_fg_event_blocks_approval_in_background(self):
"""approve_tools should block until foregrounded."""
mgr, bg, fg = self._make_background_ws()
bg.ui.auto_approve = True # so we don't need actual input()
result = [None]
def call_approve():
result[0] = bg.ui.approve_tools(
[{"needs_approval": True, "header": "test", "func_name": "bash"}]
)
t = threading.Thread(target=call_approve)
t.start()
time.sleep(0.1)
assert t.is_alive(), "approve_tools should be blocking"
# Bring to foreground — should unblock
mgr.switch(bg.id)
bg.ui.set_foreground(True)
t.join(timeout=2)
assert not t.is_alive()
assert result[0] == (True, None) # auto-approved
# ---------------------------------------------------------------------------
# WebUI workstream support
# ---------------------------------------------------------------------------
class TestWebUI:
def test_ws_id_assigned(self):
from turnstone.server import WebUI
ui = WebUI(ws_id="test-123")
assert ui.ws_id == "test-123"
def test_on_state_change_broadcasts(self):
"""on_state_change should put an event on the global queue."""
import queue
from turnstone.server import WebUI
gq = queue.Queue()
old = WebUI._global_queue
WebUI._global_queue = gq
try:
ui = WebUI(ws_id="abc")
ui.on_state_change("thinking")
event = gq.get_nowait()
assert event["type"] == "ws_state"
assert event["ws_id"] == "abc"
assert event["state"] == "thinking"
finally:
WebUI._global_queue = old
def test_on_state_change_no_global_queue(self):
"""on_state_change should not crash if no global queue is set."""
from turnstone.server import WebUI
old = WebUI._global_queue
WebUI._global_queue = None
try:
ui = WebUI(ws_id="xyz")
ui.on_state_change("running") # should not raise
finally:
WebUI._global_queue = old
def test_resolve_approval(self):
from turnstone.server import WebUI
ui = WebUI(ws_id="test")
ui._approval_event.clear()
# Resolve in a thread
def resolve():
time.sleep(0.05)
ui.resolve_approval(True, "looks good")
t = threading.Thread(target=resolve)
t.start()
ui._approval_event.wait(timeout=2)
assert ui._approval_result == (True, "looks good")
t.join()
def test_resolve_plan(self):
from turnstone.server import WebUI
ui = WebUI(ws_id="test")
ui._plan_event.clear()
def resolve():
time.sleep(0.05)
ui.resolve_plan("approved")
t = threading.Thread(target=resolve)
t.start()
ui._plan_event.wait(timeout=2)
assert ui._plan_result == "approved"
t.join()
# ---------------------------------------------------------------------------
# Integration: WorkstreamManager + session state transitions
# ---------------------------------------------------------------------------
class TestStateTransitions:
def test_full_lifecycle(self):
"""Verify the expected state transition sequence."""
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
# Simulate the state transitions that ChatSession.send() would emit
mgr.set_state(ws.id, WorkstreamState.THINKING)
assert ws.state == WorkstreamState.THINKING
mgr.set_state(ws.id, WorkstreamState.RUNNING)
assert ws.state == WorkstreamState.RUNNING
mgr.set_state(ws.id, WorkstreamState.ATTENTION)
assert ws.state == WorkstreamState.ATTENTION
mgr.set_state(ws.id, WorkstreamState.RUNNING)
assert ws.state == WorkstreamState.RUNNING
mgr.set_state(ws.id, WorkstreamState.IDLE)
assert ws.state == WorkstreamState.IDLE
def test_error_recovery(self):
"""After an error, sending again should transition back to thinking."""
mgr = WorkstreamManager(_fake_factory)
ws = mgr.create(ui_factory=lambda wid: FakeUI(wid))
mgr.set_state(ws.id, WorkstreamState.ERROR, "API failed")
assert ws.state == WorkstreamState.ERROR
assert ws.error_message == "API failed"
mgr.set_state(ws.id, WorkstreamState.THINKING)
assert ws.state == WorkstreamState.THINKING
assert ws.error_message == ""
# ---------------------------------------------------------------------------
# Design polish: thread-safe buffer, approval context, NO_COLOR
# ---------------------------------------------------------------------------
class TestBufferThreadSafety:
"""Verify that _buffer() uses the lock and flush_buffer copies under lock."""
def test_concurrent_buffer_and_flush(self):
"""Simultaneous buffering and flushing should not lose or corrupt events."""
from turnstone.cli import WorkstreamTerminalUI
mgr = WorkstreamManager(_fake_factory)
bg = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
fg = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
mgr.switch(fg.id)
bg.ui.set_foreground(False)
n_events = 200
done = threading.Event()
def do_buffer():
for i in range(n_events):
bg.ui._buffer("content", f"token-{i}")
done.set()
t = threading.Thread(target=do_buffer)
t.start()
done.wait()
# All events should be in the buffer
with bg.ui._print_lock:
count = len(bg.ui._output_buffer)
assert count == n_events
t.join()
class TestApprovalContextMessage:
"""Verify approval in background buffers a context message."""
def test_approval_buffers_tool_names(self):
from turnstone.cli import WorkstreamTerminalUI
mgr = WorkstreamManager(_fake_factory)
bg = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
fg = mgr.create(ui_factory=lambda wid: WorkstreamTerminalUI(wid, mgr))
mgr.switch(fg.id)
bg.ui.set_foreground(False)
bg.ui.auto_approve = True
result = [None]
def call_approve():
result[0] = bg.ui.approve_tools(
[
{
"needs_approval": True,
"header": "test",
"func_name": "bash",
"approval_label": "bash: ls",
},
]
)
t = threading.Thread(target=call_approve)
t.start()
time.sleep(0.1)
# Should have a waiting-for-approval message in the buffer
with bg.ui._print_lock:
info_msgs = [text for ev, text in bg.ui._output_buffer if ev == "info"]
assert any("bash: ls" in msg for msg in info_msgs)
# Unblock
mgr.switch(bg.id)
bg.ui.set_foreground(True)
t.join(timeout=2)
assert result[0] == (True, None)
class TestNoColor:
"""Verify NO_COLOR support in colors module."""
def test_no_color_env_disables_ansi(self):
import importlib
import os
import turnstone.ui.colors as colors_mod
old_env = os.environ.get("NO_COLOR")
try:
os.environ["NO_COLOR"] = "1"
importlib.reload(colors_mod)
assert colors_mod.RESET == ""
assert colors_mod.BOLD == ""
assert colors_mod.RED == ""
assert colors_mod.red("test") == "test"
finally:
if old_env is None:
os.environ.pop("NO_COLOR", None)
else:
os.environ["NO_COLOR"] = old_env
importlib.reload(colors_mod)
turnstone/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""turnstone - Single-file AI chat client with tool use."""
__version__ = "0.1.0"
turnstone/chat.py
Executable
+51
View File
@@ -0,0 +1,51 @@
"""chat.py — Backward-compatibility shim.
All functionality has been moved to submodules:
- turnstone.core.session: ChatSession, SessionUI
- turnstone.core.tools: TOOLS, AGENT_TOOLS, TASK_AGENT_TOOLS
- turnstone.core.edit: find_occurrences, pick_nearest
- turnstone.core.sandbox: validate_math_code, execute_math_sandboxed
- turnstone.core.safety: is_command_blocked, sanitize_command
- turnstone.core.web: strip_html, check_ssrf
- turnstone.core.memory: open_db, load_memories, save_message, etc.
- turnstone.ui.colors: ANSI constants and helpers
- turnstone.ui.markdown: MarkdownRenderer
- turnstone.ui.spinner: Spinner
- turnstone.cli: TerminalUI, main, detect_model
"""
# Re-export public API for backward compatibility
from turnstone.core.session import ChatSession, SessionUI # noqa: F401
from turnstone.core.tools import TOOLS, AGENT_TOOLS, TASK_AGENT_TOOLS # noqa: F401
from turnstone.core.edit import (
find_occurrences as _find_occurrences,
pick_nearest as _pick_nearest,
) # noqa: F401
from turnstone.core.sandbox import (
validate_math_code as _validate_math_code,
auto_print_wrap as _auto_print_wrap,
execute_math_sandboxed as _execute_math_sandboxed,
) # noqa: F401
from turnstone.core.safety import (
is_command_blocked,
sanitize_command as _sanitize_command,
BLOCKED_PATTERNS,
) # noqa: F401
from turnstone.core.web import strip_html as _strip_html # noqa: F401
from turnstone.core.memory import ( # noqa: F401
open_db as _open_db,
load_memories as _load_memories,
save_message as _save_message,
normalize_key as _normalize_key,
search_history as _search_history,
search_history_recent as _search_history_recent,
escape_like as _escape_like,
fts5_query as _fts5_query,
get_tavily_key as _get_tavily_key,
db_override as _db_override,
db_initialized as _db_initialized,
)
from turnstone.ui.colors import * # noqa: F401, F403
from turnstone.ui.markdown import MarkdownRenderer # noqa: F401
from turnstone.ui.spinner import Spinner # noqa: F401
from turnstone.cli import main, detect_model # noqa: F401
turnstone/cli.py
+987
View File
@@ -0,0 +1,987 @@
"""Terminal CLI frontend for turnstone.
Provides TerminalUI (implementing the SessionUI protocol), readline setup,
model auto-detection, workstream management, and the main() REPL entry point.
"""
import argparse
import os
import readline
import sys
import textwrap
import threading
from openai import OpenAI
from turnstone.core.session import ChatSession, SessionUI
from turnstone.core.tools import TOOLS
from turnstone.core.workstream import WorkstreamManager, WorkstreamState
from turnstone.ui.colors import (
BOLD,
CYAN,
DIM,
GRAY,
GREEN,
RED,
RESET,
YELLOW,
bold,
cyan,
dim,
green,
red,
yellow,
)
from turnstone.ui.markdown import MarkdownRenderer
from turnstone.ui.spinner import Spinner
# ─── Readline ─────────────────────────────────────────────────────────────
SLASH_COMMANDS = [
"/persona",
"/instructions",
"/clear",
"/new",
"/sessions",
"/resume",
"/name",
"/delete",
"/history",
"/model",
"/raw",
"/reason",
"/compact",
"/creative",
"/debug",
"/help",
"/exit",
"/quit",
"/q",
"/ws",
"/cluster",
]
def _completer(text, state):
"""Tab-complete slash commands."""
if text.startswith("/"):
matches = [c for c in SLASH_COMMANDS if c.startswith(text)]
else:
matches = []
if state < len(matches):
return matches[state] + " "
return None
def setup_readline():
"""Set up readline with tab completion."""
readline.set_history_length(1000)
readline.set_completer(_completer)
readline.set_completer_delims("") # treat entire line as completion input
readline.parse_and_bind("tab: complete")
# ─── TerminalUI ───────────────────────────────────────────────────────────
class TerminalUI(SessionUI):
"""Terminal-based UI using ANSI colors, MarkdownRenderer, and Spinner."""
def __init__(self):
self.md = MarkdownRenderer()
self.spinner = None
self._print_lock = threading.Lock()
self.auto_approve = False
def on_thinking_start(self):
self.spinner = Spinner("Thinking")
self.spinner.start()
def on_thinking_stop(self):
if self.spinner:
self.spinner.stop()
self.spinner = None
def on_reasoning_token(self, text):
sys.stdout.write(f"{DIM}{text}{RESET}")
sys.stdout.flush()
def on_content_token(self, text):
rendered = self.md.feed(text)
if rendered:
sys.stdout.write(rendered)
sys.stdout.flush()
def on_stream_end(self):
remainder = self.md.flush()
if remainder:
sys.stdout.write(remainder)
self.md.in_code_block = False
sys.stdout.write("\n")
sys.stdout.flush()
def approve_tools(self, items):
"""Display tool previews and prompt for batch approval.
Returns (approved: bool, feedback: str | None).
"""
pending = [
it for it in items if it.get("needs_approval") and not it.get("error")
]
with self._print_lock:
# Print all headers and previews
for item in items:
if item.get("error"):
sys.stdout.write(f" {red(item['header'])}\n")
else:
sys.stdout.write(f" {yellow(item['header'])}\n")
if item.get("preview"):
sys.stdout.write(item["preview"] + "\n")
sys.stdout.flush()
if not pending or self.auto_approve:
return True, None
# Prompt
try:
if len(pending) == 1:
label = pending[0].get("approval_label", pending[0]["func_name"])
prompt_text = (
f" \001{BOLD}\002Allow {label}?\001{RESET}\002 "
f"\001{DIM}\002[y/n/a(lways), optional message]\001{RESET}\002 "
)
else:
labels = ", ".join(
it.get("approval_label", it["func_name"]) for it in pending
)
prompt_text = (
f" \001{BOLD}\002Allow {len(pending)} tools ({labels})?\001{RESET}\002 "
f"\001{DIM}\002[y/n/a(lways), optional message]\001{RESET}\002 "
)
resp = input(prompt_text).strip()
except (EOFError, KeyboardInterrupt):
sys.stdout.write("\n")
resp = "n"
# Parse decision and optional feedback: "y, use absolute path"
decision = resp.lower()
feedback = None
for sep in (",", " "):
if sep in resp:
decision = resp[: resp.index(sep)].strip().lower()
feedback = resp[resp.index(sep) + 1 :].strip() or None
break
if decision in ("a", "always"):
self.auto_approve = True
return True, feedback
elif decision in ("y", "yes"):
return True, feedback
else:
denial_msg = "Denied by user"
if feedback:
denial_msg += f": {feedback}"
for item in pending:
item["denied"] = True
item["denial_msg"] = denial_msg
return False, None
def on_tool_result(self, name, output):
pass # Optional: display summary
def on_status(self, usage, context_window, effort):
total_tok = usage["prompt_tokens"] + usage["completion_tokens"]
pct = total_tok / context_window * 100 if context_window > 0 else 0
parts = [f"{total_tok:,} / {context_window:,} tokens ({pct:.0f}%)"]
if effort != "medium":
parts.append(f"reasoning: {effort}")
sys.stdout.write(f"\n {DIM}[{' · '.join(parts)}]{RESET}\n")
sys.stdout.flush()
def on_plan_review(self, content):
sys.stdout.write(f"\n{DIM}{'─' * 60}{RESET}\n")
for line in content.splitlines():
sys.stdout.write(f" {line}\n")
sys.stdout.write(f"{DIM}{'─' * 60}{RESET}\n")
sys.stdout.flush()
try:
prompt_text = (
f" \001{BOLD}\002Plan ready.\001{RESET}\002 "
f"\001{DIM}\002[enter to approve, or give feedback]\001{RESET}\002 "
)
resp = input(prompt_text).strip()
except EOFError:
resp = ""
except KeyboardInterrupt:
resp = "reject"
return resp
def on_info(self, message):
print(message)
def on_error(self, message):
sys.stdout.write(f"{RED}{message}{RESET}\n")
sys.stdout.flush()
def on_state_change(self, state):
pass # base TerminalUI ignores state changes
def on_rename(self, name: str):
pass # base TerminalUI ignores renames
# ─── WorkstreamTerminalUI ─────────────────────────────────────────────────
# State display config: (symbol, color_fn, label)
_STATE_DISPLAY = {
WorkstreamState.IDLE: ("·", dim, "idle"),
WorkstreamState.THINKING: ("◌", cyan, "thinking"),
WorkstreamState.RUNNING: ("▸", green, "running"),
WorkstreamState.ATTENTION: ("◆", yellow, "attention"),
WorkstreamState.ERROR: ("✖", red, "error"),
}
class WorkstreamTerminalUI(TerminalUI):
"""TerminalUI with workstream awareness: buffers output when in background,
blocks on approval until foregrounded."""
def __init__(self, ws_id: str, manager: WorkstreamManager):
super().__init__()
self.ws_id = ws_id
self.manager = manager
self._output_buffer: list[tuple[str, str]] = [] # (event_type, text)
self._fg_event = threading.Event()
self._fg_event.set() # starts as foreground
@property
def is_foreground(self) -> bool:
return self.manager.active_id == self.ws_id
def set_foreground(self, fg: bool):
if fg:
self._fg_event.set()
else:
self._fg_event.clear()
def on_state_change(self, state: str):
try:
ws_state = WorkstreamState(state)
except ValueError:
return
self.manager.set_state(self.ws_id, ws_state)
# -- output buffering when in background --------------------------------
def on_thinking_start(self):
if self.is_foreground:
super().on_thinking_start()
def on_thinking_stop(self):
if self.is_foreground:
super().on_thinking_stop()
elif self.spinner:
self.spinner.stop()
self.spinner = None
def _buffer(self, event_type: str, text: str):
with self._print_lock:
self._output_buffer.append((event_type, text))
def on_reasoning_token(self, text):
if self.is_foreground:
super().on_reasoning_token(text)
else:
self._buffer("reasoning", text)
def on_content_token(self, text):
if self.is_foreground:
super().on_content_token(text)
else:
self._buffer("content", text)
def on_stream_end(self):
if self.is_foreground:
super().on_stream_end()
else:
self._buffer("stream_end", "")
def on_status(self, usage, context_window, effort):
if self.is_foreground:
super().on_status(usage, context_window, effort)
# silently drop status for background streams
def on_info(self, message):
if self.is_foreground:
super().on_info(message)
else:
self._buffer("info", message)
def on_error(self, message):
if self.is_foreground:
super().on_error(message)
else:
self._buffer("error", message)
def on_tool_result(self, name, output):
if self.is_foreground:
super().on_tool_result(name, output)
def on_plan_review(self, content):
# Must wait until foregrounded to show plan review
if not self.is_foreground:
self._buffer(
"info",
f"{YELLOW}[Plan ready — switch to this workstream to review]{RESET}",
)
self._fg_event.wait()
return super().on_plan_review(content)
def approve_tools(self, items):
"""Block until foregrounded if in background, then show approval prompt."""
if not self.is_foreground:
tool_names = ", ".join(
it.get("approval_label", it.get("func_name", "?"))
for it in items
if it.get("needs_approval") and not it.get("error")
)
if tool_names:
self._buffer(
"info", f"{YELLOW}Waiting for approval: {tool_names}{RESET}"
)
self._fg_event.wait()
return super().approve_tools(items)
def flush_buffer(self):
"""Replay buffered output when switching to foreground."""
with self._print_lock:
if not self._output_buffer:
return
buf = list(self._output_buffer)
self._output_buffer.clear()
sys.stdout.write(
f"\n {DIM}--- buffered output ({len(buf)} events) ---{RESET}\n"
)
replay_md = MarkdownRenderer()
for event_type, text in buf:
if event_type == "reasoning":
sys.stdout.write(f"{DIM}{text}{RESET}")
elif event_type == "content":
rendered = replay_md.feed(text)
if rendered:
sys.stdout.write(rendered)
elif event_type == "stream_end":
remainder = replay_md.flush()
if remainder:
sys.stdout.write(remainder)
replay_md.in_code_block = False
sys.stdout.write("\n")
elif event_type == "info":
sys.stdout.write(f"{text}\n")
elif event_type == "error":
sys.stdout.write(f"{RED}{text}{RESET}\n")
sys.stdout.write(f" {DIM}--- end buffered output ---{RESET}\n\n")
sys.stdout.flush()
# ─── Workstream commands ──────────────────────────────────────────────────
def _print_ws_status_line(manager: WorkstreamManager):
"""Print a one-line status of background workstreams that are active."""
active_id = manager.active_id
parts = []
for ws in manager.list_all():
if ws.id == active_id:
continue
if ws.state in (WorkstreamState.IDLE,):
continue
sym, color_fn, label = _STATE_DISPLAY[ws.state]
idx = manager.index_of(ws.id)
parts.append(color_fn(f"{sym} {idx}:{ws.name} ({label})"))
if parts:
sys.stderr.write(f" {' '.join(parts)}\n")
sys.stderr.flush()
def _handle_ws_command(
manager: WorkstreamManager,
cmd_line: str,
skip_permissions: bool,
):
"""Handle /ws subcommands. Returns (switched: bool)."""
parts = cmd_line.strip().split()
sub = parts[1] if len(parts) > 1 else "list"
if sub == "list":
active_id = manager.active_id
all_ws = manager.list_all()
max_name = max((len(ws.name) for ws in all_ws), default=0)
for ws in all_ws:
idx = manager.index_of(ws.id)
sym, color_fn, label = _STATE_DISPLAY[ws.state]
marker = " *" if ws.id == active_id else " "
padded = ws.name.ljust(max_name)
print(f" {marker}{idx}. {color_fn(f'{sym} {padded}')} {dim(label)}")
return False
elif sub == "new":
name = parts[2] if len(parts) > 2 else ""
try:
ws = manager.create(
name=name,
ui_factory=lambda wid: WorkstreamTerminalUI(wid, manager),
)
except RuntimeError as e:
print(red(str(e)))
return False
if skip_permissions:
ws.ui.auto_approve = True
# Mark old active as background
old = manager.get_active()
if old and old.ui and hasattr(old.ui, "set_foreground"):
old.ui.set_foreground(False)
manager.switch(ws.id)
ws.ui.set_foreground(True)
print(f"Created workstream {cyan(ws.name)} (#{manager.index_of(ws.id)})")
return True
elif sub.isdigit():
idx = int(sub)
old = manager.get_active()
ws = manager.switch_by_index(idx)
if ws:
if old and old.ui and hasattr(old.ui, "set_foreground"):
old.ui.set_foreground(False)
if hasattr(ws.ui, "set_foreground"):
ws.ui.set_foreground(True)
if hasattr(ws.ui, "flush_buffer"):
ws.ui.flush_buffer()
print(f"Switched to {cyan(ws.name)}")
return True
else:
print(red(f"No workstream #{idx}"))
return False
elif sub == "close":
target_idx = int(parts[2]) if len(parts) > 2 and parts[2].isdigit() else None
if target_idx is not None:
all_ws = manager.list_all()
if 1 <= target_idx <= len(all_ws):
ws_id = all_ws[target_idx - 1].id
else:
print(red(f"No workstream #{target_idx}"))
return False
else:
ws_id = manager.active_id
ws_name = manager.get(ws_id).name if manager.get(ws_id) else "?"
if manager.close(ws_id):
print(f"Closed workstream {ws_name}")
# Ensure new active is foregrounded
new_active = manager.get_active()
if new_active and hasattr(new_active.ui, "set_foreground"):
new_active.ui.set_foreground(True)
return True
else:
print(red("Cannot close the last workstream"))
return False
elif sub == "rename":
new_name = " ".join(parts[2:]) if len(parts) > 2 else ""
if not new_name:
print(red("Usage: /ws rename <name>"))
return False
ws = manager.get_active()
if ws:
old_name = ws.name
ws.name = new_name
print(f"Renamed {old_name} -> {cyan(new_name)}")
return False
else:
print(f"Unknown /ws subcommand: {sub}")
print(f"Usage: /ws [list|new [name]|<N>|close [N]|rename <name>]")
return False
# ─── Cluster commands ─────────────────────────────────────────────────────
def _handle_cluster_command(
cmd_line: str, console_url: str | None, auth_token: str = ""
):
"""Handle /cluster subcommands querying the turnstone-console API."""
import httpx
if not console_url:
print(
red(
"No console URL configured. Use --console-url or set [console] url in config."
)
)
return
headers: dict[str, str] = {}
if auth_token:
headers["Authorization"] = f"Bearer {auth_token}"
parts = cmd_line.strip().split()
sub = parts[1] if len(parts) > 1 else "status"
try:
if sub == "status":
resp = httpx.get(
f"{console_url}/api/cluster/overview", timeout=5, headers=headers
)
data = resp.json()
states = data.get("states", {})
agg = data.get("aggregate", {})
print(f"\n {bold('Cluster Overview')}")
print(
f" Nodes: {cyan(str(data.get('nodes', 0)))} "
f"Workstreams: {cyan(str(data.get('workstreams', 0)))}"
)
print()
for state_name in ["running", "thinking", "attention", "idle", "error"]:
count = states.get(state_name, 0)
sym, color_fn, label = _STATE_DISPLAY.get(
{
"running": WorkstreamState.RUNNING,
"thinking": WorkstreamState.THINKING,
"attention": WorkstreamState.ATTENTION,
"idle": WorkstreamState.IDLE,
"error": WorkstreamState.ERROR,
}[state_name],
("?", dim, state_name),
)
print(f" {color_fn(f'{sym} {label}')}: {count}")
print()
tokens = agg.get("total_tokens", 0)
tools = agg.get("total_tool_calls", 0)
tok_str = f"{tokens / 1000:.1f}k" if tokens >= 1000 else str(tokens)
print(f" {dim(f'{tok_str} tokens · {tools} tool calls')}")
print()
elif sub == "nodes":
limit = 20
resp = httpx.get(
f"{console_url}/api/cluster/nodes?sort=activity&limit={limit}",
timeout=5,
headers=headers,
)
data = resp.json()
nodes = data.get("nodes", [])
total = data.get("total", len(nodes))
if not nodes:
print(dim(" No nodes discovered."))
return
# Column widths
max_name = max(len(n["node_id"]) for n in nodes)
print(
f"\n {'NODE'.ljust(max_name)} {'WS':>4} {'RUN':>4} {'ATTN':>4} {'TOKENS':>8}"
)
print(
f" {'-' * max_name} {'----':>4} {'----':>4} {'----':>4} {'--------':>8}"
)
for n in nodes:
name = n["node_id"].ljust(max_name)
ws = str(n.get("ws_total", 0))
run = n.get("ws_running", 0)
attn = n.get("ws_attention", 0)
tok = n.get("total_tokens", 0)
tok_str = f"{tok / 1000:.1f}k" if tok >= 1000 else str(tok)
run_str = green(str(run)) if run else dim("0")
attn_str = yellow(str(attn)) if attn else dim("0")
print(
f" {cyan(name)} {ws:>4} {run_str:>4} {attn_str:>4} {dim(tok_str):>8}"
)
if total > len(nodes):
print(dim(f"\n Showing {len(nodes)} of {total} nodes"))
print()
elif sub == "workstreams" or sub == "ws":
params = "sort=state&per_page=20"
# Parse optional filters: /cluster ws running, /cluster ws node=X
for arg in parts[2:]:
if "=" in arg:
key, val = arg.split("=", 1)
if key in ("state", "node", "search"):
params += f"&{key}={val}"
else:
params += f"&state={arg}"
resp = httpx.get(
f"{console_url}/api/cluster/workstreams?{params}",
timeout=5,
headers=headers,
)
data = resp.json()
ws_list = data.get("workstreams", [])
total = data.get("total", len(ws_list))
if not ws_list:
print(dim(" No matching workstreams."))
return
max_name = max(len(w.get("name", "")[:20]) for w in ws_list)
max_node = max(len(w.get("node", "")[:16]) for w in ws_list)
print(
f"\n {'STATE':<8} {'NAME'.ljust(max_name)} {'NODE'.ljust(max_node)} {'TOKENS':>8} {'CTX':>4}"
)
for w in ws_list:
state = w.get("state", "idle")
ws_state = {
"running": WorkstreamState.RUNNING,
"thinking": WorkstreamState.THINKING,
"attention": WorkstreamState.ATTENTION,
"idle": WorkstreamState.IDLE,
"error": WorkstreamState.ERROR,
}.get(state, WorkstreamState.IDLE)
sym, color_fn, label = _STATE_DISPLAY[ws_state]
name = w.get("name", "")[:20].ljust(max_name)
node = w.get("node", "")[:16].ljust(max_node)
tok = w.get("tokens", 0)
tok_str = f"{tok / 1000:.1f}k" if tok >= 1000 else str(tok)
ctx = w.get("context_ratio", 0)
ctx_str = f"{int(ctx * 100)}%" if ctx > 0 else ""
print(
f" {color_fn(f'{sym} {label:<5}')} {bold(name)} {dim(node)} {dim(tok_str):>8} {ctx_str:>4}"
)
if total > len(ws_list):
print(dim(f"\n Showing {len(ws_list)} of {total} workstreams"))
print()
elif sub == "node":
if len(parts) < 3:
print(red("Usage: /cluster node <node_id>"))
return
node_id = parts[2]
resp = httpx.get(
f"{console_url}/api/cluster/node/{node_id}", timeout=5, headers=headers
)
data = resp.json()
if "error" in data:
print(red(data["error"]))
return
ws_list = data.get("workstreams", [])
print(f"\n {bold(node_id)} ({data.get('server_url', '')})")
if not ws_list:
print(dim(" No workstreams."))
return
for w in ws_list:
state = w.get("state", "idle")
ws_state = {
"running": WorkstreamState.RUNNING,
"thinking": WorkstreamState.THINKING,
"attention": WorkstreamState.ATTENTION,
"idle": WorkstreamState.IDLE,
"error": WorkstreamState.ERROR,
}.get(state, WorkstreamState.IDLE)
sym, color_fn, _ = _STATE_DISPLAY[ws_state]
name = w.get("name", "")
title = w.get("title", "")
activity = w.get("activity", "")
print(f" {color_fn(sym)} {bold(name)} {dim(title)}")
if activity:
print(f" {dim(activity)}")
print()
else:
print(f"Unknown /cluster subcommand: {sub}")
print("Usage: /cluster [status|nodes|workstreams [state|node=X]|node <id>]")
except httpx.ConnectError:
print(red(f"Cannot connect to console at {console_url}"))
except Exception as e:
print(red(f"Cluster command failed: {e}"))
# ─── Model auto-detection ─────────────────────────────────────────────────
def detect_model(client: OpenAI) -> str:
"""Auto-detect the model from vLLM's /v1/models endpoint."""
try:
models = client.models.list()
model_ids = [m.id for m in models.data]
if not model_ids:
print(red("No models found at server. Use --model to specify."))
sys.exit(1)
if len(model_ids) == 1:
return model_ids[0]
# Multiple models -- pick first, but inform user
print(f"Available models: {', '.join(model_ids)}")
print(f"Using: {bold(model_ids[0])} (override with --model)")
return model_ids[0]
except Exception as e:
print(red(f"Could not connect to server: {e}"))
print("Is vLLM running? Start it or use --base-url to point elsewhere.")
sys.exit(1)
# ─── Main ──────────────────────────────────────────────────────────────────
def main():
parser = argparse.ArgumentParser(
description="Interactive CLI for vLLM models with tool calling.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=textwrap.dedent("""\
Examples:
python3 chat.py # auto-detect model
python3 chat.py --persona lawful_evil # with persona
python3 chat.py --model kappa_20b_131k # explicit model
python3 chat.py --temperature 0.7 # lower temperature
"""),
)
parser.add_argument(
"--base-url",
default="http://localhost:8000/v1",
help="vLLM API base URL (default: http://localhost:8000/v1)",
)
parser.add_argument(
"--model",
default=None,
help="Model name (default: auto-detect from server)",
)
parser.add_argument(
"--persona",
default=None,
help="Persona name injected as system message",
)
parser.add_argument(
"--instructions",
default=None,
help="Developer instructions injected as developer message",
)
parser.add_argument(
"--temperature",
type=float,
default=0.5,
help="Sampling temperature (default: 0.5)",
)
parser.add_argument(
"--max-tokens",
type=int,
default=32768,
help="Max completion tokens (default: 32768)",
)
parser.add_argument(
"--tool-timeout",
type=int,
default=30,
help="Bash command timeout in seconds (default: 30)",
)
parser.add_argument(
"--reasoning-effort",
default="medium",
choices=["low", "medium", "high"],
help="Reasoning effort level (default: medium)",
)
parser.add_argument(
"--context-window",
type=int,
default=131072,
help="Context window size in tokens (default: 131072)",
)
parser.add_argument(
"--compact-max-tokens",
type=int,
default=32768,
help="Max tokens for compaction summary (default: 32768)",
)
parser.add_argument(
"--auto-compact-pct",
type=float,
default=0.8,
help="Auto-compact when prompt exceeds this fraction of context window (default: 0.8)",
)
parser.add_argument(
"--agent-max-turns",
type=int,
default=-1,
help="Max tool turns for agent sub-sessions, -1 for unlimited (default: -1)",
)
parser.add_argument(
"--tool-truncation",
type=int,
default=0,
help="Tool output truncation limit in chars, 0 for auto (50%% of context window) (default: 0)",
)
parser.add_argument(
"--resume",
default=None,
metavar="SESSION",
help="Resume a previous session by alias or session_id",
)
parser.add_argument(
"--skip-permissions",
action="store_true",
help="Auto-approve all tool calls (no confirmation prompts)",
)
parser.add_argument(
"--api-key",
default=None,
help="API key (default: $OPENAI_API_KEY, or 'dummy' for local servers)",
)
parser.add_argument(
"--session-retention-days",
type=int,
default=90,
metavar="DAYS",
help="Delete unnamed sessions older than DAYS days on startup, 0 to disable (default: 90)",
)
parser.add_argument(
"--console-url",
default=None,
help="Turnstone console URL for /cluster commands (e.g., http://localhost:8090)",
)
parser.add_argument(
"--auth-token",
default=os.environ.get("TURNSTONE_AUTH_TOKEN", ""),
help="Bearer token for authenticating to turnstone services (default: $TURNSTONE_AUTH_TOKEN)",
)
from turnstone.core.config import apply_config
apply_config(parser, ["api", "model", "session", "tools", "console", "auth"])
args = parser.parse_args()
# Prune stale / empty sessions on startup
from turnstone.core.memory import prune_sessions
prune_sessions(retention_days=args.session_retention_days, log_fn=print)
# Set up readline
setup_readline()
# Create client
api_key = args.api_key or os.environ.get("OPENAI_API_KEY") or "dummy"
client = OpenAI(
base_url=args.base_url,
api_key=api_key,
)
# Detect or use provided model
if args.model:
model = args.model
else:
model = detect_model(client)
# Session factory — captures shared config for creating workstream sessions
def session_factory(ui):
return ChatSession(
client=client,
model=model,
ui=ui,
persona=args.persona,
instructions=args.instructions,
temperature=args.temperature,
max_tokens=args.max_tokens,
tool_timeout=args.tool_timeout,
reasoning_effort=args.reasoning_effort,
context_window=args.context_window,
compact_max_tokens=args.compact_max_tokens,
auto_compact_pct=args.auto_compact_pct,
agent_max_turns=args.agent_max_turns,
tool_truncation=args.tool_truncation,
)
# Create workstream manager and initial workstream
manager = WorkstreamManager(session_factory)
ws = manager.create(
ui_factory=lambda wid: WorkstreamTerminalUI(wid, manager),
)
if args.skip_permissions:
ws.ui.auto_approve = True
# Handle --resume
if args.resume:
from turnstone.core.memory import resolve_session
target_id = resolve_session(args.resume)
if not target_id:
print(red(f"Session not found: {args.resume}"))
sys.exit(1)
if not ws.session.resume_session(target_id):
print(red(f"Session '{args.resume}' has no messages."))
sys.exit(1)
print(
f"Resumed session {bold(target_id)} ({len(ws.session.messages)} messages)"
)
# Background attention notification — write to stderr while user types
def _bg_attention_notify(ws_id, state):
if state == WorkstreamState.ATTENTION and ws_id != manager.active_id:
bg_ws = manager.get(ws_id)
if bg_ws:
idx = manager.index_of(ws_id)
sys.stderr.write(
f"\a\r\033[s\033[1A\033[K"
f" {YELLOW}\u25c6 {idx}:{bg_ws.name} needs attention{RESET}"
f"\033[u"
)
sys.stderr.flush()
manager._on_state_change = _bg_attention_notify
# Print banner
print(f"\n{bold('Chat')} with {cyan(model)}")
if args.persona:
print(f"Persona: {cyan(args.persona)}")
print(f"Type /help for commands, /ws for workstreams, /exit or Ctrl+D to quit.\n")
# Prompt string -- use a short display name
display_name = model.split("/")[-1] # strip path prefixes if any
if len(display_name) > 30:
display_name = display_name[:27] + "..."
# Main loop
while True:
try:
# Show background workstream status if any need attention
if manager.count > 1:
_print_ws_status_line(manager)
# Build prompt with workstream info
active = manager.get_active()
if manager.count > 1:
idx = manager.index_of(active.id)
prompt_str = f"\001{BOLD}\002{idx}:{active.name}\001{RESET}\002 > "
else:
prompt_str = f"\001{BOLD}\002[{display_name}]\001{RESET}\002 > "
user_input = input(prompt_str)
except (EOFError, KeyboardInterrupt):
print()
break
user_input = user_input.strip()
if not user_input:
continue
if user_input.startswith("/ws"):
_handle_ws_command(manager, user_input, args.skip_permissions)
continue
if user_input.startswith("/cluster"):
_handle_cluster_command(user_input, args.console_url, args.auth_token)
continue
active = manager.get_active()
if user_input.startswith("/"):
should_exit = active.session.handle_command(user_input)
if should_exit:
break
else:
try:
active.session.send(user_input)
except KeyboardInterrupt:
print(f"\n{yellow('Interrupted.')}")
except Exception as e:
print(f"\n{red(f'Error: {e}')}")
print("Goodbye.")
if __name__ == "__main__":
main()
turnstone/console/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Cluster dashboard service for turnstone."""
turnstone/console/collector.py
+448
View File
@@ -0,0 +1,448 @@
"""Cluster state collector — aggregates data from all turnstone nodes.
Discovers nodes via Redis heartbeat keys, polls each node's /api/dashboard
endpoint for workstream data, and subscribes to the cluster event channel
for real-time state changes.
"""
from __future__ import annotations
import json
import logging
import queue
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
import httpx
from turnstone.mq.broker import RedisBroker
log = logging.getLogger("turnstone.console.collector")
@dataclass
class NodeSnapshot:
"""In-memory snapshot of a single node's state."""
node_id: str = ""
server_url: str = ""
started: float = 0.0
last_seen: float = 0.0 # monotonic time of last successful poll
max_ws: int = 10 # max workstreams (capacity)
workstreams: dict[str, dict] = field(default_factory=dict)
health: dict = field(default_factory=dict)
aggregate: dict = field(default_factory=dict)
reachable: bool = True
class ClusterCollector:
"""Aggregates cluster state from Redis and per-node HTTP APIs.
Three daemon threads:
1. Event subscriber — real-time state changes from {prefix}:events:cluster
2. Node discovery — scans heartbeat keys every ``discovery_interval`` seconds
3. Poll loop — fetches /api/dashboard from each node every ``poll_interval`` seconds
"""
def __init__(
self,
broker: RedisBroker,
prefix: str = "turnstone",
poll_interval: float = 10.0,
discovery_interval: float = 15.0,
max_poll_workers: int = 50,
http_timeout: float = 5.0,
auth_token: str = "",
):
self._broker = broker
self._prefix = prefix
self._poll_interval = poll_interval
self._discovery_interval = discovery_interval
self._max_poll_workers = max_poll_workers
self._http_timeout = http_timeout
self._lock = threading.Lock()
self._nodes: dict[str, NodeSnapshot] = {}
self._running = False
self._threads: list[threading.Thread] = []
self._poll_pool = ThreadPoolExecutor(max_workers=max_poll_workers)
headers = {}
if auth_token:
headers["Authorization"] = f"Bearer {auth_token}"
self._http_client = httpx.Client(timeout=http_timeout, headers=headers)
# SSE fan-out to browser clients
self._listeners: list[queue.Queue] = []
self._listeners_lock = threading.Lock()
# -- lifecycle -----------------------------------------------------------
def start(self) -> None:
"""Start background threads."""
self._running = True
for target, name in [
(self._event_loop, "console-events"),
(self._discovery_loop, "console-discovery"),
(self._poll_loop, "console-poll"),
]:
t = threading.Thread(target=target, name=name, daemon=True)
t.start()
self._threads.append(t)
log.info("ClusterCollector started")
def stop(self) -> None:
"""Stop all threads and clean up resources."""
self._running = False
self._poll_pool.shutdown(wait=False)
self._http_client.close()
log.info("ClusterCollector stopped")
# -- event subscription --------------------------------------------------
def _event_loop(self) -> None:
"""Subscribe to cluster events for real-time updates."""
while self._running:
try:
self._broker.subscribe_cluster(self._on_cluster_event)
while self._running:
time.sleep(1)
except Exception:
log.exception("Cluster subscription error, reconnecting in 5s")
time.sleep(5)
def _on_cluster_event(self, raw: str) -> None:
"""Handle a cluster event from Redis pub/sub."""
try:
data = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return
etype = data.get("type", "")
ws_id = data.get("ws_id", "")
node_id = data.get("node_id", "")
with self._lock:
if etype == "cluster_state" and node_id in self._nodes:
node = self._nodes[node_id]
if ws_id in node.workstreams:
ws = node.workstreams[ws_id]
ws["state"] = data.get("state", ws.get("state", "idle"))
if "tokens" in data:
ws["tokens"] = data["tokens"]
if "context_ratio" in data:
ws["context_ratio"] = data["context_ratio"]
if "activity" in data:
ws["activity"] = data["activity"]
if "activity_state" in data:
ws["activity_state"] = data["activity_state"]
elif etype == "ws_created" and node_id:
if node_id in self._nodes:
self._nodes[node_id].workstreams[ws_id] = {
"id": ws_id,
"name": data.get("name", ""),
"state": "idle",
"node": node_id,
"title": "",
"tokens": 0,
"context_ratio": 0.0,
"activity": "",
"activity_state": "",
"tool_calls": 0,
}
elif etype == "ws_closed":
for node in self._nodes.values():
node.workstreams.pop(ws_id, None)
elif etype == "ws_rename":
for node in self._nodes.values():
if ws_id in node.workstreams:
node.workstreams[ws_id]["name"] = data.get("name", "")
# Fan out to SSE listeners
self._fanout(data)
def _fanout(self, event: dict) -> None:
"""Copy an event to all registered SSE listener queues."""
with self._listeners_lock:
for q in self._listeners:
try:
q.put_nowait(event)
except queue.Full:
pass
# -- node discovery ------------------------------------------------------
def _discovery_loop(self) -> None:
"""Periodically scan Redis for active nodes."""
while self._running:
try:
self._discover_nodes()
except Exception:
log.exception("Node discovery error")
time.sleep(self._discovery_interval)
def _discover_nodes(self) -> None:
"""Scan heartbeat keys and update the node map."""
active = self._broker.list_nodes()
active_ids = set()
pending_events = []
with self._lock:
for meta in active:
nid = meta.get("node_id", "")
if not nid:
continue
active_ids.add(nid)
if nid not in self._nodes:
self._nodes[nid] = NodeSnapshot(
node_id=nid,
server_url=meta.get("server_url", ""),
started=meta.get("started", 0.0),
max_ws=meta.get("max_ws", 10),
)
pending_events.append({"type": "node_joined", "node_id": nid})
log.info("Discovered node: %s", nid)
else:
self._nodes[nid].server_url = meta.get(
"server_url", self._nodes[nid].server_url
)
# Remove nodes whose heartbeats expired
lost = [nid for nid in self._nodes if nid not in active_ids]
for nid in lost:
del self._nodes[nid]
pending_events.append({"type": "node_lost", "node_id": nid})
log.info("Lost node: %s", nid)
for event in pending_events:
self._fanout(event)
# -- polling -------------------------------------------------------------
def _poll_loop(self) -> None:
"""Periodically fetch /api/dashboard from each node."""
while self._running:
try:
self._poll_all_nodes()
except Exception:
log.exception("Poll loop error")
time.sleep(self._poll_interval)
def _poll_all_nodes(self) -> None:
"""Fetch dashboard data from all known nodes in parallel."""
with self._lock:
targets = [
(n.node_id, n.server_url)
for n in self._nodes.values()
if n.server_url and n.server_url.startswith("http")
]
if not targets:
return
futures = {
self._poll_pool.submit(self._fetch_node, nid, url): nid
for nid, url in targets
}
for future in as_completed(futures):
nid = futures[future]
try:
dashboard, health = future.result()
self._apply_poll(nid, dashboard, health)
except Exception:
log.debug("Failed to poll node %s", nid)
with self._lock:
if nid in self._nodes:
self._nodes[nid].reachable = False
def _fetch_node(self, node_id: str, server_url: str) -> tuple[dict, dict]:
"""Fetch /api/dashboard and /health from a single node."""
base = server_url.rstrip("/")
dash_resp = self._http_client.get(f"{base}/api/dashboard")
dash_data = dash_resp.json()
try:
health_resp = self._http_client.get(f"{base}/health")
health_data = health_resp.json()
except Exception:
health_data = {}
return dash_data, health_data
def _apply_poll(self, node_id: str, dashboard: dict, health: dict) -> None:
"""Apply polled data to the in-memory node snapshot."""
ws_list = dashboard.get("workstreams", [])
aggregate = dashboard.get("aggregate", {})
with self._lock:
node = self._nodes.get(node_id)
if not node:
return
node.last_seen = time.monotonic()
node.reachable = True
node.health = health
node.aggregate = aggregate
# Replace workstreams entirely from the authoritative poll
node.workstreams = {}
for ws in ws_list:
ws["node"] = node_id
node.workstreams[ws.get("id", "")] = ws
# -- query methods (thread-safe) -----------------------------------------
def get_overview(self) -> dict:
"""Return cluster overview: state counts, totals, aggregate stats."""
states = {"running": 0, "thinking": 0, "attention": 0, "idle": 0, "error": 0}
total_tokens = 0
total_tool_calls = 0
total_ws = 0
with self._lock:
for node in self._nodes.values():
for ws in node.workstreams.values():
state = ws.get("state", "idle")
states[state] = states.get(state, 0) + 1
total_ws += 1
total_tokens += node.aggregate.get("total_tokens", 0)
total_tool_calls += node.aggregate.get("total_tool_calls", 0)
node_count = len(self._nodes)
return {
"nodes": node_count,
"workstreams": total_ws,
"states": states,
"aggregate": {
"total_tokens": total_tokens,
"total_tool_calls": total_tool_calls,
},
}
def get_nodes(
self, sort_by: str = "activity", limit: int = 100, offset: int = 0
) -> tuple[list[dict], int]:
"""Return sorted, paginated node list with per-node counts."""
with self._lock:
items = []
for node in self._nodes.values():
ws_states = {
"running": 0,
"thinking": 0,
"attention": 0,
"idle": 0,
"error": 0,
}
for ws in node.workstreams.values():
s = ws.get("state", "idle")
ws_states[s] = ws_states.get(s, 0) + 1
# Use aggregate tokens if available, else sum from workstreams
agg_tokens = node.aggregate.get("total_tokens", 0)
if not agg_tokens:
agg_tokens = sum(
ws.get("tokens", 0) for ws in node.workstreams.values()
)
items.append(
{
"node_id": node.node_id,
"server_url": node.server_url,
"ws_total": len(node.workstreams),
"ws_running": ws_states["running"],
"ws_thinking": ws_states["thinking"],
"ws_attention": ws_states["attention"],
"ws_idle": ws_states["idle"],
"ws_error": ws_states["error"],
"total_tokens": agg_tokens,
"ws_tokens": agg_tokens,
"max_ws": node.max_ws,
"started": node.started,
"last_seen": node.last_seen,
"reachable": node.reachable,
"health": node.health,
}
)
total = len(items)
# Sort
if sort_by == "activity":
items.sort(key=lambda n: n["ws_running"] + n["ws_attention"], reverse=True)
elif sort_by == "tokens":
items.sort(key=lambda n: n["total_tokens"], reverse=True)
elif sort_by == "name":
items.sort(key=lambda n: n["node_id"])
return items[offset : offset + limit], total
def get_workstreams(
self,
state: str | None = None,
node: str | None = None,
search: str | None = None,
sort_by: str = "state",
page: int = 1,
per_page: int = 50,
) -> tuple[list[dict], int]:
"""Return filtered, sorted, paginated workstreams + total count."""
with self._lock:
all_ws = []
for n in self._nodes.values():
for ws in n.workstreams.values():
all_ws.append(dict(ws))
# Filter
if state:
all_ws = [ws for ws in all_ws if ws.get("state") == state]
if node:
all_ws = [ws for ws in all_ws if ws.get("node") == node]
if search:
q = search.lower()
all_ws = [
ws
for ws in all_ws
if q in ws.get("name", "").lower()
or q in ws.get("title", "").lower()
or q in ws.get("node", "").lower()
]
# Sort
state_order = {
"running": 0,
"thinking": 1,
"attention": 2,
"error": 3,
"idle": 4,
}
if sort_by == "state":
all_ws.sort(key=lambda ws: state_order.get(ws.get("state", "idle"), 9))
elif sort_by == "tokens":
all_ws.sort(key=lambda ws: ws.get("tokens", 0), reverse=True)
elif sort_by == "name":
all_ws.sort(key=lambda ws: ws.get("name", ""))
total = len(all_ws)
start = (page - 1) * per_page
page_ws = all_ws[start : start + per_page]
return page_ws, total
def get_node_detail(self, node_id: str) -> dict | None:
"""Return a single node's workstreams and health."""
with self._lock:
node = self._nodes.get(node_id)
if not node:
return None
return {
"node_id": node.node_id,
"server_url": node.server_url,
"health": dict(node.health),
"workstreams": [dict(ws) for ws in node.workstreams.values()],
"aggregate": dict(node.aggregate),
"reachable": node.reachable,
}
# -- SSE listener management ---------------------------------------------
def register_listener(self, q: queue.Queue) -> None:
"""Register a queue for SSE event fan-out."""
with self._listeners_lock:
self._listeners.append(q)
def unregister_listener(self, q: queue.Queue) -> None:
"""Unregister a queue from SSE event fan-out."""
with self._listeners_lock:
if q in self._listeners:
self._listeners.remove(q)
turnstone/console/server.py
+395
View File
@@ -0,0 +1,395 @@
"""Cluster dashboard HTTP server for turnstone.
Serves the cluster-level dashboard UI and provides REST/SSE APIs
backed by the ClusterCollector.
"""
import argparse
import json
import logging
import math
import os
import queue
import textwrap
import threading
import time
from http.server import HTTPServer, BaseHTTPRequestHandler
from pathlib import Path
from socketserver import ThreadingMixIn
from urllib.parse import urlparse, parse_qs
from turnstone.console.collector import ClusterCollector
from turnstone.mq.broker import RedisBroker
log = logging.getLogger("turnstone.console.server")
# ---------------------------------------------------------------------------
# Static assets — loaded once at startup
# ---------------------------------------------------------------------------
_STATIC_DIR = Path(__file__).parent / "static"
_HTML = ""
_CSS = ""
_JS = ""
def _load_static() -> None:
global _HTML, _CSS, _JS
_HTML = (_STATIC_DIR / "index.html").read_text(encoding="utf-8")
_CSS = (_STATIC_DIR / "style.css").read_text(encoding="utf-8")
_JS = (_STATIC_DIR / "app.js").read_text(encoding="utf-8")
# ---------------------------------------------------------------------------
# HTTP handler
# ---------------------------------------------------------------------------
class ConsoleHTTPHandler(BaseHTTPRequestHandler):
"""HTTP handler for the cluster dashboard."""
def log_message(self, format, *args):
pass # suppress default logging
def _set_headers(self, status=200, content_type="application/json"):
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Cache-Control", "no-cache")
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
def _send_json(self, data: dict, status=200):
self._set_headers(status, "application/json")
self.wfile.write(json.dumps(data).encode("utf-8"))
def _check_auth(self, method: str, path: str) -> bool:
"""Return True if authorized. Sends 401/403 and returns False otherwise."""
from turnstone.core.auth import check_request
auth_config = self.server.auth_config # type: ignore[attr-defined]
auth_header = self.headers.get("Authorization")
cookie_header = self.headers.get("Cookie")
allowed, status, msg = check_request(
auth_config, method, path, auth_header, cookie_header
)
if not allowed:
self._send_json({"error": msg}, status)
return allowed
def _read_body(self) -> dict:
length = int(self.headers.get("Content-Length", 0))
if length == 0:
return {}
raw = self.rfile.read(length)
try:
return json.loads(raw.decode("utf-8"))
except (json.JSONDecodeError, UnicodeDecodeError, ValueError):
return {}
def do_POST(self):
# Login/logout pass through _check_auth because they are in PUBLIC_PATHS.
if not self._check_auth("POST", self.path):
return
if self.path == "/api/auth/login":
from turnstone.core.auth import make_set_cookie
body = self._read_body()
token = body.get("token", "")
auth_config = self.server.auth_config # type: ignore[attr-defined]
role = auth_config.check(token)
if role:
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Set-Cookie", make_set_cookie(token))
self.send_header("Cache-Control", "no-cache")
self.end_headers()
self.wfile.write(
json.dumps({"status": "ok", "role": role}).encode("utf-8")
)
else:
self._send_json({"error": "Invalid token"}, 401)
elif self.path == "/api/auth/logout":
from turnstone.core.auth import make_clear_cookie
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Set-Cookie", make_clear_cookie())
self.send_header("Cache-Control", "no-cache")
self.end_headers()
self.wfile.write(b'{"status":"ok"}')
else:
self._send_json({"error": "Not found"}, 404)
def do_GET(self):
parsed = urlparse(self.path)
try:
if not self._check_auth("GET", parsed.path):
return
self._do_GET(parsed)
except Exception:
log.exception("Error handling GET %s", self.path)
self._send_json({"error": "Internal server error"}, 500)
@staticmethod
def _parse_int(
qs: dict, name: str, default: int, minimum: int = 0, maximum: int = 10000
) -> int:
try:
val = int(qs.get(name, [str(default)])[0])
except (ValueError, IndexError):
val = default
return max(minimum, min(val, maximum))
def _do_GET(self, parsed):
collector: ClusterCollector = self.server.collector # type: ignore[attr-defined]
if parsed.path == "/":
self._set_headers(200, "text/html; charset=utf-8")
self.wfile.write(_HTML.encode("utf-8"))
elif parsed.path == "/static/style.css":
self.send_response(200)
self.send_header("Content-Type", "text/css; charset=utf-8")
self.send_header("Cache-Control", "no-cache")
self.end_headers()
self.wfile.write(_CSS.encode("utf-8"))
elif parsed.path == "/static/app.js":
self.send_response(200)
self.send_header("Content-Type", "application/javascript; charset=utf-8")
self.send_header("Cache-Control", "no-cache")
self.end_headers()
self.wfile.write(_JS.encode("utf-8"))
elif parsed.path == "/api/cluster/overview":
self._send_json(collector.get_overview())
elif parsed.path == "/api/cluster/nodes":
qs = parse_qs(parsed.query)
sort_by = qs.get("sort", ["activity"])[0]
limit = self._parse_int(qs, "limit", 100, minimum=1, maximum=1000)
offset = self._parse_int(qs, "offset", 0)
nodes, total = collector.get_nodes(
sort_by=sort_by, limit=limit, offset=offset
)
self._send_json({"nodes": nodes, "total": total})
elif parsed.path == "/api/cluster/workstreams":
qs = parse_qs(parsed.query)
state = qs.get("state", [None])[0]
node = qs.get("node", [None])[0]
search = qs.get("search", [None])[0]
sort_by = qs.get("sort", ["state"])[0]
page = self._parse_int(qs, "page", 1, minimum=1)
per_page = self._parse_int(qs, "per_page", 50, minimum=1, maximum=200)
ws_list, total = collector.get_workstreams(
state=state,
node=node,
search=search,
sort_by=sort_by,
page=page,
per_page=per_page,
)
pages = math.ceil(total / per_page) if per_page > 0 else 0
self._send_json(
{
"workstreams": ws_list,
"total": total,
"page": page,
"per_page": per_page,
"pages": pages,
}
)
elif parsed.path.startswith("/api/cluster/node/"):
node_id = parsed.path[len("/api/cluster/node/") :]
if not node_id or "/" in node_id or len(node_id) > 256:
self._send_json({"error": "Invalid node ID"}, 400)
else:
detail = collector.get_node_detail(node_id)
if detail:
self._send_json(detail)
else:
self._send_json({"error": "Node not found"}, 404)
elif parsed.path == "/api/cluster/events":
self._handle_sse(collector)
elif parsed.path == "/health":
overview = collector.get_overview()
self._send_json(
{
"status": "ok",
"service": "turnstone-console",
"nodes": overview["nodes"],
"workstreams": overview["workstreams"],
}
)
else:
self._set_headers(404, "text/plain")
self.wfile.write(b"Not found")
def _handle_sse(self, collector: ClusterCollector):
"""Server-Sent Events stream for cluster updates."""
self.send_response(200)
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.send_header("Connection", "keep-alive")
self.send_header("Access-Control-Allow-Origin", "*")
self.end_headers()
client_queue: queue.Queue = queue.Queue(maxsize=500)
collector.register_listener(client_queue)
try:
while True:
try:
event = client_queue.get(timeout=5)
data = json.dumps(event)
self.wfile.write(f"data: {data}\n\n".encode("utf-8"))
self.wfile.flush()
except queue.Empty:
self.wfile.write(b": keepalive\n\n")
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError, OSError):
pass
finally:
collector.unregister_listener(client_queue)
def do_OPTIONS(self):
"""Handle CORS preflight."""
self.send_response(200)
self.send_header("Access-Control-Allow-Origin", "*")
self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
self.send_header("Access-Control-Allow-Headers", "Content-Type, Authorization")
self.end_headers()
# ---------------------------------------------------------------------------
# Threaded HTTP server
# ---------------------------------------------------------------------------
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
daemon_threads = True
# ---------------------------------------------------------------------------
# Main entry point
# ---------------------------------------------------------------------------
def main():
parser = argparse.ArgumentParser(
description="turnstone console — cluster dashboard service.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=textwrap.dedent("""\
Examples:
turnstone-console # default Redis on localhost
turnstone-console --port 9090 # custom port
turnstone-console --redis-host redis.internal # remote Redis
"""),
)
parser.add_argument(
"--host",
default="0.0.0.0",
help="Host to bind to (default: 0.0.0.0)",
)
parser.add_argument(
"--port",
type=int,
default=8090,
help="Port to listen on (default: 8090)",
)
parser.add_argument(
"--redis-host",
default="localhost",
help="Redis host (default: localhost)",
)
parser.add_argument(
"--redis-port",
type=int,
default=6379,
help="Redis port (default: 6379)",
)
parser.add_argument(
"--redis-password",
default=os.environ.get("REDIS_PASSWORD"),
help="Redis password (default: $REDIS_PASSWORD)",
)
parser.add_argument(
"--redis-db",
type=int,
default=0,
help="Redis DB number (default: 0)",
)
parser.add_argument(
"--poll-interval",
type=float,
default=10.0,
help="Node polling interval in seconds (default: 10)",
)
parser.add_argument(
"--log-level",
default="INFO",
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
help="Log level (default: INFO)",
)
parser.add_argument(
"--auth-token",
default=os.environ.get("TURNSTONE_AUTH_TOKEN", ""),
help="Bearer token for polling turnstone-server nodes (default: $TURNSTONE_AUTH_TOKEN)",
)
from turnstone.core.config import apply_config
apply_config(parser, ["console", "redis", "auth"])
args = parser.parse_args()
logging.basicConfig(
level=getattr(logging, args.log_level),
format="%(asctime)s %(name)s %(levelname)s %(message)s",
)
broker = RedisBroker(
host=args.redis_host,
port=args.redis_port,
db=args.redis_db,
password=args.redis_password,
)
collector = ClusterCollector(
broker=broker,
poll_interval=args.poll_interval,
auth_token=args.auth_token,
)
collector.start()
_load_static()
from turnstone.core.auth import load_auth_config
auth_config = load_auth_config()
server = ThreadedHTTPServer((args.host, args.port), ConsoleHTTPHandler)
server.collector = collector # type: ignore[attr-defined]
server.auth_config = auth_config # type: ignore[attr-defined]
print(f"turnstone console running on http://{args.host}:{args.port}")
if auth_config.enabled:
print(f"Auth: enabled ({len(auth_config.tokens)} token(s) configured)")
print("Press Ctrl+C to stop.")
try:
server.serve_forever()
except KeyboardInterrupt:
print("\nShutting down.")
collector.stop()
broker.close()
server.shutdown()
if __name__ == "__main__":
main()
turnstone/console/static/app.js
+794
View File
@@ -0,0 +1,794 @@
// --- Theme ---
function toggleTheme() {
var next = document.documentElement.dataset.theme === "light" ? "" : "light";
document.documentElement.dataset.theme = next;
localStorage.setItem("turnstone-theme", next || "dark");
var btn = document.getElementById("theme-toggle");
if (btn) btn.textContent = next === "light" ? "\u2600" : "\u263E";
}
(function initTheme() {
var stored = localStorage.getItem("turnstone-theme");
if (stored === "light") {
document.documentElement.dataset.theme = "light";
} else if (
!stored &&
window.matchMedia &&
window.matchMedia("(prefers-color-scheme: light)").matches
) {
document.documentElement.dataset.theme = "light";
}
var btn = document.getElementById("theme-toggle");
if (btn)
btn.textContent =
document.documentElement.dataset.theme === "light" ? "\u2600" : "\u263E";
})();
/* Auth-aware fetch — shows login overlay on 401 */
function authFetch(url, opts) {
return fetch(url, opts).then(function (r) {
if (r.status === 401) {
showLogin();
throw new Error("auth");
}
return r;
});
}
// --- State ---
var currentView = "overview"; // "overview" | "node" | "filtered"
var currentNodeId = null;
var currentFilter = { state: null, node: null, page: 1, per_page: 50 };
var evtSource = null;
var retryDelay = 1000;
// --- Constants ---
var STATE_DISPLAY = {
running: { symbol: "\u25b8", label: "run" },
thinking: { symbol: "\u25cc", label: "think" },
attention: { symbol: "\u25c6", label: "attn" },
idle: { symbol: "\u00b7", label: "idle" },
error: { symbol: "\u2716", label: "err" },
};
var STATE_ORDER = ["running", "thinking", "attention", "error", "idle"];
// --- Helpers ---
function escapeHtml(s) {
var el = document.createElement("span");
el.textContent = s;
return el.innerHTML;
}
function formatTokens(n) {
if (n >= 1000000) return (n / 1000000).toFixed(1) + "M";
if (n >= 1000) return (n / 1000).toFixed(1) + "k";
return String(n || 0);
}
function ctxClass(ratio) {
if (ratio <= 0) return "ctx-idle";
var pct = ratio * 100;
if (pct < 30) return "ctx-low";
if (pct < 50) return "ctx-mid";
if (pct < 80) return "ctx-high";
return "ctx-danger";
}
function formatUptime(seconds) {
if (!seconds) return "";
if (seconds < 60) return seconds + "s";
var min = Math.floor(seconds / 60);
if (min < 60) return min + "m";
var hr = Math.floor(min / 60);
return hr + "h " + (min % 60) + "m";
}
function formatCount(n) {
if (n >= 1000) return (n / 1000).toFixed(1) + "k";
return String(n);
}
// --- SSE Connection ---
function connectSSE() {
if (evtSource) {
evtSource.close();
evtSource = null;
}
evtSource = new EventSource("/api/cluster/events");
var statusBar = document.getElementById("status-bar");
evtSource.onmessage = function (e) {
retryDelay = 1000;
statusBar.classList.remove("disconnected");
statusBar.textContent = "";
try {
var data = JSON.parse(e.data);
handleClusterEvent(data);
} catch (err) {
/* ignore malformed SSE */
}
};
evtSource.onerror = function () {
evtSource.close();
evtSource = null;
statusBar.textContent = "Reconnecting\u2026";
statusBar.classList.add("disconnected");
// Raw fetch (not authFetch) — need to inspect status before throwing
fetch("/api/cluster/overview")
.then(function (r) {
if (r.status === 401) {
showLogin();
return;
}
setTimeout(connectSSE, retryDelay);
retryDelay = Math.min(retryDelay * 2, 30000);
})
.catch(function () {
setTimeout(connectSSE, retryDelay);
retryDelay = Math.min(retryDelay * 2, 30000);
});
};
}
var _refreshTimer = null;
function scheduleRefresh() {
if (_refreshTimer) return;
_refreshTimer = setTimeout(function () {
_refreshTimer = null;
if (currentView === "overview") loadOverview();
else if (currentView === "node" && currentNodeId)
loadNodeDetail(currentNodeId);
else if (currentView === "filtered") loadFilteredWorkstreams();
}, 250);
}
function handleClusterEvent(data) {
if (
data.type === "cluster_state" ||
data.type === "ws_created" ||
data.type === "ws_closed" ||
data.type === "ws_rename" ||
data.type === "node_joined" ||
data.type === "node_lost"
) {
scheduleRefresh();
}
}
// --- Overview View ---
function showOverview() {
currentView = "overview";
currentNodeId = null;
currentFilter = { state: null, node: null, page: 1, per_page: 50 };
document.getElementById("view-overview").style.display = "";
document.getElementById("view-node").style.display = "none";
document.getElementById("view-filtered").style.display = "none";
document.getElementById("breadcrumb").style.display = "none";
document.getElementById("main").scrollTop = 0;
loadOverview();
history.pushState({ view: "overview" }, "");
}
function loadOverview() {
var overviewP = authFetch("/api/cluster/overview").then(function (r) {
return r.json();
});
var nodesP = authFetch("/api/cluster/nodes?sort=activity&limit=50").then(
function (r) {
return r.json();
},
);
Promise.all([overviewP, nodesP])
.then(function (res) {
renderStateCards(res[0].states);
renderAggregateBar(res[0]);
renderNodeTable(res[1].nodes, res[1].total);
document.getElementById("cluster-summary").textContent =
res[0].nodes +
" nodes \u00b7 " +
formatCount(res[0].workstreams) +
" workstreams";
})
.catch(function () {
document.getElementById("node-table").innerHTML =
'<div class="dashboard-empty">Failed to load</div>';
});
}
function renderStateCards(states) {
var container = document.getElementById("state-cards");
container.innerHTML = "";
STATE_ORDER.forEach(function (state) {
var count = states[state] || 0;
var sd = STATE_DISPLAY[state] || STATE_DISPLAY.idle;
var card = document.createElement("div");
card.className = "state-card";
card.dataset.state = state;
card.setAttribute("role", "button");
card.setAttribute("tabindex", "0");
card.setAttribute("aria-label", sd.label + ": " + count + " workstreams");
card.innerHTML =
'<div class="state-card-count">' +
formatCount(count) +
"</div>" +
'<div class="state-card-label">' +
sd.symbol +
" " +
sd.label +
"</div>";
card.onclick = function () {
drillDownByState(state);
};
card.onkeydown = function (e) {
if (e.key === "Enter" || e.key === " ") {
e.preventDefault();
drillDownByState(state);
}
};
container.appendChild(card);
});
}
function renderAggregateBar(overview) {
var agg = overview.aggregate || {};
var parts = [];
if (agg.total_tokens) parts.push(formatTokens(agg.total_tokens) + " tokens");
if (agg.total_tool_calls)
parts.push(formatCount(agg.total_tool_calls) + " tool calls");
document.getElementById("aggregate-bar").textContent = parts.join(" \u00b7 ");
}
function renderNodeTable(nodes, total) {
var table = document.getElementById("node-table");
table.innerHTML = "";
if (!nodes.length) {
table.innerHTML = '<div class="dashboard-empty">No nodes discovered</div>';
return;
}
nodes.forEach(function (node) {
var row = document.createElement("div");
row.className = "node-row";
if (node.ws_attention > 0) row.classList.add("has-attention");
else if (node.ws_running > 0) row.classList.add("has-running");
else if (node.ws_thinking > 0) row.classList.add("has-thinking");
else if (node.ws_error > 0) row.classList.add("has-error");
row.setAttribute("role", "button");
row.setAttribute("tabindex", "0");
row.setAttribute(
"aria-label",
node.node_id +
": " +
node.ws_total +
" workstreams, " +
node.ws_running +
" running, " +
node.ws_attention +
" attention, " +
formatTokens(node.total_tokens) +
" tokens",
);
var dotClass = node.reachable ? "node-dot" : "node-dot unreachable";
// Use aggregate tokens, fall back to summed workstream tokens
var displayTokens = node.total_tokens || node.ws_tokens || 0;
// Load = workstream count / max capacity
var maxWs = node.max_ws || 10;
var healthPct = Math.round((node.ws_total / maxWs) * 100);
var healthFillClass =
healthPct < 50 ? "low" : healthPct < 80 ? "mid" : "high";
var healthFillHtml =
healthPct > 0
? '<span class="health-bar-fill ' +
healthFillClass +
'" style="width:' +
healthPct +
'%"></span>'
: "";
row.innerHTML =
'<span class="node-cell node-cell-name"><span class="' +
dotClass +
'"></span>' +
escapeHtml(node.node_id) +
"</span>" +
'<span class="node-cell node-cell-num' +
(node.ws_total > 0 ? " has-value" : "") +
'">' +
node.ws_total +
"</span>" +
'<span class="node-cell node-cell-num' +
(node.ws_running > 0 ? " has-value" : "") +
'">' +
node.ws_running +
"</span>" +
'<span class="node-cell node-cell-num' +
(node.ws_attention > 0 ? " has-value" : "") +
'">' +
node.ws_attention +
"</span>" +
'<span class="node-cell node-cell-num">' +
formatTokens(displayTokens) +
"</span>" +
'<span class="node-cell node-cell-health">' +
'<span class="health-bar">' +
healthFillHtml +
"</span>" +
" " +
healthPct +
"%" +
"</span>";
row.onclick = function () {
drillDownToNode(node.node_id, node.server_url);
};
row.onkeydown = function (e) {
if (e.key === "Enter" || e.key === " ") {
e.preventDefault();
drillDownToNode(node.node_id, node.server_url);
}
};
table.appendChild(row);
});
// Pagination hint
var pag = document.getElementById("node-pagination");
pag.innerHTML = "";
if (total > nodes.length) {
pag.textContent = "Showing " + nodes.length + " of " + total + " nodes";
}
}
// --- Drill-down: Node ---
function drillDownToNode(nodeId, serverUrl) {
currentView = "node";
currentNodeId = nodeId;
document.getElementById("view-overview").style.display = "none";
document.getElementById("view-node").style.display = "";
document.getElementById("view-filtered").style.display = "none";
document.getElementById("breadcrumb").style.display = "";
document.getElementById("breadcrumb-label").textContent = nodeId;
if (serverUrl) {
var link = document.getElementById("node-link");
link.href = serverUrl;
link.style.display = "";
}
document.getElementById("main").scrollTop = 0;
loadNodeDetail(nodeId);
document.getElementById("breadcrumb-home").focus();
history.pushState({ view: "node", nodeId: nodeId, serverUrl: serverUrl }, "");
}
function loadNodeDetail(nodeId) {
authFetch("/api/cluster/node/" + encodeURIComponent(nodeId))
.then(function (r) {
return r.json();
})
.then(function (data) {
if (data.error) {
document.getElementById("node-ws-table").innerHTML =
'<div class="dashboard-empty">' + escapeHtml(data.error) + "</div>";
return;
}
var ws = data.workstreams || [];
var active = ws.filter(function (w) {
return w.state !== "idle";
}).length;
document.getElementById("node-ws-summary").textContent =
active + " active \u00b7 " + ws.length + " total";
renderWsTable(document.getElementById("node-ws-table"), ws);
});
}
// --- Drill-down: Filtered ---
function drillDownByState(state) {
currentView = "filtered";
currentFilter = { state: state, node: null, page: 1, per_page: 50 };
document.getElementById("view-overview").style.display = "none";
document.getElementById("view-node").style.display = "none";
document.getElementById("view-filtered").style.display = "";
document.getElementById("breadcrumb").style.display = "";
var sd = STATE_DISPLAY[state] || STATE_DISPLAY.idle;
document.getElementById("breadcrumb-label").textContent =
sd.symbol + " " + sd.label;
document.getElementById("filtered-title").textContent =
"WORKSTREAMS — " + sd.label.toUpperCase();
document.getElementById("main").scrollTop = 0;
loadFilteredWorkstreams();
document.getElementById("breadcrumb-home").focus();
history.pushState({ view: "filtered", filter: currentFilter }, "");
}
function drillDownByNode(nodeId) {
currentView = "filtered";
currentFilter = { state: null, node: nodeId, page: 1, per_page: 50 };
document.getElementById("view-overview").style.display = "none";
document.getElementById("view-node").style.display = "none";
document.getElementById("view-filtered").style.display = "";
document.getElementById("breadcrumb").style.display = "";
document.getElementById("breadcrumb-label").textContent = nodeId;
document.getElementById("filtered-title").textContent =
"WORKSTREAMS — " + nodeId;
document.getElementById("main").scrollTop = 0;
loadFilteredWorkstreams();
document.getElementById("breadcrumb-home").focus();
history.pushState({ view: "filtered", filter: currentFilter }, "");
}
function loadFilteredWorkstreams() {
var params =
"page=" + currentFilter.page + "&per_page=" + currentFilter.per_page;
if (currentFilter.state)
params += "&state=" + encodeURIComponent(currentFilter.state);
if (currentFilter.node)
params += "&node=" + encodeURIComponent(currentFilter.node);
authFetch("/api/cluster/workstreams?" + params)
.then(function (r) {
return r.json();
})
.then(function (data) {
document.getElementById("main").scrollTop = 0;
document.getElementById("filtered-summary").textContent =
"Page " +
data.page +
" of " +
data.pages +
" (" +
data.total +
" total)";
renderWsTable(
document.getElementById("filtered-ws-table"),
data.workstreams,
);
renderPagination(
document.getElementById("filtered-pagination"),
data.page,
data.pages,
);
})
.catch(function () {
document.getElementById("filtered-ws-table").innerHTML =
'<div class="dashboard-empty">Failed to load</div>';
});
}
function renderPagination(container, page, pages) {
container.innerHTML = "";
if (pages <= 1) return;
var prev = document.createElement("button");
prev.textContent = "\u25c4 Prev";
prev.disabled = page <= 1;
prev.onclick = function () {
currentFilter.page--;
loadFilteredWorkstreams();
};
container.appendChild(prev);
var info = document.createElement("span");
info.textContent = page + " / " + pages;
container.appendChild(info);
var next = document.createElement("button");
next.textContent = "Next \u25ba";
next.disabled = page >= pages;
next.onclick = function () {
currentFilter.page++;
loadFilteredWorkstreams();
};
container.appendChild(next);
}
// --- Workstream table renderer (shared) ---
function renderWsTable(container, wsList) {
container.innerHTML = "";
if (!wsList.length) {
container.innerHTML = '<div class="dashboard-empty">No workstreams</div>';
return;
}
wsList.forEach(function (ws) {
var state = ws.state || "idle";
var sd = STATE_DISPLAY[state] || STATE_DISPLAY.idle;
var row = document.createElement("div");
row.className = "dash-row";
row.dataset.wsId = ws.id || "";
row.dataset.state = state;
row.setAttribute("tabindex", "0");
row.setAttribute("role", "button");
var ariaLabel = sd.label + ": " + (ws.name || ws.id || "unnamed");
if (ws.node) ariaLabel += " on " + ws.node;
if (ws.title) ariaLabel += ", task: " + ws.title;
if (ws.tokens) ariaLabel += ", " + formatTokens(ws.tokens) + " tokens";
if (ws.context_ratio > 0)
ariaLabel += ", " + Math.round(ws.context_ratio * 100) + "% context";
row.setAttribute("aria-label", ariaLabel);
var main = document.createElement("div");
main.className = "dash-row-main";
// STATE
var stateCell = document.createElement("span");
stateCell.className = "dash-cell-state";
stateCell.innerHTML =
'<span class="dash-state-dot" data-state="' +
escapeHtml(state) +
'" aria-hidden="true"></span>' +
'<span class="dash-state-label" data-state="' +
escapeHtml(state) +
'">' +
sd.symbol +
" " +
sd.label +
"</span>";
main.appendChild(stateCell);
// NAME
var nameCell = document.createElement("span");
nameCell.className = "dash-cell-name";
nameCell.textContent = ws.name || ws.id || "";
main.appendChild(nameCell);
// NODE (clickable)
var nodeCell = document.createElement("span");
nodeCell.className = "dash-cell-node";
nodeCell.textContent = ws.node || "";
nodeCell.onclick = function (e) {
e.stopPropagation();
if (ws.node) drillDownByNode(ws.node);
};
main.appendChild(nodeCell);
// TASK
var taskCell = document.createElement("span");
taskCell.className = "dash-cell-task";
taskCell.textContent = ws.title || "";
main.appendChild(taskCell);
// TOKENS
var tokensCell = document.createElement("span");
tokensCell.className = "dash-cell-tokens";
tokensCell.textContent = ws.tokens ? formatTokens(ws.tokens) : "";
main.appendChild(tokensCell);
// CTX
var ctxCell = document.createElement("span");
ctxCell.className = "dash-cell-ctx " + ctxClass(ws.context_ratio || 0);
ctxCell.textContent =
ws.context_ratio > 0 ? Math.round(ws.context_ratio * 100) + "%" : "";
main.appendChild(ctxCell);
row.appendChild(main);
// Sub-line
var sub = document.createElement("div");
sub.className = "dash-row-sub";
if (ws.activity_state === "approval") sub.classList.add("sub-attention");
sub.textContent = ws.activity || "";
row.appendChild(sub);
container.appendChild(row);
});
}
// --- Login Overlay ---
var _loginTrapHandler = null;
var _loginBusy = false;
function initLogin() {
var overlay = document.createElement("div");
overlay.id = "login-overlay";
overlay.style.display = "none";
overlay.setAttribute("role", "dialog");
overlay.setAttribute("aria-modal", "true");
overlay.setAttribute("aria-labelledby", "login-title");
overlay.innerHTML =
'<div id="login-box">' +
'<h2 id="login-title">turnstone console</h2>' +
'<div id="login-error" role="alert" aria-live="assertive"></div>' +
'<label for="login-token" class="sr-only">Auth token</label>' +
'<input id="login-token" type="password" placeholder="Enter auth token" autocomplete="off">' +
'<button id="login-submit">Sign in</button>' +
"</div>";
document.body.appendChild(overlay);
document.getElementById("login-submit").onclick = submitLogin;
document
.getElementById("login-token")
.addEventListener("keydown", function (e) {
if (e.key === "Enter") submitLogin();
if (e.key === "Escape") {
var errEl = document.getElementById("login-error");
if (errEl && errEl.style.display !== "none") {
errEl.style.display = "none";
errEl.textContent = "";
}
}
});
}
function showLogin() {
var overlay = document.getElementById("login-overlay");
if (!overlay) return;
overlay.style.display = "flex";
document.body.style.overflow = "hidden";
var logoutBtn = document.getElementById("logout-btn");
if (logoutBtn) logoutBtn.style.display = "none";
var errEl = document.getElementById("login-error");
if (errEl) {
errEl.style.display = "none";
errEl.textContent = "";
}
setTimeout(function () {
var inp = document.getElementById("login-token");
if (inp) {
inp.value = "";
inp.focus();
}
}, 50);
// Focus trap
if (_loginTrapHandler)
document.removeEventListener("keydown", _loginTrapHandler);
_loginTrapHandler = function (e) {
if (e.key === "Tab") {
var box = document.getElementById("login-box");
var focusable = box.querySelectorAll("input, button");
var first = focusable[0];
var last = focusable[focusable.length - 1];
if (e.shiftKey) {
if (document.activeElement === first) {
e.preventDefault();
last.focus();
}
} else {
if (document.activeElement === last) {
e.preventDefault();
first.focus();
}
}
}
};
document.addEventListener("keydown", _loginTrapHandler);
}
function hideLogin() {
var overlay = document.getElementById("login-overlay");
if (overlay) overlay.style.display = "none";
document.body.style.overflow = "";
if (_loginTrapHandler) {
document.removeEventListener("keydown", _loginTrapHandler);
_loginTrapHandler = null;
}
}
function submitLogin() {
if (_loginBusy) return;
var token = (document.getElementById("login-token").value || "").trim();
if (!token) {
var errEl = document.getElementById("login-error");
if (errEl) {
errEl.textContent = "Token is required";
errEl.style.display = "block";
}
document.getElementById("login-token").focus();
return;
}
_loginBusy = true;
var btn = document.getElementById("login-submit");
var inp = document.getElementById("login-token");
btn.disabled = true;
btn.textContent = "Signing in\u2026";
inp.disabled = true;
fetch("/api/auth/login", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ token: token }),
})
.then(function (r) {
if (r.status === 401 || r.status === 403) throw new Error("invalid");
if (!r.ok) throw new Error("server");
return r.json();
})
.then(function () {
_loginBusy = false;
btn.disabled = false;
btn.textContent = "Sign in";
inp.disabled = false;
hideLogin();
document.getElementById("logout-btn").style.display = "";
connectSSE();
if (currentView === "overview") loadOverview();
else if (currentView === "node") drillDownToNode(currentNodeId);
else if (currentView === "filtered") loadFilteredWorkstreams();
})
.catch(function (err) {
_loginBusy = false;
btn.disabled = false;
btn.textContent = "Sign in";
inp.disabled = false;
var errEl = document.getElementById("login-error");
if (errEl) {
errEl.textContent =
err.message === "invalid"
? "Invalid token"
: "Connection failed \u2014 try again";
errEl.style.display = "block";
}
});
}
function logout() {
fetch("/api/auth/logout", { method: "POST" }).then(function () {
if (evtSource) {
evtSource.close();
evtSource = null;
}
showLogin();
});
}
// --- Navigation ---
window.addEventListener("popstate", function (e) {
var overlay = document.getElementById("login-overlay");
if (overlay && overlay.style.display !== "none") return;
if (!e.state) {
showOverview();
return;
}
if (e.state.view === "overview") showOverview();
else if (e.state.view === "node" && e.state.nodeId)
drillDownToNode(e.state.nodeId, e.state.serverUrl);
else if (e.state.view === "filtered" && e.state.filter) {
currentFilter = e.state.filter;
if (currentFilter.state) drillDownByState(currentFilter.state);
else if (currentFilter.node) drillDownByNode(currentFilter.node);
}
});
// --- Keyboard shortcuts help ---
function showKbHelp() {
var existing = document.getElementById("kb-overlay");
if (existing) {
existing.remove();
}
var overlay = document.createElement("div");
overlay.id = "kb-overlay";
overlay.innerHTML =
'<div id="kb-box" role="dialog" aria-modal="true" aria-label="Keyboard shortcuts">' +
"<h2>Keyboard shortcuts</h2>" +
'<div class="kb-section">Navigation</div>' +
'<div class="kb-row"><span class="kb-desc">Activate card / row</span><span class="kb-key">Enter</span></div>' +
'<div class="kb-row"><span class="kb-desc">Activate card / row</span><span class="kb-key">Space</span></div>' +
'<div class="kb-row"><span class="kb-desc">Navigate rows</span><span class="kb-key">\u2191</span> <span class="kb-key">\u2193</span></div>' +
'<div class="kb-section">General</div>' +
'<div class="kb-row"><span class="kb-desc">Show this help</span><span class="kb-key">?</span></div>' +
'<div class="kb-row"><span class="kb-desc">Close overlay</span><span class="kb-key">Esc</span></div>' +
'<div class="kb-hint">Press <span class="kb-key">Esc</span> to close</div>' +
"</div>";
overlay.onclick = function (e) {
if (e.target === overlay) hideKbHelp();
};
document.body.appendChild(overlay);
document.getElementById("kb-box").focus();
}
function hideKbHelp() {
var el = document.getElementById("kb-overlay");
if (el) el.remove();
}
document.addEventListener("keydown", function (e) {
// Don't trigger when typing in inputs or when login overlay is open
if (e.target.tagName === "INPUT" || e.target.tagName === "TEXTAREA") return;
var login = document.getElementById("login-overlay");
if (login && login.style.display !== "none") return;
if (e.key === "?" && !e.ctrlKey && !e.metaKey) {
e.preventDefault();
showKbHelp();
}
if (e.key === "Escape") {
var kb = document.getElementById("kb-overlay");
if (kb) {
e.preventDefault();
hideKbHelp();
}
}
});
// --- Init ---
history.replaceState({ view: "overview" }, "");
initLogin();
connectSSE();
loadOverview();
// Try loading — if auth required, login overlay will show
turnstone/console/static/index.html
+83
View File
@@ -0,0 +1,83 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>turnstone console</title>
<link rel="stylesheet" href="/static/style.css">
</head>
<body>
<div id="header">
<h1>turnstone console</h1>
<span id="cluster-summary" aria-live="polite"></span>
<span id="status-bar" role="status" aria-live="assertive"></span>
<button id="logout-btn" onclick="logout()" style="display:none; background:none; border:1px solid var(--border); color:var(--fg-dim); border-radius:var(--radius); padding:4px 8px; cursor:pointer; font:inherit; font-size:12px;">logout</button>
<button id="theme-toggle" onclick="toggleTheme()" aria-label="Toggle light/dark theme" style="margin-left:auto; background:none; border:1px solid var(--border); color:var(--fg); border-radius:var(--radius); padding:4px 8px; cursor:pointer; font:inherit; font-size:12px;">&#9790;</button>
</div>
<nav id="breadcrumb" class="breadcrumb" style="display:none" aria-label="Breadcrumb">
<a href="#" id="breadcrumb-home" onclick="showOverview(); return false">Cluster</a>
<span class="breadcrumb-sep" aria-hidden="true">&gt;</span>
<span id="breadcrumb-label" aria-current="page"></span>
</nav>
<div id="main">
<!-- CLUSTER OVERVIEW -->
<div id="view-overview">
<div class="state-cards" id="state-cards"></div>
<div class="aggregate-bar" id="aggregate-bar"></div>
<div class="section-header">NODES</div>
<div class="node-colheaders" aria-hidden="true">
<span class="ncol ncol-node">NODE</span>
<span class="ncol ncol-ws">WS</span>
<span class="ncol ncol-run">RUN</span>
<span class="ncol ncol-attn">ATTN</span>
<span class="ncol ncol-tokens">TOKENS</span>
<span class="ncol ncol-health">LOAD</span>
</div>
<div id="node-table" role="group" aria-label="Nodes" aria-live="polite">
<div class="dashboard-empty">Loading cluster data...</div>
</div>
<div id="node-pagination" class="pagination"></div>
</div>
<!-- NODE DRILL-DOWN -->
<div id="view-node" style="display:none">
<div class="dash-header">
<span class="dash-header-title">WORKSTREAMS</span>
<span class="dash-header-summary" id="node-ws-summary"></span>
</div>
<div class="dash-colheaders" aria-hidden="true">
<span class="dash-col dash-col-state">STATE</span>
<span class="dash-col dash-col-name">NAME</span>
<span class="dash-col dash-col-node">NODE</span>
<span class="dash-col dash-col-task">TASK</span>
<span class="dash-col dash-col-tokens">TOKENS</span>
<span class="dash-col dash-col-ctx">CTX</span>
</div>
<div id="node-ws-table" class="dash-table" role="group" aria-label="Workstreams" aria-live="polite"></div>
<a id="node-link" class="node-link" target="_blank" rel="noopener">Open node dashboard</a>
</div>
<!-- FILTERED WORKSTREAMS -->
<div id="view-filtered" style="display:none">
<div class="dash-header">
<span class="dash-header-title" id="filtered-title">WORKSTREAMS</span>
<span class="dash-header-summary" id="filtered-summary"></span>
</div>
<div class="dash-colheaders" aria-hidden="true">
<span class="dash-col dash-col-state">STATE</span>
<span class="dash-col dash-col-name">NAME</span>
<span class="dash-col dash-col-node">NODE</span>
<span class="dash-col dash-col-task">TASK</span>
<span class="dash-col dash-col-tokens">TOKENS</span>
<span class="dash-col dash-col-ctx">CTX</span>
</div>
<div id="filtered-ws-table" class="dash-table" role="group" aria-label="Workstreams" aria-live="polite"></div>
<div id="filtered-pagination" class="pagination"></div>
</div>
</div>
<script src="/static/app.js"></script>
</body>
</html>
turnstone/console/static/style.css
+205
View File
@@ -0,0 +1,205 @@
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
:root {
--bg: #1a1b26; --bg-surface: #24283b; --bg-highlight: #292e42;
--fg: #c8d1f5; --fg-dim: #828db5; --fg-bright: #a9b1d6;
--accent: #7aa2f7; --green: #9ece6a; --red: #f7768e;
--yellow: #e0af68; --cyan: #7dcfff; --magenta: #bb9af7;
--border: #3b4261; --code-bg: #1f2335;
--radius: 8px;
--dash-grid: 72px 120px 100px 1fr 60px 48px;
}
[data-theme="light"] {
--bg: #f5f5f5; --bg-surface: #ffffff; --bg-highlight: #e8e8ec;
--fg: #1a1a2e; --fg-dim: #4b5563; --fg-bright: #374151;
--accent: #1d4ed8; --green: #15803d; --red: #b91c1c;
--yellow: #92400e; --cyan: #0e7490; --magenta: #7e22ce;
--border: #d1d5db; --code-bg: #eaeaef;
}
html, body { height: 100%; background: var(--bg); color: var(--fg); font-family: 'SF Mono', 'Cascadia Code', 'Fira Code', 'Consolas', monospace; font-size: 14px; }
body { display: flex; flex-direction: column; }
/* Header */
#header { padding: 8px 16px; background: var(--bg-surface); border-bottom: 1px solid var(--border); display: flex; align-items: center; gap: 12px; flex-shrink: 0; }
#header h1 { font-size: 16px; color: var(--accent); font-weight: 600; }
#cluster-summary { font-size: 12px; color: var(--fg-dim); }
#status-bar { font-size: 12px; color: var(--fg-dim); margin-left: auto; }
#status-bar.disconnected { color: var(--red); }
/* Main content */
#main { flex: 1; overflow-y: auto; padding: 16px; max-width: 1100px; margin: 0 auto; width: 100%; }
/* Breadcrumb */
.breadcrumb { padding: 8px 16px; font-size: 12px; color: var(--fg-dim); background: var(--bg-surface); border-bottom: 1px solid var(--border); }
.breadcrumb a { color: var(--accent); text-decoration: none; }
.breadcrumb a:hover { text-decoration: underline; }
.breadcrumb-sep { margin: 0 6px; color: var(--fg-dim); }
/* State cards */
.state-cards { display: flex; gap: 10px; margin-bottom: 16px; flex-wrap: wrap; }
.state-card { flex: 1; min-width: 100px; background: var(--bg-surface); border: 1px solid var(--border); border-top: 3px solid var(--fg-dim); border-radius: var(--radius); padding: 12px 14px; cursor: pointer; transition: border-color 0.15s, background 0.15s; text-align: center; }
.state-card:hover { background: var(--bg-highlight); border-color: var(--accent); }
.state-card:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
.state-card[data-state="running"] { border-top-color: var(--green); }
.state-card[data-state="thinking"] { border-top-color: var(--accent); }
.state-card[data-state="attention"] { border-top-color: var(--yellow); }
.state-card[data-state="idle"] { border-top-color: var(--fg-dim); }
.state-card[data-state="error"] { border-top-color: var(--red); }
.state-card-count { font-size: 24px; font-weight: bold; color: var(--fg-bright); margin-bottom: 2px; }
.state-card-label { font-size: 11px; color: var(--fg-dim); text-transform: uppercase; letter-spacing: 0.05em; }
/* Aggregate bar */
.aggregate-bar { font-size: 11px; color: var(--fg-dim); margin-bottom: 20px; }
/* Section header */
.section-header { font-size: 12px; font-weight: bold; color: var(--accent); letter-spacing: 0.05em; margin-bottom: 8px; }
/* Node table */
.node-colheaders { display: grid; grid-template-columns: 1fr 50px 50px 50px 70px 140px; padding: 4px 16px; background: var(--bg-surface); border-bottom: 1px solid var(--border); font-size: 11px; color: var(--fg-dim); text-transform: uppercase; letter-spacing: 0.03em; position: sticky; top: 0; z-index: 10; }
.ncol-ws, .ncol-run, .ncol-attn, .ncol-tokens { text-align: right; }
.ncol-health, .node-cell-health { padding-left: 10px; }
.node-row { display: grid; grid-template-columns: 1fr 50px 50px 50px 70px 140px; padding: 8px 16px; cursor: pointer; transition: background 0.15s; border-left: 3px solid transparent; }
.node-row:nth-child(odd) { background: var(--bg); }
.node-row:nth-child(even) { background: var(--bg-surface); }
.node-row:hover { background: var(--bg-highlight); box-shadow: inset 0 0 0 1px var(--border); }
.node-row:focus-visible { outline: 2px solid var(--accent); outline-offset: -2px; }
.node-row.has-attention { border-left-color: var(--yellow); }
.node-row.has-running { border-left-color: var(--green); }
.node-row.has-thinking { border-left-color: var(--accent); }
.node-row.has-error { border-left-color: var(--red); }
.node-cell { font-size: 12px; display: flex; align-items: center; }
.node-cell-name { color: var(--fg-bright); font-weight: bold; gap: 6px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.node-cell-name .node-dot { width: 6px; height: 6px; border-radius: 50%; background: var(--green); flex-shrink: 0; }
.node-cell-name .node-dot.unreachable { background: var(--red); }
.node-cell-num { color: var(--fg-dim); font-size: 11px; justify-content: flex-end; }
.node-cell-num.has-value { color: var(--fg-bright); }
.node-cell-health { gap: 6px; font-size: 11px; color: var(--fg-dim); }
.health-bar { width: 80px; height: 6px; background: var(--bg-highlight); border-radius: 3px; overflow: hidden; }
.health-bar-fill { display: block; height: 100%; min-width: 4px; border-radius: 3px; transition: width 0.3s; }
.health-bar-fill.low { background: var(--green); }
.health-bar-fill.mid { background: var(--yellow); }
.health-bar-fill.high { background: var(--red); }
/* Dashboard table (reused from per-node dashboard) */
.dash-header { display: flex; justify-content: space-between; align-items: center; padding: 8px 16px; background: var(--code-bg); border-radius: var(--radius) var(--radius) 0 0; }
.dash-header-title { color: var(--accent); font-size: 12px; font-weight: bold; letter-spacing: 0.05em; }
.dash-header-summary { color: var(--fg-dim); font-size: 11px; }
.dash-colheaders { display: grid; grid-template-columns: var(--dash-grid); padding: 4px 16px; background: var(--bg-surface); border-bottom: 1px solid var(--border); font-size: 11px; color: var(--fg-dim); text-transform: uppercase; letter-spacing: 0.03em; position: sticky; top: 0; z-index: 10; }
.dash-col-tokens, .dash-col-ctx { text-align: right; }
.dash-table { min-height: 40px; }
.dash-row { position: relative; border-left: 3px solid transparent; cursor: pointer; transition: background 0.15s; }
.dash-row:nth-child(odd) { background: var(--bg); }
.dash-row:nth-child(even) { background: var(--bg-surface); }
.dash-row:hover { background: var(--bg-highlight); box-shadow: inset 0 0 0 1px var(--border); }
.dash-row:focus-visible { outline: 2px solid var(--accent); outline-offset: -2px; }
.dash-row[data-state="running"] { border-left-color: var(--green); }
.dash-row[data-state="thinking"] { border-left-color: var(--accent); }
.dash-row[data-state="attention"] { border-left-color: var(--yellow); }
.dash-row[data-state="idle"] { border-left-color: var(--fg-dim); opacity: 0.7; }
.dash-row[data-state="error"] { border-left-color: var(--red); }
.dash-row-main { display: grid; grid-template-columns: var(--dash-grid); padding: 8px 16px 2px; align-items: center; font-size: 12px; }
.dash-row-sub { padding: 0 16px 8px 88px; font-size: 11px; color: var(--fg-dim); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.dash-row-sub.sub-attention { color: var(--yellow); }
.dash-cell-state { display: flex; align-items: center; gap: 6px; font-size: 11px; }
.dash-state-dot { width: 6px; height: 6px; border-radius: 50%; flex-shrink: 0; }
.dash-state-dot[data-state="running"] { background: var(--green); border-radius: 2px; animation: pulse 2s infinite; }
.dash-state-dot[data-state="thinking"] { background: var(--accent); animation: pulse 2.2s infinite; }
.dash-state-dot[data-state="attention"] { background: var(--yellow); border-radius: 1px; transform: rotate(45deg); animation: pulse 1.8s infinite; }
.dash-state-dot[data-state="idle"] { background: var(--fg-dim); }
.dash-state-dot[data-state="error"] { background: var(--red); border-radius: 0; }
@keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.3; } }
.dash-state-label { white-space: nowrap; }
.dash-state-label[data-state="running"] { color: var(--green); }
.dash-state-label[data-state="thinking"] { color: var(--accent); }
.dash-state-label[data-state="attention"] { color: var(--yellow); }
.dash-state-label[data-state="idle"] { color: var(--fg-dim); }
.dash-state-label[data-state="error"] { color: var(--red); }
.dash-cell-name { font-weight: bold; color: var(--fg-bright); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.dash-row[data-state="idle"] .dash-cell-name { color: var(--fg-dim); }
.dash-cell-node { color: var(--accent); font-size: 11px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; cursor: pointer; }
.dash-cell-node:hover { text-decoration: underline; }
.dash-cell-task { color: var(--fg-bright); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.dash-row[data-state="idle"] .dash-cell-task { color: var(--fg-dim); }
.dash-cell-tokens { text-align: right; color: var(--fg-dim); font-size: 11px; }
.dash-cell-ctx { text-align: right; font-size: 11px; }
.dash-cell-ctx.ctx-low { color: var(--green); }
.dash-cell-ctx.ctx-mid { color: var(--yellow); }
.dash-cell-ctx.ctx-high { color: var(--red); }
.dash-cell-ctx.ctx-danger { color: var(--red); font-weight: bold; }
.dash-cell-ctx.ctx-idle { color: var(--fg-dim); }
/* Node link */
.node-link { display: inline-block; margin-top: 12px; color: var(--accent); font-size: 12px; text-decoration: none; }
.node-link:hover { text-decoration: underline; }
/* Pagination */
.pagination { display: flex; align-items: center; justify-content: center; gap: 8px; padding: 12px 0; font-size: 12px; color: var(--fg-dim); }
.pagination button { background: var(--bg-surface); border: 1px solid var(--border); color: var(--fg-bright); border-radius: 4px; padding: 4px 10px; font: inherit; font-size: 12px; cursor: pointer; }
.pagination button:hover { background: var(--bg-highlight); }
.pagination button:disabled { opacity: 0.3; cursor: not-allowed; }
/* Empty state */
.dashboard-empty { color: var(--fg-dim); font-size: 13px; padding: 16px 0; text-align: center; }
/* Focus */
:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
/* Scrollbar */
::-webkit-scrollbar { width: 8px; }
::-webkit-scrollbar-track { background: var(--bg); }
::-webkit-scrollbar-thumb { background: var(--border); border-radius: 4px; }
/* Reduced motion */
@media (prefers-reduced-motion: reduce) {
.dash-state-dot[data-state="running"],
.dash-state-dot[data-state="thinking"],
.dash-state-dot[data-state="attention"] { animation: none; opacity: 1; }
}
/* Responsive */
@media (max-width: 700px) {
:root { --dash-grid: 68px 110px 1fr 56px 44px; }
.dash-col-node, .dash-cell-node { display: none; }
.node-colheaders, .node-row { grid-template-columns: 1fr 40px 40px 40px 60px; }
.ncol-health, .node-cell-health { display: none; }
}
@media (max-width: 480px) {
:root { --dash-grid: 50px 1fr 50px; }
.dash-col-node, .dash-cell-node, .dash-col-task, .dash-cell-task, .dash-col-ctx, .dash-cell-ctx { display: none; }
.state-cards { flex-wrap: nowrap; overflow-x: auto; gap: 8px; }
.state-card { min-width: 70px; flex: 0 0 auto; padding: 8px 10px; }
.state-card-count { font-size: 18px; }
.state-card-label { font-size: 10px; }
}
/* Login overlay */
#login-overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.7); display: flex; align-items: center; justify-content: center; z-index: 1000; }
#login-box { background: var(--bg-surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 32px; width: 320px; max-width: 90vw; box-shadow: 0 8px 32px rgba(0,0,0,0.4); }
#login-box h2 { color: var(--accent); font-size: 16px; margin-bottom: 16px; }
#login-box input { width: 100%; padding: 10px 12px; background: var(--bg); border: 1px solid var(--border); border-radius: 4px; color: var(--fg); font: inherit; font-size: 13px; margin-bottom: 12px; }
#login-box input:focus-visible { border-color: var(--accent); outline: none; box-shadow: 0 0 0 2px color-mix(in srgb, var(--accent) 30%, transparent); }
#login-box input::placeholder { color: var(--fg-dim); }
#login-box button { width: 100%; padding: 12px; background: var(--accent); color: var(--bg); border: none; border-radius: 4px; font: inherit; font-size: 13px; font-weight: 600; cursor: pointer; }
#login-box button:hover { opacity: 0.9; }
#login-box button:focus-visible { outline: 2px solid var(--fg); outline-offset: 2px; }
#login-box button:disabled { opacity: 0.5; cursor: not-allowed; }
#login-error { color: var(--red); font-size: 12px; margin-bottom: 8px; display: none; }
.sr-only { position: absolute; width: 1px; height: 1px; padding: 0; margin: -1px; overflow: hidden; clip: rect(0,0,0,0); white-space: nowrap; border: 0; }
@media (max-width: 380px) { #login-box { padding: 24px 20px; } }
/* Keyboard shortcuts overlay */
#kb-overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.7); display: flex; align-items: center; justify-content: center; z-index: 999; }
#kb-box { background: var(--bg-surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 24px 28px; width: 360px; max-width: 90vw; max-height: 80vh; overflow-y: auto; box-shadow: 0 8px 32px rgba(0,0,0,0.4); }
#kb-box h2 { color: var(--accent); font-size: 14px; margin-bottom: 14px; }
.kb-row { display: flex; justify-content: space-between; padding: 4px 0; font-size: 12px; }
.kb-key { color: var(--fg-bright); background: var(--bg-highlight); border: 1px solid var(--border); border-radius: 3px; padding: 1px 6px; font-family: inherit; font-size: 11px; white-space: nowrap; }
.kb-desc { color: var(--fg-dim); }
.kb-section { color: var(--fg-dim); font-size: 11px; text-transform: uppercase; letter-spacing: 0.05em; margin-top: 12px; margin-bottom: 4px; }
.kb-section:first-child { margin-top: 0; }
#kb-box .kb-hint { color: var(--fg-dim); font-size: 11px; text-align: center; margin-top: 14px; }
turnstone/core/__init__.py
View File
turnstone/core/auth.py
+232
View File
@@ -0,0 +1,232 @@
"""Bearer token authentication and authorization for turnstone HTTP servers.
Opt-in via the ``[auth]`` section in ``config.toml``. When auth is disabled
(the default), all requests pass through unchecked. When enabled, API
requests must include a valid ``Authorization: Bearer <token>`` header or
a ``turnstone_auth`` cookie (set via the ``/api/auth/login`` endpoint).
Each token has a role: ``"read"`` or ``"full"``.
Public paths (``/``, ``/static/*``, ``/health``, ``/metrics``,
``/api/auth/login``, ``/api/auth/logout``) are always accessible
without authentication.
"""
from __future__ import annotations
import hmac
import logging
import os
from dataclasses import dataclass, field
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Public / write path classification
# ---------------------------------------------------------------------------
AUTH_COOKIE = "turnstone_auth"
PUBLIC_PATHS: frozenset[str] = frozenset(
{"/", "/health", "/metrics", "/api/auth/login", "/api/auth/logout"}
)
PUBLIC_PREFIXES: tuple[str, ...] = ("/static/",)
WRITE_PATHS: frozenset[str] = frozenset(
{
"/api/send",
"/api/approve",
"/api/plan",
"/api/command",
"/api/workstreams/new",
"/api/workstreams/close",
}
)
# ---------------------------------------------------------------------------
# AuthConfig
# ---------------------------------------------------------------------------
@dataclass
class AuthConfig:
"""Auth configuration loaded once at startup (not modified after creation)."""
enabled: bool = False
tokens: dict[str, str] = field(default_factory=dict) # token_value → role
def check(self, token: str | None) -> str | None:
"""Return the role (``"read"`` or ``"full"``) for a valid token, or *None*."""
if not token:
return None
for known_token, role in self.tokens.items():
if hmac.compare_digest(token, known_token):
return role
return None
# ---------------------------------------------------------------------------
# Loading
# ---------------------------------------------------------------------------
def load_auth_config() -> AuthConfig:
"""Build :class:`AuthConfig` from ``config.toml`` ``[auth]`` + env vars.
Config format::
[auth]
enabled = true
[[auth.tokens]]
value = "tok_abc123"
role = "full"
Environment variable fallbacks:
- ``TURNSTONE_AUTH_ENABLED=1`` — enables auth
- ``TURNSTONE_AUTH_TOKEN=<token>`` — registers a single full-access token
"""
from turnstone.core.config import load_config
auth_cfg = load_config("auth")
enabled = bool(auth_cfg.get("enabled", False))
tokens: dict[str, str] = {}
# Tokens from config file (TOML array-of-tables)
for entry in auth_cfg.get("tokens", []):
value = entry.get("value", "") if isinstance(entry, dict) else ""
role = entry.get("role", "read") if isinstance(entry, dict) else ""
if value and role in ("read", "full"):
tokens[value] = role
# Environment variable fallbacks
if os.environ.get("TURNSTONE_AUTH_ENABLED", "").strip() in ("1", "true", "yes"):
enabled = True
env_token = os.environ.get("TURNSTONE_AUTH_TOKEN", "").strip()
if env_token:
tokens[env_token] = "full"
if enabled and not tokens:
log.warning(
"Auth enabled but no tokens configured — all API requests will be rejected"
)
return AuthConfig(enabled=enabled, tokens=tokens)
# ---------------------------------------------------------------------------
# Path helpers
# ---------------------------------------------------------------------------
def is_public_path(path: str) -> bool:
"""Return *True* if the path should be accessible without authentication."""
if path in PUBLIC_PATHS:
return True
for prefix in PUBLIC_PREFIXES:
if path.startswith(prefix):
return True
return False
def required_role(method: str, path: str) -> str:
"""Return the minimum role needed for *method* + *path*.
Returns ``"full"`` for state-modifying POST endpoints, ``"read"`` otherwise.
"""
if method == "POST" and path in WRITE_PATHS:
return "full"
return "read"
# ---------------------------------------------------------------------------
# Request checking — single entry point for HTTP handlers
# ---------------------------------------------------------------------------
def check_request(
auth_config: AuthConfig,
method: str,
path: str,
auth_header: str | None,
cookie_header: str | None = None,
) -> tuple[bool, int, str]:
"""Validate a request against the auth config.
Checks ``Authorization: Bearer <token>`` first, then falls back to the
``turnstone_auth`` cookie (set by ``/api/auth/login``).
Returns ``(allowed, status_code, message)``.
On success: ``(True, 200, "")``.
On failure: ``(False, 401|403, "error message")``.
"""
if not auth_config.enabled:
return True, 200, ""
if is_public_path(path):
return True, 200, ""
# Try Bearer header first, then cookie
token = _extract_bearer(auth_header)
if token is None:
token = _extract_cookie(cookie_header, AUTH_COOKIE)
role = auth_config.check(token)
if role is None:
return False, 401, "Unauthorized: missing or invalid token"
needed = required_role(method, path)
if needed == "full" and role != "full":
return False, 403, "Forbidden: read-only token cannot access this endpoint"
return True, 200, ""
def _extract_bearer(header: str | None) -> str | None:
"""Extract the token from ``Bearer <token>`` header value."""
if not header:
return None
parts = header.split(None, 1)
if len(parts) == 2 and parts[0].lower() == "bearer":
return parts[1]
return None
def _extract_cookie(cookie_header: str | None, name: str) -> str | None:
"""Extract a named value from a ``Cookie`` header.
Assumes token values are simple ASCII (no URL-encoding).
"""
if not cookie_header:
return None
for pair in cookie_header.split(";"):
pair = pair.strip()
if "=" in pair:
k, v = pair.split("=", 1)
if k.strip() == name:
return v.strip()
return None
# ---------------------------------------------------------------------------
# Cookie helpers for login/logout endpoints
# ---------------------------------------------------------------------------
def make_set_cookie(token: str, max_age: int = 86400 * 30, secure: bool = False) -> str:
"""Return a ``Set-Cookie`` header value that stores the auth token.
Set *secure* to ``True`` when serving over HTTPS to add the ``Secure``
flag (prevents cookie from being sent over plain HTTP).
"""
val = f"{AUTH_COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age={max_age}"
if secure:
val += "; Secure"
return val
def make_clear_cookie() -> str:
"""Return a ``Set-Cookie`` header value that expires the auth cookie."""
return f"{AUTH_COOKIE}=; Path=/; HttpOnly; SameSite=Lax; Max-Age=0"
turnstone/core/config.py
+120
View File
@@ -0,0 +1,120 @@
"""Unified configuration for turnstone.
Loads ``~/.config/turnstone/config.toml`` and applies values as argparse defaults.
Precedence: CLI args > env vars > config file > hardcoded defaults.
"""
from __future__ import annotations
import argparse
import logging
from pathlib import Path
import tomllib
log = logging.getLogger(__name__)
CONFIG_DIR = Path("~/.config/turnstone").expanduser()
CONFIG_PATH = CONFIG_DIR / "config.toml"
# Cache: None = not loaded yet, {} = loaded but empty/missing
_cache: dict | None = None
def load_config(section: str | None = None) -> dict:
"""Load config.toml and return the full dict or a specific section.
Returns empty dict if file doesn't exist or can't be parsed.
Result is cached after first call.
"""
global _cache
if _cache is None:
_cache = {}
if CONFIG_PATH.is_file():
try:
_cache = tomllib.loads(CONFIG_PATH.read_text(encoding="utf-8"))
except Exception as exc:
log.warning("Failed to parse %s: %s", CONFIG_PATH, exc)
if section:
return _cache.get(section, {})
return _cache
# ---------------------------------------------------------------------------
# Section → {config_key: argparse_dest}
# ---------------------------------------------------------------------------
_CONFIG_MAP: dict[str, dict[str, str]] = {
"api": {
"base_url": "base_url",
"api_key": "api_key",
},
"model": {
"name": "model",
"temperature": "temperature",
"max_tokens": "max_tokens",
"reasoning_effort": "reasoning_effort",
"context_window": "context_window",
},
"session": {
"persona": "persona",
"instructions": "instructions",
"retention_days": "session_retention_days",
"compact_max_tokens": "compact_max_tokens",
"auto_compact_pct": "auto_compact_pct",
},
"tools": {
"timeout": "tool_timeout",
"truncation": "tool_truncation",
"agent_max_turns": "agent_max_turns",
"skip_permissions": "skip_permissions",
},
"server": {
"host": "host",
"port": "port",
"workstream_idle_timeout": "workstream_idle_timeout",
},
"bridge": {
"server_url": "server_url",
"node_id": "node_id",
"approval_timeout": "approval_timeout",
"heartbeat_ttl": "heartbeat_ttl",
"log_level": "log_level",
},
"redis": {
"host": "redis_host",
"port": "redis_port",
"password": "redis_password",
"db": "redis_db",
},
"console": {
"host": "host",
"port": "port",
"url": "console_url",
"poll_interval": "poll_interval",
"log_level": "log_level",
},
"auth": {
"token": "auth_token",
},
}
def apply_config(parser: argparse.ArgumentParser, sections: list[str]) -> None:
"""Set argparse defaults from config file.
Only sets defaults for keys present in the config file.
Called before ``parse_args()`` so CLI flags still override.
"""
cfg = load_config()
if not cfg:
return
defaults: dict[str, object] = {}
for section in sections:
mapping = _CONFIG_MAP.get(section, {})
section_data = cfg.get(section, {})
for config_key, argparse_dest in mapping.items():
if config_key in section_data:
defaults[argparse_dest] = section_data[config_key]
if defaults:
parser.set_defaults(**defaults)
turnstone/core/edit.py
+60
View File
@@ -0,0 +1,60 @@
"""Edit helpers for precise string replacement in files."""
def find_occurrences(content: str, old_string: str) -> list[int]:
"""Return 1-based line numbers where each occurrence of old_string starts."""
if not old_string:
return []
# Build a prefix-sum of line starts for O(1) line-number lookup.
line_starts = [0]
for i, ch in enumerate(content):
if ch == "\n":
line_starts.append(i + 1)
results = []
start = 0
while True:
idx = content.find(old_string, start)
if idx == -1:
break
# bisect: find the line containing idx
lo, hi = 0, len(line_starts) - 1
while lo < hi:
mid = (lo + hi + 1) // 2
if line_starts[mid] <= idx:
lo = mid
else:
hi = mid - 1
results.append(lo + 1) # 1-based
start = idx + 1
return results
def pick_nearest(content: str, old_string: str, near_line: int) -> int:
"""Return the char index of the occurrence of old_string nearest to near_line."""
line_starts = [0]
for i, ch in enumerate(content):
if ch == "\n":
line_starts.append(i + 1)
best_idx = -1
best_dist = float("inf")
start = 0
while True:
idx = content.find(old_string, start)
if idx == -1:
break
# Find line number for this occurrence
lo, hi = 0, len(line_starts) - 1
while lo < hi:
mid = (lo + hi + 1) // 2
if line_starts[mid] <= idx:
lo = mid
else:
hi = mid - 1
line_num = lo + 1
dist = abs(line_num - near_line)
if dist < best_dist:
best_dist = dist
best_idx = idx
start = idx + 1
return best_idx
turnstone/core/memory.py
+541
View File
@@ -0,0 +1,541 @@
"""SQLite database for persistent memories and conversation history."""
import os
import sqlite3
from datetime import datetime, timedelta
TURNSTONE_DB = os.path.join(os.getcwd(), ".turnstone.db")
db_override: str | None = None
db_initialized: set[str] = set()
_fts5_available: bool = False
_tavily_key: str | None = None
_tavily_key_loaded: bool = False
def get_tavily_key() -> str | None:
"""Load Tavily API key (cached after first call).
Precedence: config.toml [api] tavily_key → $TAVILY_API_KEY
"""
global _tavily_key, _tavily_key_loaded
if _tavily_key_loaded:
return _tavily_key
_tavily_key_loaded = True
from turnstone.core.config import load_config
cfg_key = load_config("api").get("tavily_key", "").strip()
if cfg_key:
_tavily_key = cfg_key
return _tavily_key
env_key = os.environ.get("TAVILY_API_KEY", "").strip()
if env_key:
_tavily_key = env_key
return _tavily_key
def open_db() -> sqlite3.Connection:
"""Open the turnstone database, creating tables on first use per path."""
global _fts5_available
path = db_override or TURNSTONE_DB
conn = sqlite3.connect(path)
if path not in db_initialized:
conn.execute(
"CREATE TABLE IF NOT EXISTS memories "
"(key TEXT PRIMARY KEY, value TEXT NOT NULL, "
"created TEXT NOT NULL, updated TEXT NOT NULL)"
)
conn.execute(
"CREATE TABLE IF NOT EXISTS conversations "
"(id INTEGER PRIMARY KEY AUTOINCREMENT, "
"session_id TEXT NOT NULL, timestamp TEXT NOT NULL, "
"role TEXT NOT NULL, content TEXT, "
"tool_name TEXT, tool_args TEXT)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_conv_session ON conversations(session_id)"
)
# Migration: add tool_call_id column if missing (for session resume)
try:
conn.execute("SELECT tool_call_id FROM conversations LIMIT 0")
except sqlite3.OperationalError:
conn.execute("ALTER TABLE conversations ADD COLUMN tool_call_id TEXT")
conn.commit()
# Sessions table — maps session_id to human-friendly alias/title
conn.execute(
"CREATE TABLE IF NOT EXISTS sessions "
"(session_id TEXT PRIMARY KEY, alias TEXT UNIQUE, "
"title TEXT, created TEXT NOT NULL, updated TEXT NOT NULL)"
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_sessions_alias ON sessions(alias)")
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_sessions_updated ON sessions(updated)"
)
try:
# Check if FTS table already exists
fts_exists = conn.execute(
"SELECT 1 FROM sqlite_master "
"WHERE type='table' AND name='conversations_fts'"
).fetchone()
if not fts_exists:
conn.execute(
"CREATE VIRTUAL TABLE conversations_fts "
"USING fts5(content, content=conversations, content_rowid=id)"
)
conn.execute(
"INSERT INTO conversations_fts(conversations_fts) VALUES('rebuild')"
)
conn.commit()
_fts5_available = True
except Exception:
_fts5_available = False
db_initialized.add(path)
return conn
def normalize_key(key: str) -> str:
"""Normalize a memory key for consistent lookup."""
return key.lower().replace("-", "_").replace(" ", "_")
def load_memories() -> list[tuple[str, str]]:
"""Return all (key, value) pairs sorted by key."""
try:
conn = open_db()
try:
return conn.execute(
"SELECT key, value FROM memories ORDER BY key"
).fetchall()
finally:
conn.close()
except Exception:
return []
def save_message(
session_id: str,
role: str,
content: str | None,
tool_name: str | None = None,
tool_args: str | None = None,
tool_call_id: str | None = None,
) -> None:
"""Log a message to the conversations table."""
global _fts5_available
try:
conn = open_db()
try:
conn.execute(
"INSERT INTO conversations (session_id, timestamp, role, content, "
"tool_name, tool_args, tool_call_id) "
"VALUES (?, datetime('now'), ?, ?, ?, ?, ?)",
(session_id, role, content, tool_name, tool_args, tool_call_id),
)
if _fts5_available and content:
try:
rowid = conn.execute("SELECT last_insert_rowid()").fetchone()[0]
conn.execute(
"INSERT INTO conversations_fts(rowid, content) VALUES (?, ?)",
(rowid, content),
)
except Exception:
_fts5_available = False # degrade to LIKE for rest of session
# Bump session updated timestamp
conn.execute(
"UPDATE sessions SET updated = datetime('now') WHERE session_id = ?",
(session_id,),
)
conn.commit()
finally:
conn.close()
except Exception:
pass # Don't let logging failures break the session
def escape_like(s: str) -> str:
"""Escape LIKE metacharacters for use with ESCAPE '\\\\'."""
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
def fts5_query(query: str) -> str:
"""Convert a plain search string into a safe FTS5 query.
Quotes each term so FTS5 special characters (*, -, etc.) are treated
as literals, then joins with implicit AND. Embedded double quotes
are doubled per FTS5 quoting convention.
"""
terms = query.split()
safe = []
for t in terms:
if t:
safe.append(f'"{t.replace(chr(34), chr(34) + chr(34))}"')
return " ".join(safe)
def search_history(query: str, limit: int = 20) -> list[tuple]:
"""Search conversation history. Returns (timestamp, session_id, role, content, tool_name)."""
if not query or not query.strip():
return []
try:
conn = open_db()
try:
if _fts5_available:
return conn.execute(
"SELECT c.timestamp, c.session_id, c.role, c.content, c.tool_name "
"FROM conversations_fts f "
"JOIN conversations c ON c.id = f.rowid "
"WHERE conversations_fts MATCH ? "
"ORDER BY f.rank ASC LIMIT ?",
(fts5_query(query), min(limit, 100)),
).fetchall()
return conn.execute(
"SELECT timestamp, session_id, role, content, tool_name "
"FROM conversations WHERE content LIKE ? ESCAPE '\\' "
"ORDER BY timestamp DESC LIMIT ?",
(f"%{escape_like(query)}%", min(limit, 100)),
).fetchall()
finally:
conn.close()
except Exception:
return []
def search_history_recent(limit: int = 20) -> list[tuple]:
"""Return most recent conversation messages."""
try:
conn = open_db()
try:
return conn.execute(
"SELECT timestamp, session_id, role, content, tool_name "
"FROM conversations ORDER BY timestamp DESC LIMIT ?",
(min(limit, 100),),
).fetchall()
finally:
conn.close()
except Exception:
return []
# ── Session management ────────────────────────────────────────────────
def register_session(session_id: str, title: str | None = None) -> None:
"""Create a sessions row for a new session (no-op if already exists)."""
try:
conn = open_db()
try:
conn.execute(
"INSERT OR IGNORE INTO sessions "
"(session_id, title, created, updated) "
"VALUES (?, ?, datetime('now'), datetime('now'))",
(session_id, title),
)
conn.commit()
finally:
conn.close()
except Exception:
pass
def update_session_title(session_id: str, title: str) -> None:
"""Set or update the auto-generated title for a session."""
try:
conn = open_db()
try:
conn.execute(
"UPDATE sessions SET title = ? WHERE session_id = ?",
(title, session_id),
)
conn.commit()
finally:
conn.close()
except Exception:
pass
def set_session_alias(session_id: str, alias: str) -> bool:
"""Set a human-friendly alias for a session. Returns False if alias is taken."""
try:
conn = open_db()
try:
existing = conn.execute(
"SELECT session_id FROM sessions WHERE alias = ?", (alias,)
).fetchone()
if existing and existing[0] != session_id:
return False
conn.execute(
"UPDATE sessions SET alias = ? WHERE session_id = ?",
(alias, session_id),
)
conn.commit()
return True
finally:
conn.close()
except Exception:
return False
def get_session_name(session_id: str) -> str | None:
"""Return the alias (or title if no alias) for a session, or None if unset."""
try:
conn = open_db()
try:
row = conn.execute(
"SELECT alias, title FROM sessions WHERE session_id = ?",
(session_id,),
).fetchone()
if row:
return row[0] or row[1] or None
finally:
conn.close()
except Exception:
pass
return None
def resolve_session(alias_or_id: str) -> str | None:
"""Resolve an alias or session_id (or prefix) to a full session_id."""
try:
conn = open_db()
try:
# 1. Exact alias match
row = conn.execute(
"SELECT session_id FROM sessions WHERE alias = ?",
(alias_or_id,),
).fetchone()
if row:
return row[0]
# 2. Exact session_id match
row = conn.execute(
"SELECT session_id FROM sessions WHERE session_id = ?",
(alias_or_id,),
).fetchone()
if row:
return row[0]
# 3. Session_id prefix match
rows = conn.execute(
"SELECT session_id FROM sessions WHERE session_id LIKE ?",
(alias_or_id + "%",),
).fetchall()
if len(rows) == 1:
return rows[0][0]
# 4. Fallback: check conversations table for legacy sessions
row = conn.execute(
"SELECT DISTINCT session_id FROM conversations "
"WHERE session_id = ? LIMIT 1",
(alias_or_id,),
).fetchone()
if row:
# Auto-register legacy session
conn.execute(
"INSERT OR IGNORE INTO sessions "
"(session_id, created, updated) VALUES ("
"?, "
"(SELECT MIN(timestamp) FROM conversations WHERE session_id = ?), "
"(SELECT MAX(timestamp) FROM conversations WHERE session_id = ?))",
(row[0], row[0], row[0]),
)
conn.commit()
return row[0]
return None
finally:
conn.close()
except Exception:
return None
def prune_sessions(
retention_days: int = 90,
log_fn=None,
) -> tuple[int, int]:
"""Prune orphaned and stale sessions.
Removes:
- Sessions with no messages (orphaned registrations from process startup).
- Sessions whose ``updated`` timestamp is older than ``retention_days``
**and** that have no alias (named sessions are kept indefinitely).
Pass ``retention_days=0`` to skip age-based pruning (only orphans removed).
Returns:
(orphans_removed, stale_removed)
"""
orphans = stale = 0
try:
conn = open_db()
try:
# 1. Remove sessions that have no messages at all.
cur = conn.execute(
"DELETE FROM sessions "
"WHERE NOT EXISTS "
" (SELECT 1 FROM conversations c WHERE c.session_id = sessions.session_id)"
)
orphans = cur.rowcount
# 2. Remove old unnamed sessions.
if retention_days > 0:
cutoff = (datetime.utcnow() - timedelta(days=retention_days)).strftime(
"%Y-%m-%dT%H:%M:%S"
)
cur = conn.execute(
"DELETE FROM sessions WHERE alias IS NULL AND updated < ?",
(cutoff,),
)
stale = cur.rowcount
conn.commit()
finally:
conn.close()
except Exception:
return (0, 0)
if log_fn and (orphans or stale):
parts = []
if orphans:
parts.append(f"{orphans} empty session{'s' if orphans != 1 else ''}")
if stale:
parts.append(
f"{stale} session{'s' if stale != 1 else ''} "
f"older than {retention_days} days"
)
log_fn(f"[turnstone] Session cleanup: removed {', '.join(parts)}.")
return (orphans, stale)
def list_sessions(limit: int = 20) -> list[tuple]:
"""List recent sessions.
Returns (session_id, alias, title, created, updated, msg_count)
ordered by updated DESC.
"""
try:
conn = open_db()
try:
return conn.execute(
"SELECT s.session_id, s.alias, s.title, s.created, s.updated, "
"(SELECT COUNT(*) FROM conversations c "
" WHERE c.session_id = s.session_id) "
"FROM sessions s "
"WHERE EXISTS "
" (SELECT 1 FROM conversations c WHERE c.session_id = s.session_id) "
"ORDER BY s.updated DESC LIMIT ?",
(limit,),
).fetchall()
finally:
conn.close()
except Exception:
return []
def load_session_messages(session_id: str) -> list[dict]:
"""Load messages for a session and reconstruct OpenAI message format.
Handles tool_call / tool_result rows by grouping consecutive tool_call
rows into one assistant message with tool_calls, then pairing subsequent
tool_result rows as tool messages.
"""
try:
conn = open_db()
try:
rows = conn.execute(
"SELECT role, content, tool_name, tool_args, tool_call_id "
"FROM conversations WHERE session_id = ? ORDER BY id",
(session_id,),
).fetchall()
finally:
conn.close()
except Exception:
return []
messages: list[dict] = []
i = 0
while i < len(rows):
role, content, tool_name, tool_args, tc_id = rows[i]
if role == "user":
messages.append({"role": "user", "content": content or ""})
i += 1
elif role == "assistant":
messages.append({"role": "assistant", "content": content})
i += 1
elif role == "tool_call":
# Collect consecutive tool_call rows into one assistant message.
# If the previous message was an assistant with content (text +
# tool calls in the same turn), merge tool_calls into it.
assistant_msg: dict = {
"role": "assistant",
"content": None,
"tool_calls": [],
}
if (
messages
and messages[-1]["role"] == "assistant"
and not messages[-1].get("tool_calls")
):
assistant_msg = messages.pop()
assistant_msg["tool_calls"] = []
while i < len(rows) and rows[i][0] == "tool_call":
_, _, tn, ta, stored_tc_id = rows[i]
call_id = stored_tc_id or f"call_{session_id}_{i}"
assistant_msg["tool_calls"].append(
{
"id": call_id,
"type": "function",
"function": {"name": tn or "", "arguments": ta or ""},
}
)
i += 1
messages.append(assistant_msg)
# Consume matching tool_result rows
result_idx = 0
while i < len(rows) and rows[i][0] == "tool_result":
_, result_content, _, _, result_tc_id = rows[i]
if result_tc_id:
tc_id_to_use = result_tc_id
elif result_idx < len(assistant_msg["tool_calls"]):
tc_id_to_use = assistant_msg["tool_calls"][result_idx]["id"]
else:
tc_id_to_use = f"call_orphan_{i}"
messages.append(
{
"role": "tool",
"tool_call_id": tc_id_to_use,
"content": result_content or "",
}
)
result_idx += 1
i += 1
elif role == "tool_result":
# Orphaned tool_result (no preceding tool_call) — skip
i += 1
else:
i += 1
return messages
def delete_session(session_id: str) -> bool:
"""Delete a session and all its messages. Returns True on success."""
try:
conn = open_db()
try:
conn.execute(
"DELETE FROM conversations WHERE session_id = ?",
(session_id,),
)
conn.execute(
"DELETE FROM sessions WHERE session_id = ?",
(session_id,),
)
conn.commit()
return True
finally:
conn.close()
except Exception:
return False
turnstone/core/metrics.py
+270
View File
@@ -0,0 +1,270 @@
"""Thread-safe Prometheus-compatible metrics collector for the turnstone web server."""
import threading
import time
from collections import defaultdict
class MetricsCollector:
"""Collects server metrics and generates Prometheus text exposition format."""
BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
def __init__(self):
self._lock = threading.Lock()
self.start_time = time.monotonic()
self.model: str = ""
# counters
self._req_total: dict = defaultdict(int) # (method, endpoint, status) -> int
self._tokens: dict = defaultdict(int) # ("prompt"|"completion") -> int
self._messages: int = 0
self._tool_calls: dict = defaultdict(int) # tool_name -> int
self._errors: int = 0
# histograms: (method, endpoint) -> {buckets: [count…], sum: float, count: int}
self._req_duration: dict = {}
# gauge
self._context_ratio: float = 0.0
def record_request(self, method: str, endpoint: str, status: int, duration: float):
with self._lock:
self._req_total[(method, endpoint, str(status))] += 1
key = (method, endpoint)
if key not in self._req_duration:
self._req_duration[key] = {
"buckets": [0] * len(self.BUCKETS),
"sum": 0.0,
"count": 0,
}
h = self._req_duration[key]
for i, b in enumerate(self.BUCKETS):
if duration <= b:
h["buckets"][i] += 1
h["sum"] += duration
h["count"] += 1
def record_tokens(self, prompt: int, completion: int):
with self._lock:
self._tokens["prompt"] += prompt
self._tokens["completion"] += completion
def record_tool_call(self, tool_name: str):
with self._lock:
self._tool_calls[tool_name] += 1
def record_error(self):
with self._lock:
self._errors += 1
def record_message_sent(self):
with self._lock:
self._messages += 1
def record_context_ratio(self, ratio: float):
with self._lock:
self._context_ratio = ratio
def generate_text(
self,
workstream_states: dict,
total_workstreams: int,
workstream_metrics: list[dict] | None = None,
) -> str:
"""Return Prometheus text exposition format (v0.0.4)."""
lines: list[str] = []
def gauge(name, help_text, value, labels=None):
lstr = _fmt_labels(labels)
lines.append(f"# HELP {name} {help_text}")
lines.append(f"# TYPE {name} gauge")
lines.append(f"{name}{lstr} {_fmt_value(value)}")
def counter(name, help_text, value, labels=None):
lstr = _fmt_labels(labels)
lines.append(f"# HELP {name} {help_text}")
lines.append(f"# TYPE {name} counter")
lines.append(f"{name}{lstr} {_fmt_value(value)}")
with self._lock:
uptime = time.monotonic() - self.start_time
model = self.model
req_total = dict(self._req_total)
tokens = dict(self._tokens)
messages = self._messages
tool_calls = dict(self._tool_calls)
errors = self._errors
req_duration = {k: dict(v) for k, v in self._req_duration.items()}
context_ratio = self._context_ratio
# turnstone_build_info
lines.append("# HELP turnstone_build_info Server version and model info")
lines.append("# TYPE turnstone_build_info gauge")
lines.append(f'turnstone_build_info{{version="0.2.0",model="{model}"}} 1')
# turnstone_uptime_seconds
gauge("turnstone_uptime_seconds", "Server uptime in seconds", uptime)
# turnstone_workstreams_active_total
gauge(
"turnstone_workstreams_active_total",
"Number of active workstreams",
total_workstreams,
)
# turnstone_workstreams_by_state
lines.append("# HELP turnstone_workstreams_by_state Workstreams grouped by state")
lines.append("# TYPE turnstone_workstreams_by_state gauge")
for state, count in sorted(workstream_states.items()):
lines.append(f'turnstone_workstreams_by_state{{state="{state}"}} {count}')
# turnstone_http_requests_total
lines.append("# HELP turnstone_http_requests_total Total HTTP requests handled")
lines.append("# TYPE turnstone_http_requests_total counter")
for (method, endpoint, status), count in sorted(req_total.items()):
lines.append(
f'turnstone_http_requests_total{{method="{method}",'
f'endpoint="{endpoint}",status_code="{status}"}} {count}'
)
# turnstone_http_request_duration_seconds (histogram)
lines.append(
"# HELP turnstone_http_request_duration_seconds HTTP request duration in seconds"
)
lines.append("# TYPE turnstone_http_request_duration_seconds histogram")
for (method, endpoint), h in sorted(req_duration.items()):
prefix = (
f'turnstone_http_request_duration_seconds{{method="{method}",'
f'endpoint="{endpoint}"'
)
for i, b in enumerate(self.BUCKETS):
lines.append(f'{prefix},le="{b}"}} {h["buckets"][i]}')
lines.append(f'{prefix},le="+Inf"}} {h["count"]}')
lines.append(
f'turnstone_http_request_duration_seconds_sum{{method="{method}",'
f'endpoint="{endpoint}"}} {_fmt_value(h["sum"])}'
)
lines.append(
f'turnstone_http_request_duration_seconds_count{{method="{method}",'
f'endpoint="{endpoint}"}} {h["count"]}'
)
# turnstone_messages_sent_total
counter("turnstone_messages_sent_total", "Total user messages sent to AI", messages)
# turnstone_tokens_total
lines.append("# HELP turnstone_tokens_total Total tokens consumed")
lines.append("# TYPE turnstone_tokens_total counter")
for tok_type in ("prompt", "completion"):
lines.append(
f'turnstone_tokens_total{{type="{tok_type}"}} {tokens.get(tok_type, 0)}'
)
# turnstone_tool_calls_total
lines.append("# HELP turnstone_tool_calls_total Total tool executions by name")
lines.append("# TYPE turnstone_tool_calls_total counter")
for tool, count in sorted(tool_calls.items()):
lines.append(f'turnstone_tool_calls_total{{tool="{tool}"}} {count}')
# turnstone_errors_total
counter("turnstone_errors_total", "Total errors reported by workstreams", errors)
# turnstone_context_window_used_ratio
gauge(
"turnstone_context_window_used_ratio",
"Fraction of context window currently used (0.0 - 1.0)",
context_ratio,
)
# Per-workstream metrics (only when data is provided)
if workstream_metrics:
# turnstone_workstream_info — exposes session_id as a label for joining,
# without propagating that high-cardinality label to counters.
lines.append(
"# HELP turnstone_workstream_info Workstream metadata"
" (join on session_id for per-session queries)"
)
lines.append("# TYPE turnstone_workstream_info gauge")
for wm in workstream_metrics:
lstr = _fmt_labels(
{
"ws_id": wm["ws_id"],
"name": wm["name"],
"session_id": wm["session_id"],
}
)
lines.append(f"turnstone_workstream_info{lstr} 1")
lines.append(
"# HELP turnstone_workstream_prompt_tokens_total"
" Prompt tokens consumed per workstream (lifetime of workstream)"
)
lines.append("# TYPE turnstone_workstream_prompt_tokens_total counter")
for wm in workstream_metrics:
lstr = _fmt_labels({"ws_id": wm["ws_id"], "name": wm["name"]})
lines.append(
f"turnstone_workstream_prompt_tokens_total{lstr} {wm['prompt_tokens']}"
)
lines.append(
"# HELP turnstone_workstream_completion_tokens_total"
" Completion tokens generated per workstream (lifetime of workstream)"
)
lines.append("# TYPE turnstone_workstream_completion_tokens_total counter")
for wm in workstream_metrics:
lstr = _fmt_labels({"ws_id": wm["ws_id"], "name": wm["name"]})
lines.append(
f"turnstone_workstream_completion_tokens_total{lstr}"
f" {wm['completion_tokens']}"
)
lines.append(
"# HELP turnstone_workstream_messages_total"
" User messages sent per workstream (lifetime of workstream)"
)
lines.append("# TYPE turnstone_workstream_messages_total counter")
for wm in workstream_metrics:
lstr = _fmt_labels({"ws_id": wm["ws_id"], "name": wm["name"]})
lines.append(f"turnstone_workstream_messages_total{lstr} {wm['messages']}")
lines.append(
"# HELP turnstone_workstream_tool_calls_total"
" Tool executions per workstream per tool (lifetime of workstream)"
)
lines.append("# TYPE turnstone_workstream_tool_calls_total counter")
for wm in workstream_metrics:
for tool, cnt in sorted(wm["tool_calls"].items()):
lstr = _fmt_labels(
{"ws_id": wm["ws_id"], "name": wm["name"], "tool": tool}
)
lines.append(f"turnstone_workstream_tool_calls_total{lstr} {cnt}")
lines.append(
"# HELP turnstone_workstream_context_ratio"
" Current context window utilisation per workstream (0.0-1.0)"
)
lines.append("# TYPE turnstone_workstream_context_ratio gauge")
for wm in workstream_metrics:
lstr = _fmt_labels({"ws_id": wm["ws_id"], "name": wm["name"]})
lines.append(
f"turnstone_workstream_context_ratio{lstr} {_fmt_value(wm['context_ratio'])}"
)
lines.append("") # trailing newline
return "\n".join(lines)
def _fmt_labels(labels: dict | None) -> str:
if not labels:
return ""
parts = [f'{k}="{v}"' for k, v in labels.items()]
return "{" + ",".join(parts) + "}"
def _fmt_value(v: float) -> str:
if isinstance(v, int):
return str(v)
# Use full precision but strip trailing zeros
return f"{v:.6g}"
# Module-level metrics instance — shared across all requests and WebUI instances.
metrics = MetricsCollector()
turnstone/core/safety.py
+41
View File
@@ -0,0 +1,41 @@
"""Command safety guards — soft guardrails against destructive commands."""
# Soft guardrail — catches common accidental destructive commands but is
# trivially bypassable (e.g. extra spaces, shell variable expansion).
# The user approval prompt is the primary security boundary.
BLOCKED_PATTERNS = [
"rm -rf /",
"rm -rf /*",
"mkfs",
"shutdown",
"reboot",
"halt",
"poweroff",
"dd if=",
":(){ :|:& };:", # fork bomb
"> /dev/sda",
"mv / ",
"chmod -R 777 /",
"chown -R ",
]
def sanitize_command(cmd: str) -> str:
"""Replace common unicode look-alikes that break the shell."""
return (
cmd.replace("\u2018", "'") # left single curly quote
.replace("\u2019", "'") # right single curly quote
.replace("\u201c", '"') # left double curly quote
.replace("\u201d", '"') # right double curly quote
.replace("\u2013", "-") # en dash
.replace("\u2014", "-") # em dash
)
def is_command_blocked(cmd: str) -> str | None:
"""Return reason string if command is blocked, None otherwise."""
cmd_stripped = cmd.strip()
for pattern in BLOCKED_PATTERNS:
if pattern in cmd_stripped:
return f"Blocked: command matches dangerous pattern '{pattern}'"
return None
turnstone/core/sandbox.py
+308
View File
@@ -0,0 +1,308 @@
"""Sandboxed Python executor for the math tool."""
import ast
import multiprocessing
import re
import traceback
_MATH_BLOCKED_BUILTINS = {
"open",
"exec",
"eval",
"compile",
"input",
"breakpoint",
"memoryview",
"globals",
"locals",
"vars",
}
_MATH_BLOCKED_MODULES = {
"os",
"sys",
"subprocess",
"shutil",
"pathlib",
"socket",
"http",
"urllib",
"requests",
"pickle",
"marshal",
"shelve",
"dbm",
"sqlite3",
"ctypes",
"multiprocessing",
"threading",
"asyncio",
"concurrent",
"signal",
"pty",
"tty",
"termios",
"fcntl",
"resource",
"syslog",
"tempfile",
"io",
"builtins",
"__builtin__",
"importlib",
}
class _ASTValidator(ast.NodeVisitor):
"""Validates AST for dangerous constructs."""
def __init__(self):
self.errors: list[str] = []
def visit_Import(self, node):
for alias in node.names:
if alias.name.split(".")[0] in _MATH_BLOCKED_MODULES:
self.errors.append(f"Import of '{alias.name}' is not allowed")
self.generic_visit(node)
def visit_ImportFrom(self, node):
if node.module and node.module.split(".")[0] in _MATH_BLOCKED_MODULES:
self.errors.append(f"Import from '{node.module}' is not allowed")
self.generic_visit(node)
def visit_Call(self, node):
if isinstance(node.func, ast.Name) and node.func.id in _MATH_BLOCKED_BUILTINS:
self.errors.append(f"Call to '{node.func.id}' is not allowed")
self.generic_visit(node)
def visit_Attribute(self, node):
if node.attr.startswith("__") and node.attr.endswith("__"):
if node.attr not in {"__name__", "__doc__", "__class__"}:
self.errors.append(f"Access to '{node.attr}' is not allowed")
self.generic_visit(node)
def validate_math_code(code: str) -> list[str]:
"""Validate code for dangerous constructs. Returns list of errors."""
try:
tree = ast.parse(code)
except SyntaxError as e:
lines = code.split("\n")
msg = f"Syntax error on line {e.lineno}: {e.msg}"
if e.lineno and e.lineno <= len(lines):
msg += f"\n {e.lineno}: {lines[e.lineno - 1]}"
if e.offset:
msg += f"\n {' ' * (e.offset - 1)}^"
return [msg]
except (ValueError, UnicodeError) as e:
return [f"Code contains invalid characters: {e}"]
v = _ASTValidator()
v.visit(tree)
return v.errors
def _math_exec_in_process(code: str, result_queue: multiprocessing.Queue):
"""Execute code in a subprocess, put (status, output) in queue."""
import signal as _signal
import sys as _sys
from io import StringIO
_signal.signal(_signal.SIGTERM, _signal.SIG_DFL)
_signal.signal(_signal.SIGINT, _signal.SIG_DFL)
_sys.set_int_max_str_digits(100_000)
try:
captured = StringIO()
_sys.stdout = captured
def _safe_import(name, *args, **kwargs):
if name.split(".")[0] in _MATH_BLOCKED_MODULES:
raise ImportError(f"Import of '{name}' is blocked")
return original_import(name, *args, **kwargs)
original_import = (
__builtins__["__import__"]
if isinstance(__builtins__, dict)
else __builtins__.__import__
)
safe_builtins = (
{k: v for k, v in __builtins__.items() if k not in _MATH_BLOCKED_BUILTINS}
if isinstance(__builtins__, dict)
else {
k: getattr(__builtins__, k)
for k in dir(__builtins__)
if k not in _MATH_BLOCKED_BUILTINS and not k.startswith("_")
}
)
safe_builtins["__import__"] = _safe_import
# Pre-import safe modules
import math, fractions, itertools, functools, operator
import collections, decimal, random, re, string
ns: dict = {
"__builtins__": safe_builtins,
"math": math,
"fractions": fractions,
"Fraction": fractions.Fraction,
"itertools": itertools,
"functools": functools,
"operator": operator,
"collections": collections,
"decimal": decimal,
"Decimal": decimal.Decimal,
"random": random,
"re": re,
"string": string,
}
try:
import sympy
ns["sympy"] = sympy
for name in (
"symbols",
"Symbol",
"solve",
"simplify",
"expand",
"factor",
"Eq",
"sqrt",
"Rational",
"pi",
"E",
"I",
"oo",
"sin",
"cos",
"tan",
"exp",
"log",
"factorial",
"binomial",
"gcd",
"lcm",
"prime",
"isprime",
"factorint",
"divisors",
"totient",
"mod_inverse",
"Matrix",
"integrate",
"diff",
"limit",
"series",
"Sum",
"Product",
"floor",
"ceiling",
"Abs",
):
ns[name] = getattr(sympy, name)
except ImportError:
pass
try:
import numpy as _np
ns["np"] = ns["numpy"] = _np
except ImportError:
pass
try:
import scipy, scipy.special, scipy.optimize, scipy.integrate, scipy.linalg
ns["scipy"] = scipy
ns["special"] = scipy.special
ns["optimize"] = scipy.optimize
ns["comb"] = scipy.special.comb
ns["perm"] = scipy.special.perm
ns["gamma"] = scipy.special.gamma
ns["beta"] = scipy.special.beta
except ImportError:
pass
exec(code, ns)
_sys.stdout = _sys.__stdout__
printed = captured.getvalue()
result_var = ns.get("result")
if result_var is not None:
out = (
f"{printed.rstrip()}\nresult = {result_var}"
if printed
else str(result_var)
)
elif printed:
out = printed.rstrip()
else:
out = "No output. Add print() to see results."
result_queue.put(("success", out))
except Exception as e:
_sys.stdout = _sys.__stdout__
result_queue.put(
("error", f"{type(e).__name__}: {e}\n{traceback.format_exc()}")
)
def auto_print_wrap(code: str) -> str:
"""If code has no print/result and the last statement is an expression, wrap it in print()."""
# Skip if code already has print() or assigns to 'result'
if "print(" in code or re.search(r"\bresult\s*=", code):
return code
try:
tree = ast.parse(code)
except SyntaxError:
return code
if not tree.body:
return code
last = tree.body[-1]
if isinstance(last, ast.Expr):
# Get the source of the last expression and wrap in print()
lines = code.split("\n")
last_line_start = last.lineno - 1 # 0-based
last_line_end = last.end_lineno # 1-based, exclusive after slicing
expr_lines = lines[last_line_start:last_line_end]
expr_text = "\n".join(expr_lines)
prefix = lines[:last_line_start]
wrapped = prefix + [f"print({expr_text})"]
return "\n".join(wrapped)
return code
def execute_math_sandboxed(code: str, timeout: float = 30.0) -> tuple[str, bool]:
"""Execute Python code in a sandboxed subprocess. Returns (output, is_error)."""
code = auto_print_wrap(code)
errors = validate_math_code(code)
if errors:
return "Validation errors:\n" + "\n".join(f"- {e}" for e in errors), True
result_queue: multiprocessing.Queue = multiprocessing.Queue()
proc = multiprocessing.Process(
target=_math_exec_in_process, args=(code, result_queue)
)
proc.start()
proc.join(timeout=timeout)
if proc.is_alive():
proc.terminate()
proc.join(timeout=1.0)
if proc.is_alive():
proc.kill()
proc.join()
result_queue.close()
result_queue.join_thread()
return f"Execution timed out after {timeout}s", True
if result_queue.empty():
result_queue.close()
result_queue.join_thread()
return "Execution failed with no output", True
status, output = result_queue.get()
result_queue.close()
result_queue.join_thread()
return output, status == "error"
turnstone/core/session.py
+2743
View File
File diff suppressed because it is too large Load Diff
turnstone/core/tools.py
+36
View File
@@ -0,0 +1,36 @@
"""Tool definitions — auto-loaded from turnstone/tools/*.json."""
import json
from pathlib import Path
_TOOLS_DIR = Path(__file__).resolve().parent.parent / "tools"
_META_KEYS = {"agent", "task_agent", "auto_approve", "primary_key"}
def _load_tools() -> tuple[list[dict], dict]:
"""Load all .json files from the tools directory.
Returns (tool_defs, metadata) where:
- tool_defs: list of OpenAI function-calling dicts
- metadata: dict mapping tool_name -> {agent, task_agent, auto_approve, primary_key}
"""
tools = []
meta = {}
for path in sorted(_TOOLS_DIR.glob("*.json")):
with open(path) as f:
raw = json.load(f)
name = raw["name"]
# Extract turnstone metadata, leave only OpenAI schema fields
tool_meta = {k: raw.pop(k) for k in list(raw) if k in _META_KEYS}
meta[name] = tool_meta
tools.append({"type": "function", "function": raw})
return tools, meta
TOOLS, _META = _load_tools()
AGENT_TOOLS = [t for t in TOOLS if _META[t["function"]["name"]].get("agent")]
TASK_AGENT_TOOLS = [t for t in TOOLS if _META[t["function"]["name"]].get("task_agent")]
AGENT_AUTO_TOOLS = {n for n, m in _META.items() if m.get("auto_approve")}
TASK_AUTO_TOOLS = {n for n, m in _META.items() if m.get("auto_approve")}
PRIMARY_KEY_MAP = {n: m["primary_key"] for n, m in _META.items() if "primary_key" in m}
turnstone/core/web.py
+36
View File
@@ -0,0 +1,36 @@
"""Web utilities — HTML stripping and SSRF protection."""
import ipaddress
import re
import socket
from html import unescape as _html_unescape
from urllib.parse import urlparse
_RE_TAGS = re.compile(r"<[^>]+>")
_RE_WS = re.compile(r"[ \t]+")
_RE_BLANKLINES = re.compile(r"\n{3,}")
def strip_html(html: str) -> str:
"""Convert HTML to plain text: strip tags, decode entities, collapse whitespace."""
text = _RE_TAGS.sub("", html)
text = _html_unescape(text)
text = _RE_WS.sub(" ", text)
text = _RE_BLANKLINES.sub("\n\n", text)
return text.strip()
def check_ssrf(url: str) -> str | None:
"""Return error string if URL resolves to a private/link-local address, else None."""
try:
parsed = urlparse(url)
hostname = parsed.hostname
if not hostname:
return "Invalid URL: no hostname"
addr = socket.gethostbyname(hostname)
ip = ipaddress.ip_address(addr)
if ip.is_private or ip.is_loopback or ip.is_link_local:
return f"Blocked: URL resolves to private/internal address ({addr})"
except (socket.gaierror, ValueError):
pass # DNS failure or invalid IP — let the actual fetch handle it
return None
turnstone/core/workstream.py
+225
View File
@@ -0,0 +1,225 @@
"""Workstream manager — concurrent independent chat sessions.
A workstream is an independent conversation with its own ChatSession and UI
adapter. The WorkstreamManager coordinates multiple workstreams, tracks their
states, and lets frontends (CLI, Web) multiplex user attention across them.
"""
from __future__ import annotations
import enum
import threading
import time
import uuid
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Callable
if TYPE_CHECKING:
from turnstone.core.session import ChatSession, SessionUI
# ---------------------------------------------------------------------------
# State enum
# ---------------------------------------------------------------------------
class WorkstreamState(enum.Enum):
IDLE = "idle" # waiting for user input
THINKING = "thinking" # LLM is streaming
RUNNING = "running" # tools executing
ATTENTION = "attention" # blocked on approval / plan review
ERROR = "error" # last operation failed
# ---------------------------------------------------------------------------
# Workstream dataclass
# ---------------------------------------------------------------------------
@dataclass
class Workstream:
id: str = field(default_factory=lambda: uuid.uuid4().hex[:8])
name: str = ""
state: WorkstreamState = WorkstreamState.IDLE
session: ChatSession | None = None
ui: SessionUI | None = None
worker_thread: threading.Thread | None = None
error_message: str = ""
last_active: float = field(default_factory=time.monotonic, repr=False)
_lock: threading.Lock = field(default_factory=threading.Lock, repr=False)
def __post_init__(self):
if not self.name:
self.name = f"ws-{self.id[:4]}"
# ---------------------------------------------------------------------------
# Manager
# ---------------------------------------------------------------------------
class WorkstreamManager:
"""Manages multiple concurrent workstreams, each with its own ChatSession."""
MAX_WORKSTREAMS = 10
def __init__(
self,
session_factory: Callable[[SessionUI], ChatSession],
):
"""
Args:
session_factory: callable(ui) -> ChatSession. Captures shared
config (client, model, temperature, …) so the manager can
create sessions without knowing those details.
"""
self._session_factory = session_factory
self._workstreams: dict[str, Workstream] = {}
self._order: list[str] = [] # creation order
self._active_id: str | None = None
self._lock = threading.Lock()
self._on_state_change: Callable[[str, WorkstreamState], None] | None = None
# -- creation / destruction ---------------------------------------------
def create(
self,
name: str = "",
ui_factory: Callable[..., SessionUI] | None = None,
) -> Workstream:
"""Create a new workstream. Returns the new ws."""
ws = Workstream(name=name)
if ui_factory:
ws.ui = ui_factory(ws.id)
ws.session = self._session_factory(ws.ui)
with self._lock:
if len(self._workstreams) >= self.MAX_WORKSTREAMS:
raise RuntimeError(
f"Maximum of {self.MAX_WORKSTREAMS} workstreams reached"
)
self._workstreams[ws.id] = ws
self._order.append(ws.id)
if self._active_id is None:
self._active_id = ws.id
return ws
def close(self, ws_id: str) -> bool:
"""Close a workstream. Returns False if it's the last one."""
with self._lock:
if len(self._workstreams) <= 1:
return False
ws = self._workstreams.pop(ws_id, None)
if ws is None:
return False
self._order.remove(ws_id)
if self._active_id == ws_id:
self._active_id = self._order[0]
# Unblock any waiting approval/plan events so worker thread can exit
if ws.ui:
if hasattr(ws.ui, "_approval_event"):
ws.ui._approval_result = (False, None)
ws.ui._approval_event.set()
if hasattr(ws.ui, "_plan_event"):
ws.ui._plan_result = "reject"
ws.ui._plan_event.set()
if hasattr(ws.ui, "_fg_event"):
ws.ui._fg_event.set()
return True
# -- lookup -------------------------------------------------------------
def get(self, ws_id: str) -> Workstream | None:
with self._lock:
return self._workstreams.get(ws_id)
@property
def active_id(self) -> str | None:
return self._active_id
def get_active(self) -> Workstream | None:
with self._lock:
return self._workstreams.get(self._active_id) if self._active_id else None
def list_all(self) -> list[Workstream]:
"""Return workstreams in creation order."""
with self._lock:
return [
self._workstreams[wid]
for wid in self._order
if wid in self._workstreams
]
def index_of(self, ws_id: str) -> int:
"""1-based index of a workstream, or 0 if not found."""
with self._lock:
try:
return self._order.index(ws_id) + 1
except ValueError:
return 0
@property
def count(self) -> int:
with self._lock:
return len(self._workstreams)
# -- switching ----------------------------------------------------------
def switch(self, ws_id: str) -> Workstream | None:
"""Switch active workstream. Returns new active or None."""
with self._lock:
if ws_id in self._workstreams:
self._active_id = ws_id
return self._workstreams[ws_id]
return None
def switch_by_index(self, index: int) -> Workstream | None:
"""Switch by 1-based index (creation order)."""
with self._lock:
if 1 <= index <= len(self._order):
ws_id = self._order[index - 1]
self._active_id = ws_id
return self._workstreams.get(ws_id)
return None
# -- state management ---------------------------------------------------
def set_state(self, ws_id: str, state: WorkstreamState, error_msg: str = ""):
"""Update a workstream's state. Called by UI adapters."""
ws = self._workstreams.get(ws_id)
if ws:
with ws._lock:
ws.state = state
ws.last_active = time.monotonic()
ws.error_message = error_msg
if self._on_state_change:
self._on_state_change(ws_id, state)
def close_idle(self, max_age_seconds: float) -> list[str]:
"""Close IDLE workstreams inactive for more than *max_age_seconds*.
Skips the last workstream and any workstream not in IDLE state.
Returns a list of closed ws_ids.
"""
now = time.monotonic()
with self._lock:
snapshot = list(self._workstreams.values())
expired = sorted(
[
ws
for ws in snapshot
if ws.state == WorkstreamState.IDLE
and (now - ws.last_active) > max_age_seconds
],
key=lambda ws: ws.last_active, # oldest first
)
# Never leave zero workstreams
max_closeable = max(0, len(snapshot) - 1)
to_close = [ws.id for ws in expired[:max_closeable]]
closed = []
for ws_id in to_close:
ws = self._workstreams.get(ws_id)
# Re-check state to guard against race between collection and close
if ws and ws.state == WorkstreamState.IDLE and self.close(ws_id):
closed.append(ws_id)
return closed
turnstone/eval.py
+1249
View File
File diff suppressed because it is too large Load Diff
turnstone/mq/__init__.py
+11
View File
@@ -0,0 +1,11 @@
"""Message queue integration for turnstone.
Provides a bridge service (turnstone-bridge) that connects message queues to the
turnstone-server HTTP API, and a client library for external systems to publish
commands and subscribe to progress.
"""
from turnstone.mq.broker import MessageBroker, RedisBroker
from turnstone.mq.client import TurnstoneClient, TurnResult
__all__ = ["MessageBroker", "RedisBroker", "TurnstoneClient", "TurnResult"]
turnstone/mq/bridge.py
+777
View File
@@ -0,0 +1,777 @@
"""Bridge service — connects message queues to turnstone-server's HTTP API.
The bridge (listener+speaker) reads commands from an inbound queue, drives
workstreams on the turnstone-server via HTTP, consumes SSE for progress, and
publishes events to outbound pub/sub channels.
Run as: ``turnstone-bridge --server-url http://localhost:8080``
"""
from __future__ import annotations
import json
import logging
import os
import socket
import threading
import time
import uuid
from collections.abc import Callable, Iterator
import httpx
from turnstone.mq.broker import RedisBroker
from turnstone.mq.protocol import (
AckEvent,
ApprovalRequestEvent,
ClusterStateEvent,
ContentEvent,
ErrorEvent,
HealthResponseEvent,
InboundMessage,
InfoEvent,
NodeListEvent,
OutboundEvent,
PlanReviewEvent,
ReasoningEvent,
StateChangeEvent,
StatusEvent,
StreamEndEvent,
ToolInfoEvent,
ToolResultEvent,
TurnCompleteEvent,
WorkstreamClosedEvent,
WorkstreamCreatedEvent,
WorkstreamListEvent,
WorkstreamRenameEvent,
)
log = logging.getLogger("turnstone.mq.bridge")
# Server's default safe tools (auto-approved without user confirmation)
DEFAULT_SAFE_TOOLS = frozenset(
["read_file", "search", "man", "remember", "recall", "forget"]
)
def _default_node_id() -> str:
"""Generate a default node_id: ``{hostname}_{4hex}``, or a UUID on failure."""
suffix = uuid.uuid4().hex[:4]
try:
host = socket.gethostname()
if host and host != "localhost":
return f"{host}_{suffix}"
except OSError:
pass
return uuid.uuid4().hex[:12]
class Bridge:
"""Connects a message broker to turnstone-server's HTTP API.
Threading model::
Main Thread: Inbound loop (BLPOP on broker)
Global SSE Thread: GET /api/events/global
Per-WS SSE Thread × N: GET /api/events?ws_id=X
"""
def __init__(
self,
server_url: str = "http://localhost:8080",
broker: RedisBroker | None = None,
approval_timeout: float = 300,
prefix: str = "turnstone",
node_id: str = "",
heartbeat_ttl: int = 60,
auth_token: str = "",
):
self._server_url = server_url.rstrip("/")
self._broker = broker or RedisBroker()
self._approval_timeout = approval_timeout
self._prefix = prefix
self._node_id = node_id or _default_node_id()
self._heartbeat_ttl = heartbeat_ttl
self._started_at = time.time()
self._auth_token = auth_token
# Shared httpx client for short-lived POST requests (main thread only)
headers: dict[str, str] = {}
if auth_token:
headers["Authorization"] = f"Bearer {auth_token}"
self._http = httpx.Client(
base_url=self._server_url, timeout=30, headers=headers
)
# Protected by _lock — accessed from main, global SSE, and per-ws SSE threads
self._lock = threading.Lock()
self._ws_threads: dict[str, threading.Thread] = {}
self._ws_auto_approve: dict[str, bool] = {}
self._ws_approve_tools: dict[str, set[str]] = {}
self._active_sends: dict[str, str] = {} # ws_id → correlation_id
self._running = True
# -- public entry point --------------------------------------------------
def run(self) -> None:
"""Block until shutdown (KeyboardInterrupt)."""
log.info("Bridge starting — node=%s server=%s", self._node_id, self._server_url)
self._recover_workstreams()
heartbeat_t = threading.Thread(target=self._heartbeat_loop, daemon=True)
heartbeat_t.start()
global_t = threading.Thread(target=self._global_sse_loop, daemon=True)
global_t.start()
try:
self._inbound_loop()
except KeyboardInterrupt:
log.info("Bridge shutting down")
finally:
self._running = False
self._http.close()
self._broker.close()
# -- recovery ------------------------------------------------------------
def _recover_workstreams(self) -> None:
"""Discover active workstreams on startup and register ownership."""
try:
resp = self._http.get("/api/workstreams")
data = resp.json()
for ws in data.get("workstreams", []):
ws_id = ws["id"]
log.info("Recovered workstream %s (%s)", ws_id, ws.get("name", ""))
self._broker.set_ws_owner(ws_id, self._node_id)
self._start_ws_sse(ws_id)
except Exception as exc:
log.warning("Could not recover workstreams: %s", exc)
# -- inbound loop --------------------------------------------------------
def _inbound_loop(self) -> None:
while self._running:
raw = self._broker.pop_inbound(timeout=5, node_id=self._node_id)
if raw is None:
continue
try:
msg = InboundMessage.from_json(raw)
self._dispatch(msg)
except Exception as exc:
log.error("Failed to process inbound message: %s", exc)
self._publish_global(
ErrorEvent(message=f"Failed to process message: {exc}")
)
def _dispatch(self, msg: InboundMessage) -> None:
# Messages that need routing (have ws_id or target_node)
routed_handlers = {
"send": self._handle_send,
"command": self._handle_command,
"create_workstream": self._handle_create_ws,
"close_workstream": self._handle_close_ws,
}
# Messages that are always local (no routing needed)
local_handlers = {
"approve": self._handle_approve,
"plan_feedback": self._handle_plan_feedback,
"list_workstreams": self._handle_list_ws,
"health": self._handle_health,
"list_nodes": self._handle_list_nodes,
}
if msg.type in routed_handlers:
self._route_or_process(msg, routed_handlers[msg.type])
elif msg.type in local_handlers:
local_handlers[msg.type](msg)
else:
self._publish_global(
ErrorEvent(
correlation_id=msg.correlation_id,
message=f"Unknown message type: {msg.type!r}",
)
)
def _route_or_process(
self, msg: InboundMessage, handler: Callable[[InboundMessage], None]
) -> None:
"""Route a message to the correct node, or process locally."""
target = getattr(msg, "target_node", "")
ws_id = getattr(msg, "ws_id", "")
# Directed to a different node?
if target and target != self._node_id:
log.debug("Routing to node %s: %s", target, msg.type)
self._broker.push_inbound(msg.to_json(), node_id=target)
return
# Existing workstream owned by another node?
if ws_id:
owner = self._broker.get_ws_owner(ws_id)
if owner and owner != self._node_id:
log.debug("Re-routing to owner %s for ws %s", owner, ws_id)
self._broker.push_inbound(msg.to_json(), node_id=owner)
return
handler(msg)
# -- handlers ------------------------------------------------------------
def _handle_send(self, msg: InboundMessage) -> None:
ws_id = getattr(msg, "ws_id", "")
message = getattr(msg, "message", "")
auto_approve = getattr(msg, "auto_approve", False)
auto_approve_tools = getattr(msg, "auto_approve_tools", [])
name = getattr(msg, "name", "")
# Auto-create workstream if needed
if not ws_id:
ws_id = self._create_ws_on_server(
name=name,
auto_approve=auto_approve,
auto_approve_tools=auto_approve_tools,
correlation_id=msg.correlation_id,
)
if not ws_id:
return # error already published
else:
# Update approval settings for existing workstream
with self._lock:
if auto_approve:
self._ws_auto_approve[ws_id] = True
if auto_approve_tools:
self._ws_approve_tools[ws_id] = set(auto_approve_tools)
with self._lock:
self._active_sends[ws_id] = msg.correlation_id
resp = self._http.post("/api/send", json={"message": message, "ws_id": ws_id})
data = resp.json()
self._publish_ws(
ws_id,
AckEvent(
ws_id=ws_id,
correlation_id=msg.correlation_id,
status="ok" if data.get("status") == "ok" else "error",
detail=data.get("error", ""),
),
)
def _handle_approve(self, msg: InboundMessage) -> None:
request_id = getattr(msg, "request_id", "")
if request_id:
self._broker.push_response(request_id, msg.to_json())
def _handle_plan_feedback(self, msg: InboundMessage) -> None:
request_id = getattr(msg, "request_id", "")
if request_id:
self._broker.push_response(request_id, msg.to_json())
def _handle_command(self, msg: InboundMessage) -> None:
ws_id = getattr(msg, "ws_id", "")
command = getattr(msg, "command", "")
resp = self._http.post(
"/api/command", json={"command": command, "ws_id": ws_id}
)
data = resp.json()
self._publish_ws(
ws_id,
AckEvent(
ws_id=ws_id,
correlation_id=msg.correlation_id,
status="ok" if data.get("status") == "ok" else "error",
detail=data.get("error", ""),
),
)
def _handle_create_ws(self, msg: InboundMessage) -> None:
name = getattr(msg, "name", "")
auto_approve = getattr(msg, "auto_approve", False)
auto_approve_tools = getattr(msg, "auto_approve_tools", [])
self._create_ws_on_server(
name=name,
auto_approve=auto_approve,
auto_approve_tools=auto_approve_tools,
correlation_id=msg.correlation_id,
)
def _handle_close_ws(self, msg: InboundMessage) -> None:
ws_id = getattr(msg, "ws_id", "")
resp = self._http.post("/api/workstreams/close", json={"ws_id": ws_id})
data = resp.json()
self._publish_ws(
ws_id,
AckEvent(
ws_id=ws_id,
correlation_id=msg.correlation_id,
status="ok" if data.get("status") == "ok" else "error",
detail=data.get("error", ""),
),
)
def _handle_list_ws(self, msg: InboundMessage) -> None:
resp = self._http.get("/api/workstreams")
data = resp.json()
self._publish_global(
WorkstreamListEvent(
correlation_id=msg.correlation_id,
workstreams=data.get("workstreams", []),
)
)
def _handle_health(self, msg: InboundMessage) -> None:
resp = self._http.get("/health")
data = resp.json()
self._publish_global(
HealthResponseEvent(
correlation_id=msg.correlation_id,
data=data,
)
)
# -- workstream creation helper ------------------------------------------
def _create_ws_on_server(
self,
name: str,
auto_approve: bool,
auto_approve_tools: list[str],
correlation_id: str,
) -> str:
"""Create a workstream on the server. Returns ws_id or empty on error."""
try:
resp = self._http.post(
"/api/workstreams/new",
json={"name": name, "auto_approve": auto_approve},
)
data = resp.json()
if "error" in data:
self._publish_global(
AckEvent(
correlation_id=correlation_id,
status="error",
detail=data["error"],
)
)
return ""
ws_id = data["ws_id"]
ws_name = data.get("name", "")
self._broker.set_ws_owner(ws_id, self._node_id)
with self._lock:
if auto_approve:
self._ws_auto_approve[ws_id] = True
if auto_approve_tools:
self._ws_approve_tools[ws_id] = set(auto_approve_tools)
self._start_ws_sse(ws_id)
self._publish_global(
WorkstreamCreatedEvent(
ws_id=ws_id,
name=ws_name,
correlation_id=correlation_id,
)
)
self._publish_cluster(
WorkstreamCreatedEvent(
ws_id=ws_id,
name=ws_name,
correlation_id=correlation_id,
)
)
return ws_id
except Exception as exc:
self._publish_global(
AckEvent(
correlation_id=correlation_id,
status="error",
detail=str(exc),
)
)
return ""
# -- SSE consumption -----------------------------------------------------
def _start_ws_sse(self, ws_id: str) -> None:
with self._lock:
if ws_id in self._ws_threads and self._ws_threads[ws_id].is_alive():
return
t = threading.Thread(target=self._ws_sse_loop, args=(ws_id,), daemon=True)
self._ws_threads[ws_id] = t
t.start()
def _ws_sse_loop(self, ws_id: str) -> None:
"""Consume per-workstream SSE and forward events."""
# Each SSE thread gets its own httpx client (not thread-safe to share)
sse_headers: dict[str, str] = {}
if self._auth_token:
sse_headers["Authorization"] = f"Bearer {self._auth_token}"
with httpx.Client(
base_url=self._server_url, timeout=None, headers=sse_headers
) as sse_client:
while self._running:
try:
with sse_client.stream("GET", f"/api/events?ws_id={ws_id}") as resp:
for data in _iter_sse_data(resp):
if not self._running:
break
self._handle_ws_event(ws_id, data)
except Exception as exc:
if self._running:
log.debug("WS SSE reconnecting (%s): %s", ws_id, exc)
time.sleep(2)
def _handle_ws_event(self, ws_id: str, data: dict) -> None:
etype = data.get("type", "")
if etype == "content":
self._publish_ws(
ws_id, ContentEvent(ws_id=ws_id, text=data.get("text", ""))
)
elif etype == "reasoning":
self._publish_ws(
ws_id, ReasoningEvent(ws_id=ws_id, text=data.get("text", ""))
)
elif etype == "tool_info":
self._publish_ws(
ws_id, ToolInfoEvent(ws_id=ws_id, items=data.get("items", []))
)
elif etype == "approve_request":
self._handle_approval(ws_id, data)
elif etype == "plan_review":
self._handle_plan_review(ws_id, data)
elif etype == "tool_result":
self._publish_ws(
ws_id,
ToolResultEvent(
ws_id=ws_id,
name=data.get("name", ""),
output=data.get("output", ""),
),
)
elif etype == "status":
self._publish_ws(
ws_id,
StatusEvent(
ws_id=ws_id,
prompt_tokens=data.get("prompt_tokens", 0),
completion_tokens=data.get("completion_tokens", 0),
total_tokens=data.get("total_tokens", 0),
context_window=data.get("context_window", 0),
pct=data.get("pct", 0),
effort=data.get("effort", ""),
),
)
elif etype == "error":
self._publish_ws(
ws_id, ErrorEvent(ws_id=ws_id, message=data.get("message", ""))
)
elif etype == "info":
self._publish_ws(
ws_id, InfoEvent(ws_id=ws_id, message=data.get("message", ""))
)
elif etype == "stream_end":
self._publish_ws(ws_id, StreamEndEvent(ws_id=ws_id))
def _handle_approval(self, ws_id: str, data: dict) -> None:
"""Handle an approval request — auto-approve or forward to client."""
items = data.get("items", [])
# Check if all tools can be auto-approved
with self._lock:
if self._ws_auto_approve.get(ws_id):
self._api_approve(ws_id, approved=True)
return
approve_set = self._ws_approve_tools.get(ws_id, DEFAULT_SAFE_TOOLS)
tool_names = {it.get("name", "") for it in items if it.get("needs_approval")}
if tool_names and tool_names.issubset(approve_set):
self._api_approve(ws_id, approved=True)
return
# Forward to client — spawn a thread so we don't block SSE consumption
request_id = uuid.uuid4().hex[:12]
self._publish_ws(
ws_id,
ApprovalRequestEvent(
ws_id=ws_id,
correlation_id=request_id,
items=items,
),
)
def _wait_approval() -> None:
raw_resp = self._broker.pop_response(
request_id, timeout=self._approval_timeout
)
if raw_resp:
resp_msg = InboundMessage.from_json(raw_resp)
approved = getattr(resp_msg, "approved", False)
feedback = getattr(resp_msg, "feedback", None)
always = getattr(resp_msg, "always", False)
self._api_approve(ws_id, approved=approved, feedback=feedback)
if always:
with self._lock:
self._ws_auto_approve[ws_id] = True
else:
log.warning("Approval timeout for ws %s — denying", ws_id)
self._api_approve(ws_id, approved=False, feedback="Approval timed out")
threading.Thread(target=_wait_approval, daemon=True).start()
def _handle_plan_review(self, ws_id: str, data: dict) -> None:
"""Handle a plan review request — auto-approve or forward to client."""
with self._lock:
if self._ws_auto_approve.get(ws_id):
self._http.post("/api/plan", json={"feedback": "", "ws_id": ws_id})
return
request_id = uuid.uuid4().hex[:12]
self._publish_ws(
ws_id,
PlanReviewEvent(
ws_id=ws_id,
correlation_id=request_id,
content=data.get("content", ""),
),
)
def _wait_plan() -> None:
raw_resp = self._broker.pop_response(
request_id, timeout=self._approval_timeout
)
if raw_resp:
resp_msg = InboundMessage.from_json(raw_resp)
feedback = getattr(resp_msg, "feedback", "")
self._http.post(
"/api/plan", json={"feedback": feedback, "ws_id": ws_id}
)
else:
log.warning("Plan review timeout for ws %s — rejecting", ws_id)
self._http.post(
"/api/plan", json={"feedback": "reject", "ws_id": ws_id}
)
threading.Thread(target=_wait_plan, daemon=True).start()
def _api_approve(
self,
ws_id: str,
approved: bool,
feedback: str | None = None,
) -> None:
body: dict = {"approved": approved, "ws_id": ws_id}
if feedback:
body["feedback"] = feedback
self._http.post("/api/approve", json=body)
# -- global SSE ----------------------------------------------------------
def _global_sse_loop(self) -> None:
# Own httpx client for the long-lived SSE connection
sse_headers: dict[str, str] = {}
if self._auth_token:
sse_headers["Authorization"] = f"Bearer {self._auth_token}"
with httpx.Client(
base_url=self._server_url, timeout=None, headers=sse_headers
) as sse_client:
while self._running:
try:
with sse_client.stream("GET", "/api/events/global") as resp:
for data in _iter_sse_data(resp):
if not self._running:
break
self._handle_global_event(data)
except Exception as exc:
if self._running:
log.debug("Global SSE reconnecting: %s", exc)
time.sleep(2)
def _handle_global_event(self, data: dict) -> None:
etype = data.get("type", "")
ws_id = data.get("ws_id", "")
if etype == "ws_state":
state = data.get("state", "")
self._publish_ws(ws_id, StateChangeEvent(ws_id=ws_id, state=state))
self._publish_global(StateChangeEvent(ws_id=ws_id, state=state))
self._publish_cluster(
ClusterStateEvent(
ws_id=ws_id,
state=state,
node_id=self._node_id,
tokens=data.get("tokens", 0),
context_ratio=data.get("context_ratio", 0.0),
activity=data.get("activity", ""),
activity_state=data.get("activity_state", ""),
)
)
# Completion detection
if state == "idle":
with self._lock:
cid = self._active_sends.pop(ws_id, None)
if cid:
self._publish_ws(
ws_id, TurnCompleteEvent(ws_id=ws_id, correlation_id=cid)
)
elif etype == "ws_rename":
self._publish_global(
WorkstreamRenameEvent(ws_id=ws_id, name=data.get("name", ""))
)
self._publish_cluster(
WorkstreamRenameEvent(ws_id=ws_id, name=data.get("name", ""))
)
elif etype == "ws_closed":
self._publish_global(WorkstreamClosedEvent(ws_id=ws_id))
self._publish_cluster(WorkstreamClosedEvent(ws_id=ws_id))
self._broker.del_ws_owner(ws_id)
with self._lock:
self._ws_threads.pop(ws_id, None)
self._ws_auto_approve.pop(ws_id, None)
self._ws_approve_tools.pop(ws_id, None)
self._active_sends.pop(ws_id, None)
# -- heartbeat -----------------------------------------------------------
def _heartbeat_loop(self) -> None:
"""Periodically register this node in the broker."""
while self._running:
self._broker.register_node(
self._node_id,
{"server_url": self._server_url, "started": self._started_at},
ttl=self._heartbeat_ttl,
)
time.sleep(self._heartbeat_ttl / 2)
# -- node listing --------------------------------------------------------
def _handle_list_nodes(self, msg: InboundMessage) -> None:
nodes = self._broker.list_nodes()
self._publish_global(
NodeListEvent(correlation_id=msg.correlation_id, nodes=nodes)
)
# -- publish helpers -----------------------------------------------------
def _publish_ws(self, ws_id: str, event: OutboundEvent) -> None:
channel = f"{self._prefix}:events:{ws_id}"
self._broker.publish_outbound(channel, event.to_json())
def _publish_global(self, event: OutboundEvent) -> None:
self._broker.publish_outbound(f"{self._prefix}:events:global", event.to_json())
def _publish_cluster(self, event: OutboundEvent) -> None:
self._broker.publish_outbound(f"{self._prefix}:events:cluster", event.to_json())
# ---------------------------------------------------------------------------
# SSE parsing helper
# ---------------------------------------------------------------------------
def _iter_sse_data(resp: httpx.Response) -> Iterator[dict]:
"""Yield parsed JSON dicts from an SSE stream."""
for line in resp.iter_lines():
if line.startswith("data: "):
try:
yield json.loads(line[6:])
except json.JSONDecodeError:
pass
# SSE keepalive comments (lines starting with ':') are ignored
# ---------------------------------------------------------------------------
# CLI entry point
# ---------------------------------------------------------------------------
def main() -> None:
import argparse
parser = argparse.ArgumentParser(
description="turnstone message queue bridge — connects Redis queues to turnstone-server"
)
parser.add_argument(
"--server-url",
default="http://localhost:8080",
help="turnstone-server URL (default: %(default)s)",
)
parser.add_argument(
"--redis-host", default="localhost", help="Redis host (default: %(default)s)"
)
parser.add_argument(
"--redis-port", type=int, default=6379, help="Redis port (default: %(default)s)"
)
parser.add_argument(
"--redis-password",
default=os.environ.get("REDIS_PASSWORD"),
help="Redis password (default: $REDIS_PASSWORD)",
)
parser.add_argument(
"--redis-db", type=int, default=0, help="Redis DB number (default: %(default)s)"
)
parser.add_argument(
"--approval-timeout",
type=float,
default=300,
help="Seconds to wait for approval responses (default: %(default)s)",
)
parser.add_argument(
"--node-id",
default="",
help="Node identifier for multi-node routing (default: hostname)",
)
parser.add_argument(
"--heartbeat-ttl",
type=int,
default=60,
help="Heartbeat TTL in seconds (default: %(default)s)",
)
parser.add_argument(
"--log-level",
default="INFO",
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
help="Log level (default: %(default)s)",
)
parser.add_argument(
"--auth-token",
default=os.environ.get("TURNSTONE_AUTH_TOKEN", ""),
help="Bearer token for authenticating to turnstone-server (default: $TURNSTONE_AUTH_TOKEN)",
)
from turnstone.core.config import apply_config
apply_config(parser, ["bridge", "redis", "auth"])
args = parser.parse_args()
logging.basicConfig(
level=getattr(logging, args.log_level),
format="%(asctime)s %(name)s %(levelname)s %(message)s",
)
broker = RedisBroker(
host=args.redis_host,
port=args.redis_port,
db=args.redis_db,
password=args.redis_password,
)
bridge = Bridge(
server_url=args.server_url,
broker=broker,
approval_timeout=args.approval_timeout,
node_id=args.node_id,
heartbeat_ttl=args.heartbeat_ttl,
auth_token=args.auth_token,
)
bridge.run()
if __name__ == "__main__":
main()
turnstone/mq/broker.py
+237
View File
@@ -0,0 +1,237 @@
"""Abstract message broker and Redis implementation.
The MessageBroker protocol defines the interface for inbound queuing, outbound
pub/sub, per-request response queues, and multi-node routing primitives.
RedisBroker is the default provider.
"""
from __future__ import annotations
import json
import threading
from typing import Callable, Protocol
class MessageBroker(Protocol):
"""Abstract message broker for inbound/outbound communication.
Implementations must provide:
- Reliable inbound queue (FIFO, at-least-once delivery)
- Outbound pub/sub channels (fan-out to all subscribers)
- Per-request response queues for approval request/response correlation
- Workstream ownership tracking (ws_id → node_id)
- Node registry with heartbeat
"""
def push_inbound(self, message: str, node_id: str = "") -> None:
"""Push a message onto the inbound queue.
If *node_id* is set, pushes to the per-node queue for directed
routing. Otherwise pushes to the shared queue.
"""
...
def pop_inbound(self, timeout: float = 0, node_id: str = "") -> str | None:
"""Pop next message from the inbound queue (bridge side).
If *node_id* is set, BLPOPs from both the per-node queue (priority)
and the shared queue. Otherwise BLPOPs from the shared queue only.
Returns None on timeout.
"""
...
def publish_outbound(self, channel: str, event: str) -> None:
"""Publish an event to an outbound channel."""
...
def subscribe_outbound(self, channel: str, callback: Callable[[str], None]) -> None:
"""Subscribe to an outbound channel."""
...
def unsubscribe_outbound(self, channel: str) -> None:
"""Unsubscribe from an outbound channel."""
...
def push_response(self, queue_name: str, message: str) -> None:
"""Push a response onto a named response queue."""
...
def pop_response(self, queue_name: str, timeout: float = 300) -> str | None:
"""Pop from a named response queue. Returns None on timeout."""
...
# -- routing primitives --------------------------------------------------
def set_ws_owner(self, ws_id: str, node_id: str, ttl: int = 0) -> None:
"""Register which node owns a workstream."""
...
def get_ws_owner(self, ws_id: str) -> str | None:
"""Look up the node that owns a workstream. Returns None if unowned."""
...
def del_ws_owner(self, ws_id: str) -> None:
"""Remove workstream ownership (on close)."""
...
def register_node(self, node_id: str, metadata: dict, ttl: int = 60) -> None:
"""Register or refresh a node's heartbeat with metadata."""
...
def list_nodes(self) -> list[dict]:
"""List all active nodes (those with unexpired heartbeats)."""
...
def subscribe_cluster(self, callback: Callable[[str], None]) -> None:
"""Subscribe to the cluster-wide event channel."""
...
def close(self) -> None:
"""Clean up connections."""
...
class RedisBroker:
"""Redis-backed MessageBroker using lists (queues) and pub/sub (events).
Queue keys:
``{prefix}:inbound`` — shared inbound command queue
``{prefix}:inbound:{node_id}`` — per-node directed queue
``{prefix}:resp:{request_id}`` — per-request response queues
Routing keys:
``{prefix}:ws:{ws_id}`` — workstream ownership (string)
``{prefix}:node:{node_id}`` — node heartbeat + metadata (string/JSON)
Pub/sub channels:
``{prefix}:events:global`` — global event channel
``{prefix}:events:{ws_id}`` — per-workstream event channel
``{prefix}:events:cluster`` — cluster-wide state changes
"""
def __init__(
self,
host: str = "localhost",
port: int = 6379,
db: int = 0,
prefix: str = "turnstone",
password: str | None = None,
response_ttl: int = 600,
):
import redis
self._prefix = prefix
self._response_ttl = response_ttl
self._pool = redis.ConnectionPool(
host=host,
port=port,
db=db,
password=password,
decode_responses=True,
retry_on_timeout=True,
)
self._redis = redis.Redis(connection_pool=self._pool)
self._pubsub = self._redis.pubsub(ignore_subscribe_messages=True)
self._listener_thread: threading.Thread | None = None
self._running = True
# -- inbound queue -------------------------------------------------------
def push_inbound(self, message: str, node_id: str = "") -> None:
if node_id:
self._redis.rpush(f"{self._prefix}:inbound:{node_id}", message)
else:
self._redis.rpush(f"{self._prefix}:inbound", message)
def pop_inbound(self, timeout: float = 0, node_id: str = "") -> str | None:
t = int(timeout) if timeout > 0 else 0
if node_id:
# Per-node queue first (priority), then shared queue
result = self._redis.blpop(
[f"{self._prefix}:inbound:{node_id}", f"{self._prefix}:inbound"],
timeout=t,
)
else:
result = self._redis.blpop(f"{self._prefix}:inbound", timeout=t)
return result[1] if result else None
# -- outbound pub/sub ----------------------------------------------------
def publish_outbound(self, channel: str, event: str) -> None:
self._redis.publish(channel, event)
def subscribe_outbound(self, channel: str, callback: Callable[[str], None]) -> None:
self._pubsub.subscribe(**{channel: lambda msg: callback(msg["data"])})
if self._listener_thread is None or not self._listener_thread.is_alive():
self._listener_thread = self._pubsub.run_in_thread(
sleep_time=0.1, daemon=True
)
def unsubscribe_outbound(self, channel: str) -> None:
self._pubsub.unsubscribe(channel)
# -- response queues -----------------------------------------------------
def push_response(self, queue_name: str, message: str) -> None:
key = f"{self._prefix}:resp:{queue_name}"
self._redis.rpush(key, message)
self._redis.expire(key, self._response_ttl)
def pop_response(self, queue_name: str, timeout: float = 300) -> str | None:
key = f"{self._prefix}:resp:{queue_name}"
result = self._redis.blpop(key, timeout=int(timeout))
return result[1] if result else None
# -- routing primitives --------------------------------------------------
def set_ws_owner(self, ws_id: str, node_id: str, ttl: int = 0) -> None:
key = f"{self._prefix}:ws:{ws_id}"
if ttl > 0:
self._redis.set(key, node_id, ex=ttl)
else:
self._redis.set(key, node_id)
def get_ws_owner(self, ws_id: str) -> str | None:
return self._redis.get(f"{self._prefix}:ws:{ws_id}")
def del_ws_owner(self, ws_id: str) -> None:
self._redis.delete(f"{self._prefix}:ws:{ws_id}")
def register_node(self, node_id: str, metadata: dict, ttl: int = 60) -> None:
key = f"{self._prefix}:node:{node_id}"
self._redis.set(key, json.dumps(metadata), ex=ttl)
def list_nodes(self) -> list[dict]:
pattern = f"{self._prefix}:node:*"
prefix_len = len(f"{self._prefix}:node:")
nodes = []
for key in self._redis.scan_iter(match=pattern, count=100):
raw = self._redis.get(key)
if raw:
try:
meta = json.loads(raw)
except json.JSONDecodeError:
meta = {}
meta["node_id"] = key[prefix_len:]
nodes.append(meta)
return nodes
# -- cluster event channel -----------------------------------------------
def subscribe_cluster(self, callback: Callable[[str], None]) -> None:
"""Subscribe to the cluster-wide event channel."""
channel = f"{self._prefix}:events:cluster"
self.subscribe_outbound(channel, callback)
# -- lifecycle -----------------------------------------------------------
def close(self) -> None:
self._running = False
if self._listener_thread is not None:
self._listener_thread.stop()
self._listener_thread = None
try:
self._pubsub.close()
except Exception:
pass
self._pool.disconnect()
turnstone/mq/client.py
+322
View File
@@ -0,0 +1,322 @@
"""Client library for interacting with turnstone through a message broker.
Usage::
from turnstone.mq.client import TurnstoneClient
client = TurnstoneClient()
result = client.send_and_wait(
"What files are in the current directory?",
auto_approve=True,
)
print(result.content)
client.close()
"""
from __future__ import annotations
import threading
from dataclasses import dataclass, field
from typing import Callable
from turnstone.mq.broker import MessageBroker, RedisBroker
from turnstone.mq.protocol import (
ApproveMessage,
CloseWorkstreamMessage,
CommandMessage,
ContentEvent,
CreateWorkstreamMessage,
ErrorEvent,
HealthMessage,
ListWorkstreamsMessage,
OutboundEvent,
PlanFeedbackMessage,
ReasoningEvent,
SendMessage,
ToolResultEvent,
TurnCompleteEvent,
WorkstreamCreatedEvent,
)
@dataclass
class TurnResult:
"""Aggregated result of a send_and_wait call."""
correlation_id: str = ""
ws_id: str = ""
content_parts: list[str] = field(default_factory=list)
reasoning_parts: list[str] = field(default_factory=list)
tool_results: list[tuple[str, str]] = field(default_factory=list)
errors: list[str] = field(default_factory=list)
timed_out: bool = False
@property
def content(self) -> str:
return "".join(self.content_parts)
@property
def reasoning(self) -> str:
return "".join(self.reasoning_parts)
@property
def ok(self) -> bool:
return not self.timed_out and not self.errors
class TurnstoneClient:
"""Client library for turnstone message queue integration.
All methods are synchronous. The broker handles background threads
for pub/sub subscriptions.
"""
def __init__(
self,
broker: MessageBroker | None = None,
prefix: str = "turnstone",
**redis_kwargs: object,
):
"""Create a client.
Pass ``broker`` for a custom broker, or provide Redis kwargs
(``host``, ``port``, ``db``, ``password``) to use the default
RedisBroker.
"""
self._broker: MessageBroker = broker or RedisBroker(**redis_kwargs) # type: ignore[arg-type]
self._prefix = prefix
# -- fire-and-forget commands -------------------------------------------
def send(
self,
message: str,
ws_id: str = "",
name: str = "",
auto_approve: bool = False,
auto_approve_tools: list[str] | None = None,
target_node: str = "",
) -> str:
"""Send a message. Returns correlation_id for tracking.
If *target_node* is set, the message is pushed to that node's
dedicated queue. If *ws_id* is set and *target_node* is not,
the client looks up the workstream's owning node and routes
accordingly.
"""
msg = SendMessage(
message=message,
ws_id=ws_id,
name=name,
auto_approve=auto_approve,
auto_approve_tools=auto_approve_tools or [],
target_node=target_node,
)
node = target_node or (self._broker.get_ws_owner(ws_id) if ws_id else "")
self._broker.push_inbound(msg.to_json(), node_id=node)
return msg.correlation_id
def create_workstream(
self,
name: str = "",
auto_approve: bool = False,
auto_approve_tools: list[str] | None = None,
target_node: str = "",
) -> str:
"""Create a workstream. Returns correlation_id."""
msg = CreateWorkstreamMessage(
name=name,
auto_approve=auto_approve,
auto_approve_tools=auto_approve_tools or [],
target_node=target_node,
)
self._broker.push_inbound(msg.to_json(), node_id=target_node)
return msg.correlation_id
def close_workstream(self, ws_id: str) -> str:
"""Close a workstream. Returns correlation_id."""
msg = CloseWorkstreamMessage(ws_id=ws_id)
self._broker.push_inbound(msg.to_json())
return msg.correlation_id
def command(self, ws_id: str, command: str) -> str:
"""Execute a slash command. Returns correlation_id."""
msg = CommandMessage(ws_id=ws_id, command=command)
self._broker.push_inbound(msg.to_json())
return msg.correlation_id
def list_workstreams(self) -> str:
"""Request workstream list. Returns correlation_id."""
msg = ListWorkstreamsMessage()
self._broker.push_inbound(msg.to_json())
return msg.correlation_id
def health(self) -> str:
"""Request health status. Returns correlation_id."""
msg = HealthMessage()
self._broker.push_inbound(msg.to_json())
return msg.correlation_id
def list_nodes(self) -> list[dict]:
"""List active bridge nodes (reads directly from broker)."""
return self._broker.list_nodes()
# -- approval / plan response -------------------------------------------
def approve(
self,
request_id: str,
ws_id: str = "",
approved: bool = True,
feedback: str | None = None,
always: bool = False,
) -> None:
"""Respond to a tool approval request."""
msg = ApproveMessage(
ws_id=ws_id,
request_id=request_id,
approved=approved,
feedback=feedback,
always=always,
)
self._broker.push_response(request_id, msg.to_json())
def plan_feedback(
self,
request_id: str,
ws_id: str = "",
feedback: str = "",
) -> None:
"""Respond to a plan review request."""
msg = PlanFeedbackMessage(
ws_id=ws_id,
request_id=request_id,
feedback=feedback,
)
self._broker.push_response(request_id, msg.to_json())
# -- blocking send -------------------------------------------------------
def send_and_wait(
self,
message: str,
ws_id: str = "",
name: str = "",
auto_approve: bool = True,
auto_approve_tools: list[str] | None = None,
target_node: str = "",
timeout: float = 600,
on_event: Callable[[OutboundEvent], None] | None = None,
) -> TurnResult:
"""Send a message and block until the turn completes.
Returns a TurnResult with aggregated content, tool results, etc.
"""
# Build the message but don't send yet — subscribe first to avoid
# a race where the bridge processes the message before we subscribe.
msg = SendMessage(
message=message,
ws_id=ws_id,
name=name,
auto_approve=auto_approve,
auto_approve_tools=auto_approve_tools or [],
target_node=target_node,
)
cid = msg.correlation_id
result = TurnResult(correlation_id=cid, ws_id=ws_id)
done = threading.Event()
actual_ws_id = ws_id
def _on_global(raw: str) -> None:
nonlocal actual_ws_id
event = OutboundEvent.from_json(raw)
if on_event:
on_event(event)
if (
isinstance(event, WorkstreamCreatedEvent)
and event.correlation_id == cid
):
actual_ws_id = event.ws_id
result.ws_id = event.ws_id
self._broker.subscribe_outbound(
f"{self._prefix}:events:{actual_ws_id}", _on_ws
)
def _on_ws(raw: str) -> None:
event = OutboundEvent.from_json(raw)
if on_event:
on_event(event)
if isinstance(event, ContentEvent):
result.content_parts.append(event.text)
elif isinstance(event, ReasoningEvent):
result.reasoning_parts.append(event.text)
elif isinstance(event, ToolResultEvent):
result.tool_results.append((event.name, event.output))
elif isinstance(event, ErrorEvent):
result.errors.append(event.message)
elif isinstance(event, TurnCompleteEvent) and event.correlation_id == cid:
done.set()
# Subscribe BEFORE pushing — ensures we don't miss early events
self._broker.subscribe_outbound(f"{self._prefix}:events:global", _on_global)
if actual_ws_id:
self._broker.subscribe_outbound(
f"{self._prefix}:events:{actual_ws_id}", _on_ws
)
# Now push the message (route to target node or ws owner if known)
node = target_node or (self._broker.get_ws_owner(ws_id) if ws_id else "")
self._broker.push_inbound(msg.to_json(), node_id=node)
done.wait(timeout=timeout)
# Cleanup
self._broker.unsubscribe_outbound(f"{self._prefix}:events:global")
if actual_ws_id:
self._broker.unsubscribe_outbound(f"{self._prefix}:events:{actual_ws_id}")
result.ws_id = actual_ws_id
result.timed_out = not done.is_set()
return result
# -- subscription --------------------------------------------------------
def subscribe(
self,
callback: Callable[[OutboundEvent], None],
ws_id: str = "",
) -> None:
"""Subscribe to events for a specific workstream or global events."""
if ws_id:
channel = f"{self._prefix}:events:{ws_id}"
else:
channel = f"{self._prefix}:events:global"
def _cb(raw: str) -> None:
event = OutboundEvent.from_json(raw)
callback(event)
self._broker.subscribe_outbound(channel, _cb)
def unsubscribe(self, ws_id: str = "") -> None:
"""Unsubscribe from a workstream or global channel."""
if ws_id:
channel = f"{self._prefix}:events:{ws_id}"
else:
channel = f"{self._prefix}:events:global"
self._broker.unsubscribe_outbound(channel)
# -- lifecycle -----------------------------------------------------------
def close(self) -> None:
"""Clean up broker connection."""
self._broker.close()
def __enter__(self) -> TurnstoneClient:
return self
def __exit__(self, *exc: object) -> None:
self.close()
turnstone/mq/protocol.py
+378
View File
@@ -0,0 +1,378 @@
"""Message protocol for turnstone message queue integration.
Defines all structured message types exchanged between the client and bridge.
Inbound messages flow from client → bridge via a reliable queue.
Outbound events flow from bridge → client via pub/sub channels.
"""
from __future__ import annotations
import json
import time
import uuid
from dataclasses import asdict, dataclass, field
# ---------------------------------------------------------------------------
# Inbound messages (client → bridge)
# ---------------------------------------------------------------------------
@dataclass
class InboundMessage:
"""Base for all messages sent by clients to the bridge."""
type: str = ""
correlation_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
timestamp: float = field(default_factory=time.time)
def to_json(self) -> str:
return json.dumps(asdict(self))
@classmethod
def from_json(cls, raw: str) -> InboundMessage:
data = json.loads(raw)
msg_type = data.get("type", "")
klass = _INBOUND_REGISTRY.get(msg_type)
if klass is None:
raise ValueError(f"Unknown inbound message type: {msg_type!r}")
valid = {f for f in klass.__dataclass_fields__}
return klass(**{k: v for k, v in data.items() if k in valid})
@dataclass
class SendMessage(InboundMessage):
"""Send a user message to a workstream."""
type: str = "send"
ws_id: str = ""
message: str = ""
auto_approve: bool = False
auto_approve_tools: list[str] = field(default_factory=list)
name: str = ""
target_node: str = ""
@dataclass
class ApproveMessage(InboundMessage):
"""Respond to a tool approval request."""
type: str = "approve"
ws_id: str = ""
request_id: str = ""
approved: bool = True
feedback: str | None = None
always: bool = False
@dataclass
class PlanFeedbackMessage(InboundMessage):
"""Respond to a plan review request."""
type: str = "plan_feedback"
ws_id: str = ""
request_id: str = ""
feedback: str = ""
@dataclass
class CommandMessage(InboundMessage):
"""Execute a slash command."""
type: str = "command"
ws_id: str = ""
command: str = ""
@dataclass
class CreateWorkstreamMessage(InboundMessage):
"""Create a new workstream."""
type: str = "create_workstream"
name: str = ""
auto_approve: bool = False
auto_approve_tools: list[str] = field(default_factory=list)
target_node: str = ""
@dataclass
class CloseWorkstreamMessage(InboundMessage):
"""Close a workstream."""
type: str = "close_workstream"
ws_id: str = ""
@dataclass
class ListWorkstreamsMessage(InboundMessage):
"""Request the list of active workstreams."""
type: str = "list_workstreams"
@dataclass
class HealthMessage(InboundMessage):
"""Request health status."""
type: str = "health"
@dataclass
class ListNodesMessage(InboundMessage):
"""Request the list of active bridge nodes."""
type: str = "list_nodes"
# ---------------------------------------------------------------------------
# Outbound events (bridge → client)
# ---------------------------------------------------------------------------
@dataclass
class OutboundEvent:
"""Base for all events published by the bridge."""
type: str = ""
ws_id: str = ""
correlation_id: str = ""
timestamp: float = field(default_factory=time.time)
def to_json(self) -> str:
return json.dumps(asdict(self))
@classmethod
def from_json(cls, raw: str) -> OutboundEvent:
data = json.loads(raw)
msg_type = data.get("type", "")
klass = _OUTBOUND_REGISTRY.get(msg_type, OutboundEvent)
valid = {f for f in klass.__dataclass_fields__}
return klass(**{k: v for k, v in data.items() if k in valid})
@dataclass
class AckEvent(OutboundEvent):
"""Acknowledgment that an inbound message was received."""
type: str = "ack"
status: str = "ok"
detail: str = ""
@dataclass
class ContentEvent(OutboundEvent):
"""Streamed content token from the assistant."""
type: str = "content"
text: str = ""
@dataclass
class ReasoningEvent(OutboundEvent):
"""Streamed reasoning token."""
type: str = "reasoning"
text: str = ""
@dataclass
class ToolInfoEvent(OutboundEvent):
"""Tool call info (auto-approved tools)."""
type: str = "tool_info"
items: list = field(default_factory=list)
@dataclass
class ApprovalRequestEvent(OutboundEvent):
"""Tool approval request forwarded from the server.
The client must respond with an ApproveMessage whose
request_id matches this event's correlation_id.
"""
type: str = "approval_request"
items: list = field(default_factory=list)
@dataclass
class ToolResultEvent(OutboundEvent):
"""Tool execution result."""
type: str = "tool_result"
name: str = ""
output: str = ""
@dataclass
class PlanReviewEvent(OutboundEvent):
"""Plan review request forwarded from the server.
The client must respond with a PlanFeedbackMessage whose
request_id matches this event's correlation_id.
"""
type: str = "plan_review"
content: str = ""
@dataclass
class StatusEvent(OutboundEvent):
"""Token usage status update."""
type: str = "status"
prompt_tokens: int = 0
completion_tokens: int = 0
total_tokens: int = 0
context_window: int = 0
pct: float = 0.0
effort: str = ""
@dataclass
class StateChangeEvent(OutboundEvent):
"""Workstream state transition."""
type: str = "state_change"
state: str = ""
@dataclass
class TurnCompleteEvent(OutboundEvent):
"""Emitted when a workstream finishes processing (returns to IDLE).
This is a synthetic event produced by the bridge when it detects
the ws_state transition to 'idle' after a send.
"""
type: str = "turn_complete"
@dataclass
class StreamEndEvent(OutboundEvent):
"""LLM stream ended."""
type: str = "stream_end"
@dataclass
class WorkstreamCreatedEvent(OutboundEvent):
"""New workstream created."""
type: str = "ws_created"
name: str = ""
@dataclass
class WorkstreamClosedEvent(OutboundEvent):
"""Workstream closed."""
type: str = "ws_closed"
@dataclass
class WorkstreamListEvent(OutboundEvent):
"""Workstream list response."""
type: str = "ws_list"
workstreams: list = field(default_factory=list)
@dataclass
class WorkstreamRenameEvent(OutboundEvent):
"""Workstream renamed."""
type: str = "ws_rename"
name: str = ""
@dataclass
class HealthResponseEvent(OutboundEvent):
"""Health status response."""
type: str = "health_response"
data: dict = field(default_factory=dict)
@dataclass
class ErrorEvent(OutboundEvent):
"""Error event."""
type: str = "error"
message: str = ""
@dataclass
class InfoEvent(OutboundEvent):
"""Informational event."""
type: str = "info"
message: str = ""
@dataclass
class NodeListEvent(OutboundEvent):
"""List of active bridge nodes."""
type: str = "node_list"
nodes: list = field(default_factory=list)
@dataclass
class ClusterStateEvent(OutboundEvent):
"""Workstream state change with node attribution for cluster dashboard."""
type: str = "cluster_state"
ws_id: str = ""
state: str = ""
node_id: str = ""
tokens: int = 0
context_ratio: float = 0.0
activity: str = ""
activity_state: str = ""
# ---------------------------------------------------------------------------
# Type registries (built after all classes are defined)
# ---------------------------------------------------------------------------
_INBOUND_REGISTRY: dict[str, type[InboundMessage]] = {
cls.__dataclass_fields__["type"].default: cls
for cls in [
SendMessage,
ApproveMessage,
PlanFeedbackMessage,
CommandMessage,
CreateWorkstreamMessage,
CloseWorkstreamMessage,
ListWorkstreamsMessage,
HealthMessage,
ListNodesMessage,
]
}
_OUTBOUND_REGISTRY: dict[str, type[OutboundEvent]] = {
cls.__dataclass_fields__["type"].default: cls
for cls in [
AckEvent,
ContentEvent,
ReasoningEvent,
ToolInfoEvent,
ApprovalRequestEvent,
ToolResultEvent,
PlanReviewEvent,
StatusEvent,
StateChangeEvent,
TurnCompleteEvent,
StreamEndEvent,
WorkstreamCreatedEvent,
WorkstreamClosedEvent,
WorkstreamListEvent,
WorkstreamRenameEvent,
HealthResponseEvent,
ErrorEvent,
InfoEvent,
NodeListEvent,
ClusterStateEvent,
]
}
turnstone/server.py
+1172
View File
File diff suppressed because it is too large Load Diff
turnstone/sim/__init__.py
+6
View File
@@ -0,0 +1,6 @@
"""Turnstone cluster simulator."""
from turnstone.sim.cluster import SimCluster
from turnstone.sim.config import SimConfig
__all__ = ["SimCluster", "SimConfig"]
turnstone/sim/cli.py
+186
View File
@@ -0,0 +1,186 @@
"""CLI entry point for turnstone-sim."""
from __future__ import annotations
import argparse
import asyncio
import json
import logging
import sys
from turnstone.sim.cluster import SimCluster
from turnstone.sim.config import SimConfig
from turnstone.sim.scenario import SCENARIOS
log = logging.getLogger("turnstone.sim")
def main() -> None:
parser = argparse.ArgumentParser(
prog="turnstone-sim",
description="Turnstone multi-node cluster simulator",
)
parser.add_argument(
"--nodes",
type=int,
default=10,
help="Number of simulated nodes (default: 10)",
)
parser.add_argument(
"--scenario",
choices=list(SCENARIOS.keys()),
default="steady",
help="Scenario to run (default: steady)",
)
parser.add_argument(
"--duration",
type=int,
default=60,
help="Scenario duration in seconds (default: 60)",
)
parser.add_argument(
"--mps",
type=float,
default=5.0,
help="Messages per second for steady scenario (default: 5.0)",
)
parser.add_argument(
"--burst-size",
type=int,
default=100,
help="Message count for burst scenario (default: 100)",
)
parser.add_argument(
"--llm-latency",
type=float,
default=2.0,
help="Mean LLM response latency in seconds (default: 2.0)",
)
parser.add_argument(
"--tool-latency",
type=float,
default=0.5,
help="Mean tool execution latency in seconds (default: 0.5)",
)
parser.add_argument(
"--tool-failure-rate",
type=float,
default=0.02,
help="Tool failure probability 0.0-1.0 (default: 0.02)",
)
parser.add_argument(
"--node-kill-interval",
type=float,
default=15.0,
help="Seconds between node kills for node_failure scenario (default: 15)",
)
parser.add_argument(
"--node-kill-count",
type=int,
default=1,
help="Nodes to kill per interval (default: 1)",
)
parser.add_argument("--redis-host", default="localhost")
parser.add_argument("--redis-port", type=int, default=6379)
parser.add_argument("--redis-password", default=None)
parser.add_argument("--redis-db", type=int, default=0)
parser.add_argument("--prefix", default="turnstone")
parser.add_argument(
"--seed", type=int, default=None, help="Random seed for reproducibility"
)
parser.add_argument("--metrics-file", default="", help="Write JSON metrics to file")
parser.add_argument(
"--log-level",
default="INFO",
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
)
args = parser.parse_args()
config = SimConfig(
num_nodes=args.nodes,
scenario=args.scenario,
duration=args.duration,
messages_per_second=args.mps,
burst_size=args.burst_size,
llm_latency_mean=args.llm_latency,
tool_latency_mean=args.tool_latency,
tool_failure_rate=args.tool_failure_rate,
node_kill_interval=args.node_kill_interval,
node_kill_count=args.node_kill_count,
redis_host=args.redis_host,
redis_port=args.redis_port,
redis_password=args.redis_password,
redis_db=args.redis_db,
prefix=args.prefix,
seed=args.seed,
metrics_file=args.metrics_file,
)
logging.basicConfig(
level=getattr(logging, args.log_level),
format="%(asctime)s %(name)s %(levelname)s %(message)s",
)
try:
asyncio.run(_run(config))
except KeyboardInterrupt:
log.info("Interrupted")
sys.exit(0)
async def _run(config: SimConfig) -> None:
cluster = SimCluster(config)
try:
await cluster.start()
log.info(
"Running scenario=%s nodes=%d duration=%ds",
config.scenario,
config.num_nodes,
config.duration,
)
await cluster.run_scenario()
report = cluster.report()
_print_report(report, config)
if config.metrics_file:
with open(config.metrics_file, "w") as f:
json.dump(report, f, indent=2)
log.info("Metrics written to %s", config.metrics_file)
finally:
await cluster.stop()
def _print_report(report: dict, config: SimConfig) -> None:
lat = report.get("latency", {})
tp = report.get("throughput", {})
util = report.get("utilization", {})
print("\n" + "=" * 60)
print(" SIMULATION REPORT")
print("=" * 60)
print(f" Scenario: {config.scenario}")
print(f" Nodes: {config.num_nodes}")
print(f" Duration: {report['duration_seconds']}s")
print(f" Total turns: {report['total_turns']}")
print(f" Total errors: {report['total_errors']}")
print(f" Node kills: {report['node_kills']}")
print("-" * 60)
print(" THROUGHPUT")
print(f" Messages/sec: {tp.get('messages_per_sec', 0)}")
print(f" Turns/sec: {tp.get('turns_per_sec', 0)}")
print("-" * 60)
print(" LATENCY (seconds)")
print(f" p50: {lat.get('p50', 0)}")
print(f" p90: {lat.get('p90', 0)}")
print(f" p99: {lat.get('p99', 0)}")
print(f" mean: {lat.get('mean', 0)}")
print(f" max: {lat.get('max', 0)}")
if util:
print("-" * 60)
print(" UTILIZATION")
print(f" Mean ws/node: {util.get('mean_ws_per_node', 0):.1f}")
print(f" Max ws/node: {util.get('max_ws_per_node', 0)}")
print(f" Idle nodes: {util.get('nodes_with_zero_ws', 0)}")
print("=" * 60 + "\n")
turnstone/sim/cluster.py
+319
View File
@@ -0,0 +1,319 @@
"""Cluster orchestration — manages N SimNodes, dispatchers, and metrics."""
from __future__ import annotations
import asyncio
import logging
import math
import time
from concurrent.futures import ThreadPoolExecutor
import redis
from turnstone.mq.broker import RedisBroker
from turnstone.sim.config import SimConfig
from turnstone.sim.metrics import MetricsCollector
from turnstone.sim.node import SimNode
log = logging.getLogger("turnstone.sim.cluster")
# How many node queues a single dispatcher watches via one BLPOP call.
NODES_PER_DISPATCHER = 50
class PooledBroker(RedisBroker):
"""RedisBroker that uses a shared external ConnectionPool."""
def __init__(
self,
pool: redis.ConnectionPool,
prefix: str = "turnstone",
response_ttl: int = 600,
):
# Bypass RedisBroker.__init__ — set up manually with the shared pool.
import threading
self._prefix = prefix
self._response_ttl = response_ttl
self._pool = pool
self._redis = redis.Redis(connection_pool=pool)
self._pubsub = self._redis.pubsub(ignore_subscribe_messages=True)
self._listener_thread: threading.Thread | None = None
self._running = True
def close(self) -> None:
"""No-op — the shared pool is managed by SimCluster."""
self._running = False
class InboundDispatcher:
"""Watches batches of node queues via a single BLPOP call.
Instead of one BLPOP per node (which would exhaust Redis connections at
1000 nodes), a dispatcher batches ~50 node queues into a single BLPOP
on multiple keys. This keeps total Redis connections bounded.
"""
def __init__(
self,
redis_client: redis.Redis,
node_ids: list[str],
nodes: dict[str, SimNode],
prefix: str,
):
self._redis = redis_client
self._node_ids = node_ids
self._nodes = nodes
self._prefix = prefix
self._running = True
# Build BLPOP key list: per-node queues first (priority), shared last
self._keys = [f"{prefix}:inbound:{nid}" for nid in node_ids]
self._keys.append(f"{prefix}:inbound")
# Pre-compute key → node_id mapping
self._key_to_node: dict[str, str] = {
f"{prefix}:inbound:{nid}": nid for nid in node_ids
}
async def run(self) -> None:
while self._running:
# Snapshot keys to avoid race with remove_node() during BLPOP
keys = list(self._keys)
if not keys:
await asyncio.sleep(0.5)
continue
result = await asyncio.to_thread(
self._redis.blpop,
keys,
timeout=1,
)
if result is None:
continue
queue_key, raw = result
if isinstance(queue_key, bytes):
queue_key = queue_key.decode()
if isinstance(raw, bytes):
raw = raw.decode()
node = self._resolve_target(queue_key)
if node and node._running:
await node.handle_message(raw)
def _resolve_target(self, queue_key: str) -> SimNode | None:
"""Determine which SimNode should handle this message."""
node_id = self._key_to_node.get(queue_key)
if node_id:
return self._nodes.get(node_id)
# Shared queue — pick running node with fewest workstreams and capacity
if self._nodes:
candidates = [
n
for n in self._nodes.values()
if n._running and n.workstream_count < n._config.max_ws_per_node
]
if candidates:
return min(candidates, key=lambda n: n.workstream_count)
# Fall back to any running node if all at capacity
running = [n for n in self._nodes.values() if n._running]
if running:
return min(running, key=lambda n: n.workstream_count)
return None
def stop(self) -> None:
self._running = False
def remove_node(self, node_id: str) -> None:
"""Remove a node from this dispatcher (for kill simulation)."""
self._nodes.pop(node_id, None)
key = f"{self._prefix}:inbound:{node_id}"
self._key_to_node.pop(key, None)
if key in self._keys:
self._keys.remove(key)
class SimCluster:
"""Orchestrates N SimNodes, dispatchers, heartbeats, and metrics.
Usage::
cluster = SimCluster(config)
await cluster.start()
await cluster.run_scenario()
report = cluster.report()
await cluster.stop()
"""
def __init__(self, config: SimConfig):
self._config = config
self._metrics = MetricsCollector()
self._nodes: dict[str, SimNode] = {}
self._node_order: list[str] = []
self._dispatchers: list[InboundDispatcher] = []
self._tasks: list[asyncio.Task] = []
self._pool: redis.ConnectionPool | None = None
self._redis_client: redis.Redis | None = None
self._running = True
@property
def metrics(self) -> MetricsCollector:
return self._metrics
@property
def nodes(self) -> dict[str, SimNode]:
return self._nodes
@property
def config(self) -> SimConfig:
return self._config
async def start(self) -> None:
"""Create connection pool, nodes, dispatchers; start all tasks."""
self._executor = ThreadPoolExecutor(max_workers=64)
# Shared Redis pool
self._pool = redis.ConnectionPool(
host=self._config.redis_host,
port=self._config.redis_port,
db=self._config.redis_db,
password=self._config.redis_password,
decode_responses=True,
retry_on_timeout=True,
max_connections=64,
)
self._redis_client = redis.Redis(connection_pool=self._pool)
# Create nodes
for i in range(self._config.num_nodes):
node_id = f"sim-{i:04d}"
broker = PooledBroker(
self._pool,
prefix=self._config.prefix,
)
node = SimNode(node_id, broker, self._config, self._metrics)
self._nodes[node_id] = node
self._node_order.append(node_id)
# Create dispatchers (batches of NODES_PER_DISPATCHER)
all_ids = list(self._nodes.keys())
num_dispatchers = max(1, math.ceil(len(all_ids) / NODES_PER_DISPATCHER))
for i in range(num_dispatchers):
start = i * NODES_PER_DISPATCHER
batch_ids = all_ids[start : start + NODES_PER_DISPATCHER]
# Each dispatcher gets its own Redis client from the shared pool
client = redis.Redis(connection_pool=self._pool)
dispatcher = InboundDispatcher(
client,
batch_ids,
dict(self._nodes),
self._config.prefix,
)
self._dispatchers.append(dispatcher)
self._tasks.append(asyncio.create_task(dispatcher.run()))
# Start heartbeat task
self._tasks.append(asyncio.create_task(self._heartbeat_loop()))
# Start utilization snapshot task
self._tasks.append(asyncio.create_task(self._utilization_loop()))
# Wait for all nodes to register
await self._wait_for_nodes()
log.info(
"Cluster started: %d nodes, %d dispatchers",
len(self._nodes),
len(self._dispatchers),
)
async def _heartbeat_loop(self) -> None:
"""Register heartbeats for all running nodes concurrently."""
interval = max(1, self._config.heartbeat_ttl // 2)
loop = asyncio.get_running_loop()
while self._running:
tasks = [
loop.run_in_executor(self._executor, node.heartbeat_once)
for node in self._nodes.values()
if node._running
]
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
await asyncio.sleep(interval)
async def _utilization_loop(self) -> None:
"""Periodically snapshot workstream utilization."""
while self._running:
await asyncio.sleep(self._config.metrics_interval)
counts = {
nid: node.workstream_count
for nid, node in self._nodes.items()
if node._running
}
self._metrics.snapshot_utilization(counts)
async def _wait_for_nodes(self) -> None:
"""Do an initial heartbeat and confirm registration."""
loop = asyncio.get_running_loop()
tasks = [
loop.run_in_executor(self._executor, node.heartbeat_once)
for node in self._nodes.values()
]
await asyncio.gather(*tasks, return_exceptions=True)
registered = 0
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
keys = await loop.run_in_executor(
self._executor,
self._redis_client.keys,
f"{self._config.prefix}:node:sim-*",
)
registered = len(keys)
if registered >= self._config.num_nodes:
return
await asyncio.sleep(0.5)
raise TimeoutError(
f"Only {registered}/{self._config.num_nodes} nodes registered",
)
async def run_scenario(self) -> None:
"""Run the configured scenario."""
from turnstone.sim.scenario import SCENARIOS
scenario_cls = SCENARIOS.get(self._config.scenario)
if scenario_cls is None:
raise ValueError(f"Unknown scenario: {self._config.scenario!r}")
scenario = scenario_cls()
await scenario.run(self, self._config, self._metrics)
async def kill_node(self, node_id: str) -> None:
"""Simulate a node failure: stop heartbeat, stop processing."""
node = self._nodes.get(node_id)
if node and node._running:
node.stop()
self._metrics.record_node_kill(node_id)
# Remove from dispatchers
for d in self._dispatchers:
d.remove_node(node_id)
log.info("Killed node %s", node_id)
def report(self) -> dict:
"""Generate final metrics report."""
return self._metrics.summary()
async def stop(self) -> None:
"""Shutdown all nodes and cancel tasks."""
self._running = False
for node in self._nodes.values():
node.stop()
for d in self._dispatchers:
d.stop()
for task in self._tasks:
task.cancel()
await asyncio.gather(*self._tasks, return_exceptions=True)
if hasattr(self, "_executor"):
self._executor.shutdown(wait=False)
if self._pool:
self._pool.disconnect()
log.info("Cluster stopped")
turnstone/sim/config.py
+55
View File
@@ -0,0 +1,55 @@
"""Simulation configuration."""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class SimConfig:
"""All parameters controlling a simulation run."""
# -- cluster --
num_nodes: int = 10
max_ws_per_node: int = 10
# -- redis --
redis_host: str = "localhost"
redis_port: int = 6379
redis_db: int = 0
redis_password: str | None = None
prefix: str = "turnstone"
# -- heartbeat --
heartbeat_ttl: int = 60
# -- LLM simulation --
llm_latency_mean: float = 2.0
llm_latency_stddev: float = 0.5
llm_tokens_mean: int = 200
llm_tokens_stddev: int = 50
llm_token_rate: float = 50.0 # tokens/sec streaming speed
context_window: int = 131072 # for computing context ratio
# -- tool simulation --
tool_latency_mean: float = 0.5
tool_latency_stddev: float = 0.2
tool_failure_rate: float = 0.02
tool_calls_per_turn_mean: float = 1.5
tool_calls_per_turn_max: int = 4
max_tool_rounds: int = 3
# -- scenario --
scenario: str = "steady"
duration: int = 60
messages_per_second: float = 5.0
burst_size: int = 100
node_kill_interval: float = 15.0
node_kill_count: int = 1
# -- metrics --
metrics_interval: float = 5.0
metrics_file: str = ""
# -- reproducibility --
seed: int | None = None
turnstone/sim/engine.py
+139
View File
@@ -0,0 +1,139 @@
"""LLM and tool execution simulation engine."""
from __future__ import annotations
import asyncio
import random
from turnstone.sim.config import SimConfig
_WORD_POOL = [
"the",
"result",
"shows",
"that",
"this",
"file",
"contains",
"function",
"data",
"analysis",
"implementation",
"code",
"completed",
"successfully",
"reviewed",
"output",
"processing",
"module",
"system",
"request",
"response",
"value",
"config",
"status",
"running",
"checked",
"verified",
"found",
"done",
]
_TOOL_NAMES = [
"bash",
"read_file",
"search",
"edit_file",
"write_file",
"math",
"web_fetch",
]
class ToolSimulationError(Exception):
"""Raised when a simulated tool execution fails."""
class SimEngine:
"""Simulates LLM responses and tool execution with configurable distributions.
Stateless — safe to share across workstreams on the same node.
"""
def __init__(self, config: SimConfig, rng: random.Random | None = None):
self._config = config
self._rng = rng or random.Random(config.seed)
async def simulate_llm_response(
self, first_round: bool, turn_number: int
) -> tuple[str, list[dict]]:
"""Simulate an LLM response.
Returns ``(content_text, tool_calls)`` where *tool_calls* may be
empty (final answer) or a list of ``{"name": ..., "arguments": ...}``
dicts.
"""
latency = max(
0.05,
self._rng.gauss(
self._config.llm_latency_mean,
self._config.llm_latency_stddev,
),
)
await asyncio.sleep(latency)
num_tokens = max(
10,
int(
self._rng.gauss(
self._config.llm_tokens_mean,
self._config.llm_tokens_stddev,
)
),
)
content = self._generate_content(num_tokens)
# First round has a higher chance of tool calls; decreasing per round
tool_prob = 0.6 if first_round else 0.3
if self._rng.random() < tool_prob:
num_calls = min(
max(
1,
int(
self._rng.expovariate(
1.0 / self._config.tool_calls_per_turn_mean,
)
),
),
self._config.tool_calls_per_turn_max,
)
calls = [
{
"name": self._rng.choice(_TOOL_NAMES),
"arguments": '{"simulated": true}',
}
for _ in range(num_calls)
]
return content, calls
return content, []
async def simulate_tool_execution(self, tool_name: str) -> str:
"""Simulate tool execution with latency and possible failure."""
latency = max(
0.01,
self._rng.gauss(
self._config.tool_latency_mean,
self._config.tool_latency_stddev,
),
)
await asyncio.sleep(latency)
if self._rng.random() < self._config.tool_failure_rate:
raise ToolSimulationError(f"Simulated {tool_name} failure")
return f"[sim] {tool_name} completed successfully"
def _generate_content(self, num_tokens: int) -> str:
"""Generate placeholder content of approximately *num_tokens* tokens."""
return " ".join(self._rng.choices(_WORD_POOL, k=num_tokens))
turnstone/sim/metrics.py
+110
View File
@@ -0,0 +1,110 @@
"""Simulation metrics collection."""
from __future__ import annotations
import threading
import time
from collections import defaultdict
class MetricsCollector:
"""Thread-safe metrics collector for simulation runs.
Uses ``threading.Lock`` so it works from both sync and async contexts.
"""
def __init__(self) -> None:
self._lock = threading.Lock()
self._turn_latencies: list[float] = []
self._inject_times: list[float] = []
self._complete_times: list[float] = []
self._errors: int = 0
self._error_details: list[tuple[float, str, str]] = []
self._node_kills: list[tuple[float, str]] = []
self._turns_per_node: dict[str, int] = defaultdict(int)
self._ws_counts: list[dict[str, int]] = [] # utilization snapshots
def record_turn(self, ws_id: str, node_id: str, latency: float) -> None:
with self._lock:
self._turn_latencies.append(latency)
self._complete_times.append(time.monotonic())
self._turns_per_node[node_id] += 1
def record_inject(self) -> None:
with self._lock:
self._inject_times.append(time.monotonic())
def record_error(self, node_id: str, message: str) -> None:
with self._lock:
self._errors += 1
self._error_details.append((time.monotonic(), node_id, message))
def record_node_kill(self, node_id: str) -> None:
with self._lock:
self._node_kills.append((time.monotonic(), node_id))
def snapshot_utilization(self, ws_counts: dict[str, int]) -> None:
"""Record workstream-per-node counts at a point in time."""
with self._lock:
self._ws_counts.append(dict(ws_counts))
def summary(self) -> dict:
"""Generate final metrics report with percentiles and aggregates."""
with self._lock:
latencies = sorted(self._turn_latencies)
n = len(latencies)
if n > 0 and self._inject_times and self._complete_times:
duration = self._complete_times[-1] - self._inject_times[0]
else:
duration = 0.0
# Utilization from latest snapshot
util: dict = {}
if self._ws_counts:
last = self._ws_counts[-1]
counts = list(last.values())
if counts:
util = {
"mean_ws_per_node": sum(counts) / len(counts),
"max_ws_per_node": max(counts),
"nodes_with_zero_ws": sum(1 for c in counts if c == 0),
}
return {
"total_turns": n,
"total_errors": self._errors,
"duration_seconds": round(duration, 2),
"throughput": {
"messages_per_sec": round(
len(self._inject_times) / duration,
2,
)
if duration > 0
else 0,
"turns_per_sec": round(
n / duration,
2,
)
if duration > 0
else 0,
},
"latency": {
"p50": _percentile(latencies, 0.50),
"p90": _percentile(latencies, 0.90),
"p99": _percentile(latencies, 0.99),
"mean": round(sum(latencies) / n, 4) if n else 0,
"max": round(latencies[-1], 4) if n else 0,
},
"utilization": util,
"node_kills": len(self._node_kills),
"turns_per_node": dict(self._turns_per_node),
}
def _percentile(sorted_values: list[float], pct: float) -> float:
if not sorted_values:
return 0.0
idx = int(len(sorted_values) * pct)
idx = min(idx, len(sorted_values) - 1)
return round(sorted_values[idx], 4)
turnstone/sim/node.py
+439
View File
@@ -0,0 +1,439 @@
"""Simulated turnstone node.
A SimNode replaces Bridge + Server + ChatSession with a lightweight async
coroutine that talks directly to Redis via the real RedisBroker. External
observers (TurnstoneClient, turnstone-console) see identical protocol
behaviour.
"""
from __future__ import annotations
import asyncio
import json
import logging
import time
import uuid
from turnstone.mq.protocol import (
AckEvent,
ClusterStateEvent,
ContentEvent,
ErrorEvent,
HealthResponseEvent,
InboundMessage,
NodeListEvent,
StateChangeEvent,
StatusEvent,
StreamEndEvent,
ToolResultEvent,
TurnCompleteEvent,
WorkstreamClosedEvent,
WorkstreamCreatedEvent,
WorkstreamListEvent,
)
from turnstone.sim.config import SimConfig
from turnstone.sim.engine import SimEngine, ToolSimulationError
from turnstone.sim.metrics import MetricsCollector
log = logging.getLogger("turnstone.sim.node")
class SimWorkstream:
"""Lightweight workstream state machine."""
def __init__(
self,
ws_id: str,
name: str,
node: SimNode,
engine: SimEngine,
config: SimConfig,
):
self.ws_id = ws_id
self.name = name
self.state = "idle"
self._node = node
self._engine = engine
self._config = config
self._turn_count = 0
self._total_tokens = 0 # accumulated across turns
async def process_turn(self, message: str, correlation_id: str) -> None:
"""Simulate a complete turn: LLM stream -> optional tools -> final."""
t_start = time.monotonic()
self._turn_count += 1
try:
rounds = 0
while True:
# LLM thinking + streaming
self._set_state("thinking", correlation_id)
content, tool_calls = await self._engine.simulate_llm_response(
rounds == 0,
self._turn_count,
)
await self._stream_content(content, correlation_id)
if not tool_calls or rounds >= self._config.max_tool_rounds:
break
# Tool execution
self._set_state("running", correlation_id)
for tc in tool_calls:
name = tc["name"]
try:
output = await self._engine.simulate_tool_execution(name)
except ToolSimulationError as exc:
output = f"Error: {exc}"
self._node._metrics.record_error(
self._node.node_id,
str(exc),
)
self._node._publish_ws(
self.ws_id,
ToolResultEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
name=name,
output=output,
),
)
rounds += 1
# Finished — publish status, idle, turn complete
self._publish_status(correlation_id)
self._set_state("idle", correlation_id)
self._node._publish_ws(
self.ws_id,
TurnCompleteEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
),
)
self._node._publish_global(
TurnCompleteEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
),
)
except Exception as exc:
self._set_state("error", correlation_id)
self._node._publish_ws(
self.ws_id,
ErrorEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
message=str(exc),
),
)
self._node._metrics.record_error(self._node.node_id, str(exc))
finally:
latency = time.monotonic() - t_start
self._node._metrics.record_turn(
self.ws_id,
self._node.node_id,
latency,
)
async def _stream_content(self, text: str, correlation_id: str) -> None:
"""Simulate token-by-token streaming."""
if not text:
return
# Count tokens (~1 token per word) and accumulate
self._total_tokens += len(text.split())
chunk_size = max(1, len(text) // 8)
token_delay = 1.0 / max(1, self._config.llm_token_rate)
for i in range(0, len(text), chunk_size):
chunk = text[i : i + chunk_size]
self._node._publish_ws(
self.ws_id,
ContentEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
text=chunk,
),
)
await asyncio.sleep(token_delay * len(chunk.split()))
self._node._publish_ws(
self.ws_id,
StreamEndEvent(ws_id=self.ws_id, correlation_id=correlation_id),
)
def _set_state(self, state: str, correlation_id: str) -> None:
self.state = state
self._node._publish_global(
StateChangeEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
state=state,
),
)
# prompt tokens ~= 2x completion tokens for a realistic ratio
total = self._total_tokens * 3
ctx_ratio = round(total / self._config.context_window, 3) if total else 0.0
self._node._publish_cluster(
ClusterStateEvent(
ws_id=self.ws_id,
state=state,
node_id=self._node.node_id,
tokens=total,
context_ratio=ctx_ratio,
),
)
def _publish_status(self, correlation_id: str) -> None:
total = self._total_tokens * 3 # prompt ~= 2x completion
cw = self._config.context_window
self._node._publish_ws(
self.ws_id,
StatusEvent(
ws_id=self.ws_id,
correlation_id=correlation_id,
prompt_tokens=self._total_tokens * 2,
completion_tokens=self._total_tokens,
total_tokens=total,
context_window=cw,
pct=round(total / cw, 3) if cw else 0,
effort="medium",
),
)
class SimNode:
"""A lightweight simulated turnstone node.
Replaces Bridge + Server + ChatSession with direct Redis protocol
interaction.
"""
def __init__(
self,
node_id: str,
broker: object,
config: SimConfig,
metrics: MetricsCollector,
):
self.node_id = node_id
self._broker = broker
self._config = config
self._metrics = metrics
# Derive per-node seed so each node has unique RNG sequences
import random
node_seed = None
if config.seed is not None:
node_seed = hash((config.seed, node_id))
self._engine = SimEngine(config, rng=random.Random(node_seed))
self._workstreams: dict[str, SimWorkstream] = {}
self._running = True
self._started_at = time.time()
self._prefix = config.prefix
@property
def workstream_count(self) -> int:
return len(self._workstreams)
# -- message handling ----------------------------------------------------
async def handle_message(self, raw: str) -> None:
"""Parse and dispatch an inbound message."""
try:
msg = InboundMessage.from_json(raw)
await self._dispatch(msg)
except Exception as exc:
log.error("SimNode %s dispatch error: %s", self.node_id, exc)
self._publish_global(ErrorEvent(message=f"SimNode error: {exc}"))
async def _dispatch(self, msg: InboundMessage) -> None:
handlers = {
"send": self._handle_send,
"create_workstream": self._handle_create_ws,
"close_workstream": self._handle_close_ws,
"list_workstreams": self._handle_list_ws,
"health": self._handle_health,
"list_nodes": self._handle_list_nodes,
}
handler = handlers.get(msg.type)
if handler:
await handler(msg)
else:
log.debug("SimNode %s ignoring message type: %s", self.node_id, msg.type)
async def _handle_send(self, msg: InboundMessage) -> None:
ws_id = getattr(msg, "ws_id", "")
message = getattr(msg, "message", "")
cid = msg.correlation_id
# Find or create workstream
if ws_id and ws_id in self._workstreams:
ws = self._workstreams[ws_id]
elif len(self._workstreams) >= self._config.max_ws_per_node:
self._publish_global(
ErrorEvent(
correlation_id=cid,
message=f"Node {self.node_id} at capacity ({self._config.max_ws_per_node} ws)",
),
)
return
else:
ws = self._create_workstream(
name=getattr(msg, "name", ""),
correlation_id=cid,
)
self._publish_ws(
ws.ws_id,
AckEvent(ws_id=ws.ws_id, correlation_id=cid, status="ok"),
)
await ws.process_turn(message, cid)
async def _handle_create_ws(self, msg: InboundMessage) -> None:
if len(self._workstreams) >= self._config.max_ws_per_node:
self._publish_global(
ErrorEvent(
correlation_id=msg.correlation_id,
message=f"Node {self.node_id} at capacity ({self._config.max_ws_per_node} ws)",
),
)
return
name = getattr(msg, "name", "")
ws = self._create_workstream(name=name, correlation_id=msg.correlation_id)
self._publish_ws(
ws.ws_id,
AckEvent(ws_id=ws.ws_id, correlation_id=msg.correlation_id, status="ok"),
)
async def _handle_close_ws(self, msg: InboundMessage) -> None:
ws_id = getattr(msg, "ws_id", "")
ws = self._workstreams.pop(ws_id, None)
if ws:
self._broker.del_ws_owner(ws_id)
event = WorkstreamClosedEvent(
ws_id=ws_id,
correlation_id=msg.correlation_id,
)
self._publish_global(event)
self._publish_cluster(event)
async def _handle_list_ws(self, msg: InboundMessage) -> None:
ws_list = [
{"id": ws.ws_id, "name": ws.name, "state": ws.state}
for ws in self._workstreams.values()
]
self._publish_global(
WorkstreamListEvent(
correlation_id=msg.correlation_id,
workstreams=ws_list,
),
)
async def _handle_health(self, msg: InboundMessage) -> None:
self._publish_global(
HealthResponseEvent(
correlation_id=msg.correlation_id,
data={
"status": "ok",
"node_id": self.node_id,
"sim": True,
"workstreams": len(self._workstreams),
},
),
)
async def _handle_list_nodes(self, msg: InboundMessage) -> None:
nodes = self._broker.list_nodes()
self._publish_global(
NodeListEvent(correlation_id=msg.correlation_id, nodes=nodes),
)
# -- workstream lifecycle ------------------------------------------------
def _create_workstream(
self,
name: str = "",
correlation_id: str = "",
) -> SimWorkstream:
ws_id = uuid.uuid4().hex[:8]
if not name:
name = f"sim-ws-{ws_id[:4]}"
ws = SimWorkstream(ws_id, name, self, self._engine, self._config)
self._workstreams[ws_id] = ws
self._broker.set_ws_owner(ws_id, self.node_id)
event = WorkstreamCreatedEvent(
ws_id=ws_id,
correlation_id=correlation_id,
name=name,
)
self._publish_global(event)
# Also publish to cluster channel so the console discovers the ws.
# Include node_id (the console collector keys on it).
self._publish_cluster(
ClusterStateEvent(
ws_id=ws_id,
state="idle",
node_id=self.node_id,
),
)
# The cluster channel expects a ws_created with node_id for the
# collector's _on_cluster_event handler.
self._broker.publish_outbound(
f"{self._prefix}:events:cluster",
json.dumps(
{
"type": "ws_created",
"ws_id": ws_id,
"name": name,
"node_id": self.node_id,
"correlation_id": correlation_id,
}
),
)
return ws
# -- heartbeat -----------------------------------------------------------
def heartbeat_once(self) -> None:
"""Register a single heartbeat with the broker."""
self._broker.register_node(
self.node_id,
{
"server_url": f"sim://{self.node_id}",
"started": self._started_at,
"sim": True,
"workstreams": len(self._workstreams),
"max_ws": self._config.max_ws_per_node,
},
ttl=self._config.heartbeat_ttl,
)
# -- shutdown ------------------------------------------------------------
def stop(self) -> None:
"""Mark node as stopped and clean up ownership keys."""
self._running = False
for ws_id in list(self._workstreams):
self._broker.del_ws_owner(ws_id)
self._workstreams.clear()
# -- event publishing helpers --------------------------------------------
def _publish_global(self, event: object) -> None:
self._broker.publish_outbound(
f"{self._prefix}:events:global",
event.to_json(),
)
def _publish_ws(self, ws_id: str, event: object) -> None:
self._broker.publish_outbound(
f"{self._prefix}:events:{ws_id}",
event.to_json(),
)
def _publish_cluster(self, event: object) -> None:
self._broker.publish_outbound(
f"{self._prefix}:events:cluster",
event.to_json(),
)
turnstone/sim/scenario.py
+234
View File
@@ -0,0 +1,234 @@
"""Simulation scenarios — workload patterns for cluster testing."""
from __future__ import annotations
import asyncio
import logging
import time
from typing import TYPE_CHECKING, Protocol
from turnstone.mq.broker import RedisBroker
from turnstone.mq.protocol import SendMessage
from turnstone.sim.config import SimConfig
from turnstone.sim.metrics import MetricsCollector
if TYPE_CHECKING:
from turnstone.sim.cluster import SimCluster
log = logging.getLogger("turnstone.sim.scenario")
class Scenario(Protocol):
async def run(
self,
cluster: SimCluster,
config: SimConfig,
metrics: MetricsCollector,
) -> None: ...
class SteadyStateScenario:
"""Inject messages at a constant rate for the configured duration."""
async def run(
self,
cluster: SimCluster,
config: SimConfig,
metrics: MetricsCollector,
) -> None:
broker = _make_broker(config)
interval = 1.0 / max(0.01, config.messages_per_second)
deadline = time.monotonic() + config.duration
count = 0
try:
while time.monotonic() < deadline:
count += 1
msg = SendMessage(
message=f"Steady-state message {count}",
auto_approve=True,
)
broker.push_inbound(msg.to_json())
metrics.record_inject()
await asyncio.sleep(interval)
finally:
# Allow in-flight turns to finish
await asyncio.sleep(min(10, config.llm_latency_mean * 3))
broker.close()
log.info("Steady-state scenario complete: %d messages injected", count)
class BurstScenario:
"""Inject burst_size messages as fast as possible, then wait."""
async def run(
self,
cluster: SimCluster,
config: SimConfig,
metrics: MetricsCollector,
) -> None:
broker = _make_broker(config)
try:
for i in range(config.burst_size):
msg = SendMessage(
message=f"Burst message {i}",
auto_approve=True,
)
broker.push_inbound(msg.to_json())
metrics.record_inject()
log.info("Burst injected: %d messages", config.burst_size)
# Wait for processing to complete
await asyncio.sleep(config.duration)
finally:
broker.close()
class NodeFailureScenario:
"""Steady-state load with periodic node kills."""
async def run(
self,
cluster: SimCluster,
config: SimConfig,
metrics: MetricsCollector,
) -> None:
# Start steady injection in background
steady = SteadyStateScenario()
load_task = asyncio.create_task(steady.run(cluster, config, metrics))
# Periodically kill nodes
killed = 0
max_kills = config.num_nodes // 2 # never kill more than half
node_ids = list(cluster.nodes.keys())
try:
while killed < max_kills:
await asyncio.sleep(config.node_kill_interval)
for _ in range(config.node_kill_count):
if killed < len(node_ids):
await cluster.kill_node(node_ids[killed])
killed += 1
finally:
await load_task
log.info("Node-failure scenario complete: %d nodes killed", killed)
class DirectedScenario:
"""Send messages targeted to specific nodes."""
async def run(
self,
cluster: SimCluster,
config: SimConfig,
metrics: MetricsCollector,
) -> None:
broker = _make_broker(config)
node_ids = list(cluster.nodes.keys())
count = min(config.burst_size, len(node_ids))
try:
for i in range(count):
target = node_ids[i % len(node_ids)]
msg = SendMessage(
message=f"Directed message to {target}",
auto_approve=True,
target_node=target,
)
broker.push_inbound(msg.to_json(), node_id=target)
metrics.record_inject()
log.info("Directed scenario: %d messages sent to specific nodes", count)
await asyncio.sleep(config.duration)
finally:
broker.close()
class LifecycleScenario:
"""Create, use, and close workstreams across nodes."""
async def run(
self,
cluster: SimCluster,
config: SimConfig,
metrics: MetricsCollector,
) -> None:
from turnstone.mq.protocol import (
CloseWorkstreamMessage,
CreateWorkstreamMessage,
)
broker = _make_broker(config)
ws_ids: list[str] = []
try:
# Phase 1: Create workstreams
create_count = min(50, config.num_nodes * 2)
for i in range(create_count):
msg = CreateWorkstreamMessage(
name=f"lifecycle-ws-{i}",
auto_approve=True,
)
broker.push_inbound(msg.to_json())
metrics.record_inject()
await asyncio.sleep(0.05)
# Let creations settle
await asyncio.sleep(3)
# Phase 2: Send messages to shared queue (will be routed to nodes
# that own workstreams)
for i in range(create_count):
msg = SendMessage(
message=f"Lifecycle message {i}",
auto_approve=True,
)
broker.push_inbound(msg.to_json())
metrics.record_inject()
await asyncio.sleep(0.1)
# Let turns complete
await asyncio.sleep(min(15, config.llm_latency_mean * 5))
# Phase 3: Close half the workstreams
# Collect ws_ids from nodes
for node in cluster.nodes.values():
for ws_id in list(node._workstreams.keys()):
ws_ids.append(ws_id)
close_count = len(ws_ids) // 2
for ws_id in ws_ids[:close_count]:
owner = broker.get_ws_owner(ws_id)
msg = CloseWorkstreamMessage(ws_id=ws_id)
broker.push_inbound(msg.to_json(), node_id=owner or "")
await asyncio.sleep(0.05)
await asyncio.sleep(2)
log.info(
"Lifecycle scenario complete: created %d, closed %d",
create_count,
close_count,
)
finally:
broker.close()
def _make_broker(config: SimConfig) -> RedisBroker:
"""Create a RedisBroker for scenario message injection."""
return RedisBroker(
host=config.redis_host,
port=config.redis_port,
db=config.redis_db,
prefix=config.prefix,
password=config.redis_password,
)
SCENARIOS: dict[str, type] = {
"steady": SteadyStateScenario,
"burst": BurstScenario,
"node_failure": NodeFailureScenario,
"directed": DirectedScenario,
"lifecycle": LifecycleScenario,
}
turnstone/tools/bash.json
+16
View File
@@ -0,0 +1,16 @@
{
"name": "bash",
"description": "Execute a bash command and return stdout + stderr. Use this tool freely for any task: checking time, reading files, running programs, system info, etc.",
"parameters": {
"type": "object",
"properties": {
"command": {
"type": "string",
"description": "The bash command to execute."
}
},
"required": ["command"]
},
"task_agent": true,
"primary_key": "command"
}
turnstone/tools/edit_file.json
+28
View File
@@ -0,0 +1,28 @@
{
"name": "edit_file",
"description": "Replace an exact string in a file with new content. Fails if old_string is not found or matches multiple locations (use near_line to disambiguate). Requires read_file on the same path first.",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Absolute or relative file path."
},
"old_string": {
"type": "string",
"description": "The exact text to find and replace."
},
"new_string": {
"type": "string",
"description": "The replacement text."
},
"near_line": {
"type": "integer",
"description": "When old_string matches multiple locations, pick the one nearest this line number."
}
},
"required": ["path", "old_string", "new_string"]
},
"task_agent": true,
"primary_key": "old_string"
}
turnstone/tools/forget.json
+15
View File
@@ -0,0 +1,15 @@
{
"name": "forget",
"description": "Remove a persistent memory by key. Use when the user asks to forget, remove, or delete a stored memory.",
"parameters": {
"type": "object",
"properties": {
"key": {
"type": "string",
"description": "The memory key to remove (e.g. 'user_name')."
}
},
"required": ["key"]
},
"primary_key": "key"
}
turnstone/tools/man.json
+22
View File
@@ -0,0 +1,22 @@
{
"name": "man",
"description": "Read a man page. Use this instead of bash('man ...') or web_search. Returns the full formatted manual entry.",
"parameters": {
"type": "object",
"properties": {
"page": {
"type": "string",
"description": "The man page name (e.g. 'grep', 'socket', 'printf')."
},
"section": {
"type": "string",
"description": "Manual section (e.g. '1' commands, '2' syscalls, '3' library). Optional."
}
},
"required": ["page"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "page"
}
turnstone/tools/math.json
+18
View File
@@ -0,0 +1,18 @@
{
"name": "math",
"description": "Execute Python code for math/computation. Code MUST use print() to produce output. Available: sympy, numpy, scipy (with scipy.special, scipy.optimize, scipy.integrate, scipy.linalg), math, fractions, itertools, functools, collections, decimal, operator, random, re, string. Common sympy names (symbols, solve, simplify, expand, factor, sqrt, Rational, Matrix, integrate, diff, etc.) are pre-imported. Example: x = symbols('x'); print(solve(x**2 - 4, x))",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "Python code to execute. Must use print() for output."
}
},
"required": ["code"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "code"
}
turnstone/tools/plan.json
+15
View File
@@ -0,0 +1,15 @@
{
"name": "plan",
"description": "Plan before implementing. An autonomous agent explores the codebase and writes a structured plan to .plan-<session_id>.md (unique per session to avoid workstream collisions). If a plan for this session already exists it is re-read and refined rather than overwritten from scratch. Use plan BEFORE writing code — when the user asks to build, add, refactor, or change something that touches multiple files or has unclear scope. The plan identifies files to modify, existing patterns to reuse, and risks to consider.",
"parameters": {
"type": "object",
"properties": {
"prompt": {
"type": "string",
"description": "What to plan — the goal, constraints, and scope."
}
},
"required": ["prompt"]
},
"primary_key": "prompt"
}
turnstone/tools/read_file.json
+26
View File
@@ -0,0 +1,26 @@
{
"name": "read_file",
"description": "Read the contents of a file. Returns numbered lines. Must be called before edit_file on the same path.",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Absolute or relative file path."
},
"offset": {
"type": "integer",
"description": "Line number to start reading from (1-based, default: 1)."
},
"limit": {
"type": "integer",
"description": "Maximum number of lines to read. Omit to read entire file."
}
},
"required": ["path"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "path"
}
turnstone/tools/recall.json
+18
View File
@@ -0,0 +1,18 @@
{
"name": "recall",
"description": "Search memories and past conversations. With no query, lists all saved memories. With a query, searches both memories and conversation history.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search term or phrase. Omit to list all memories."
},
"limit": {
"type": "integer",
"description": "Max conversation results to return (default 20)."
}
}
},
"primary_key": "query"
}
turnstone/tools/remember.json
+19
View File
@@ -0,0 +1,19 @@
{
"name": "remember",
"description": "Save a persistent memory. Memories persist across sessions. Use to remember IPs, paths, commands, conventions, or any fact worth recalling later.",
"parameters": {
"type": "object",
"properties": {
"key": {
"type": "string",
"description": "Short identifier (e.g. 'user_name')."
},
"value": {
"type": "string",
"description": "Content to remember."
}
},
"required": ["key", "value"]
},
"primary_key": "key"
}
turnstone/tools/search.json
+22
View File
@@ -0,0 +1,22 @@
{
"name": "search",
"description": "Search file contents for a regex pattern. Returns matching lines with file paths and line numbers. Searches recursively when path is a directory.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Regex pattern to search for (extended regex)."
},
"path": {
"type": "string",
"description": "File or directory to search in (default: current directory)."
}
},
"required": ["query"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "query"
}
turnstone/tools/task.json
+15
View File
@@ -0,0 +1,15 @@
{
"name": "task",
"description": "Delegate a general-purpose task to an autonomous sub-agent. The agent inherits all tools and can read, write, edit, search, and run commands. Use task for work that requires file modifications or command execution. Provide a clear, self-contained prompt.",
"parameters": {
"type": "object",
"properties": {
"prompt": {
"type": "string",
"description": "Complete task description for the sub-agent."
}
},
"required": ["prompt"]
},
"primary_key": "prompt"
}
turnstone/tools/web_fetch.json
+22
View File
@@ -0,0 +1,22 @@
{
"name": "web_fetch",
"description": "Fetch a URL and extract specific information from it. You must provide a question or extraction guidance — the page is fetched, analyzed, and only relevant information is returned (not raw page content).",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The URL to fetch (must start with http:// or https://)."
},
"question": {
"type": "string",
"description": "What to extract or answer from the page content."
}
},
"required": ["url", "question"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "url"
}
turnstone/tools/web_search.json
+26
View File
@@ -0,0 +1,26 @@
{
"name": "web_search",
"description": "Search the web using a text query. Returns ranked results with titles, URLs, and content snippets.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The search query."
},
"max_results": {
"type": "integer",
"description": "Max results to return (default 5, max 20)."
},
"topic": {
"type": "string",
"description": "Search topic: general, news, or finance (default general)."
}
},
"required": ["query"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "query"
}
turnstone/tools/write_file.json
+20
View File
@@ -0,0 +1,20 @@
{
"name": "write_file",
"description": "Write content to a file, creating it if needed. Overwrites existing content.",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Absolute or relative file path."
},
"content": {
"type": "string",
"description": "The full file content to write."
}
},
"required": ["path", "content"]
},
"task_agent": true,
"primary_key": "content"
}
turnstone/ui/__init__.py
View File
turnstone/ui/colors.py
+46
View File
@@ -0,0 +1,46 @@
"""ANSI color constants and helper functions.
Respects the NO_COLOR convention (https://no-color.org/) and suppresses
color when stdout is not a terminal (piped output).
"""
import os
import sys
_use_color = "NO_COLOR" not in os.environ and sys.stdout.isatty()
RESET = "\033[0m" if _use_color else ""
BOLD = "\033[1m" if _use_color else ""
DIM = "\033[2m" if _use_color else ""
ITALIC = "\033[3m" if _use_color else ""
RED = "\033[31m" if _use_color else ""
GREEN = "\033[32m" if _use_color else ""
YELLOW = "\033[33m" if _use_color else ""
BLUE = "\033[34m" if _use_color else ""
MAGENTA = "\033[35m" if _use_color else ""
CYAN = "\033[36m" if _use_color else ""
GRAY = "\033[90m" if _use_color else ""
def red(s):
return f"{RED}{s}{RESET}"
def yellow(s):
return f"{YELLOW}{s}{RESET}"
def dim(s):
return f"{DIM}{s}{RESET}"
def bold(s):
return f"{BOLD}{s}{RESET}"
def cyan(s):
return f"{CYAN}{s}{RESET}"
def green(s):
return f"{GREEN}{s}{RESET}"
turnstone/ui/markdown.py
+66
View File
@@ -0,0 +1,66 @@
"""Line-buffered markdown to ANSI renderer for streaming output."""
import re
from turnstone.ui.colors import BOLD, CYAN, DIM, ITALIC, MAGENTA, RESET
class MarkdownRenderer:
"""Line-buffered markdown → ANSI converter for streaming output.
Buffers content until a newline arrives, then renders the complete line
with regex-based markdown → ANSI conversion. Multi-line constructs
(fenced code blocks) track state across lines.
"""
def __init__(self):
self.in_code_block = False
self._buf = ""
def feed(self, text: str) -> str:
"""Feed text, return ANSI-rendered output for complete lines."""
self._buf += text
out = []
while "\n" in self._buf:
line, self._buf = self._buf.split("\n", 1)
out.append(self._render_line(line))
return "\n".join(out) + "\n" if out else ""
def flush(self) -> str:
"""Flush remaining buffer (end of stream)."""
if self._buf:
rendered = self._render_line(self._buf)
self._buf = ""
return rendered
return ""
def _render_line(self, line: str) -> str:
# Code block fence toggle
if line.strip().startswith("```"):
self.in_code_block = not self.in_code_block
return f"{DIM}{line}{RESET}"
# Inside code block — cyan, no further markdown processing
if self.in_code_block:
return f"{CYAN}{line}{RESET}"
# Headers (# H1, ## H2, ### H3, #### H4, ##### H5, ###### H6)
m = re.match(r"^(#{1,6}) (.+)", line)
if m:
return f"{BOLD}{MAGENTA}{m.group(2)}{RESET}"
# Inline formatting (order matters: bold before italic)
line = re.sub(r"\*\*(.+?)\*\*", f"{BOLD}\\1{RESET}", line)
line = re.sub(r"__(.+?)__", f"{BOLD}\\1{RESET}", line)
line = re.sub(
r"(?<!\*)\*(?!\*)(.+?)(?<!\*)\*(?!\*)", f"{ITALIC}\\1{RESET}", line
)
line = re.sub(r"`(.+?)`", f"{CYAN}\\1{RESET}", line)
# Bullet lists — cyan bullet
line = re.sub(r"^(\s*)([-*]) ", f"\\1{CYAN}\\2{RESET} ", line)
# Numbered lists — cyan number
line = re.sub(r"^(\s*)(\d+)\. ", f"\\1{CYAN}\\2.{RESET} ", line)
return line
turnstone/ui/spinner.py
+47
View File
@@ -0,0 +1,47 @@
"""Animated terminal spinner for long-running operations."""
import sys
import threading
from turnstone.ui.colors import DIM, RESET
class Spinner:
"""Braille-character animated spinner for terminal display."""
_FRAMES = "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏"
def __init__(self, message: str = "Thinking"):
self.message = message
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
def start(self):
self._stop_event.clear()
self._thread = threading.Thread(target=self._spin, daemon=True)
self._thread.start()
def _spin(self):
i = 0
while not self._stop_event.wait(0.08):
frame = self._FRAMES[i % len(self._FRAMES)]
sys.stderr.write(f"\r{DIM}{frame} {self.message}…{RESET} ")
sys.stderr.flush()
i += 1
def stop(self):
if self._stop_event.is_set():
return
self._stop_event.set()
if self._thread:
self._thread.join()
self._thread = None
sys.stderr.write("\r\033[2K")
sys.stderr.flush()
def __enter__(self):
self.start()
return self
def __exit__(self, *_):
self.stop()
turnstone/ui/static/app.js
+1851
View File
File diff suppressed because it is too large Load Diff
turnstone/ui/static/index.html
+88
View File
@@ -0,0 +1,88 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>turnstone</title>
<link rel="stylesheet" href="/static/style.css">
</head>
<body>
<div id="header">
<div id="hamburger-wrap">
<button id="hamburger-btn" onclick="toggleHamburger()" aria-label="Menu" aria-haspopup="true" aria-expanded="false" aria-controls="hamburger-menu">
<span></span><span></span><span></span>
</button>
<div id="hamburger-menu" role="menu">
<button class="hmenu-item" role="menuitem" tabindex="-1" onclick="hamburgerDashboard()">
<span class="hmenu-icon">&#8962;</span> Dashboard
</button>
<div class="hmenu-sep" role="separator"></div>
<button class="hmenu-item" role="menuitem" tabindex="-1" id="theme-menu-item" onclick="hamburgerTheme()">
<span class="hmenu-icon" id="theme-menu-icon">&#9790;</span> <span id="theme-menu-label">Light mode</span>
</button>
</div>
</div>
<h1>pcode</h1>
<span id="model-name"></span>
<span id="status-bar"></span>
<button id="logout-btn" onclick="logout()" style="display:none; background:none; border:1px solid var(--border); color:var(--fg-dim); border-radius:var(--radius); padding:4px 8px; cursor:pointer; font:inherit; font-size:12px;">logout</button>
</div>
<div id="tab-bar" role="tablist">
<button id="new-tab-btn" onclick="newWorkstream()" title="New workstream (Ctrl+T)" aria-label="New workstream">+</button>
</div>
<div id="dashboard" class="dashboard-overlay" role="dialog" aria-modal="true" aria-label="Dashboard">
<div class="dashboard-content">
<div class="dashboard-input-row">
<input type="text" id="dashboard-input" class="dashboard-input"
placeholder="What are you working on?" aria-label="Start a new conversation">
<button class="dashboard-new-btn" onclick="dashboardNewChat()" aria-label="New empty chat">New Chat</button>
</div>
<div class="dash-header">
<span class="dash-header-title">WORKSTREAMS</span>
<span class="dash-header-summary" id="dash-summary"></span>
</div>
<div class="dash-colheaders" aria-hidden="true">
<span class="dash-col dash-col-state">STATE</span>
<span class="dash-col dash-col-name">NAME</span>
<span class="dash-col dash-col-node">NODE</span>
<span class="dash-col dash-col-task">TASK</span>
<span class="dash-col dash-col-tokens">TOKENS</span>
<span class="dash-col dash-col-ctx">CTX</span>
</div>
<div class="dash-table" id="dash-ws-table" role="group" aria-label="Workstreams"></div>
<div class="dash-footer" id="dash-footer">
<span class="dash-footer-nodes" id="dash-footer-nodes"></span>
<span class="dash-footer-stats" id="dash-footer-stats"></span>
</div>
<section class="dashboard-section" id="dashboard-sessions" aria-label="Recent sessions">
<h2 class="dashboard-section-title">Recent Sessions</h2>
<div class="dashboard-cards" id="dashboard-session-cards"></div>
</section>
</div>
</div>
<div id="messages" role="log" aria-live="polite" aria-label="Chat messages"></div>
<!-- Plan review dialog -->
<div id="plan-overlay">
<div id="plan-dialog" role="dialog" aria-modal="true" aria-labelledby="plan-dialog-title">
<h3 id="plan-dialog-title">Plan Review</h3>
<div id="plan-content"></div>
<input type="text" id="plan-feedback" placeholder="Feedback (empty = approve)...">
<div id="plan-buttons">
<button id="btn-plan-reject" onclick="resolvePlan('reject')">Reject</button>
<button id="btn-plan-approve" onclick="resolvePlan('')">Approve</button>
</div>
</div>
</div>
<div id="input-area">
<textarea id="input" rows="1" placeholder="Type a message... (Shift+Enter for newline)" aria-label="Message input"></textarea>
<button id="send-btn" onclick="sendMessage()">Send</button>
</div>
<script src="/static/app.js"></script>
</body>
</html>
turnstone/ui/static/style.css
+299
View File
@@ -0,0 +1,299 @@
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
:root {
--bg: #1a1b26; --bg-surface: #24283b; --bg-highlight: #292e42;
--fg: #c8d1f5; --fg-dim: #828db5; --fg-bright: #a9b1d6;
--accent: #7aa2f7; --green: #9ece6a; --red: #f7768e;
--yellow: #e0af68; --cyan: #7dcfff; --magenta: #bb9af7;
--border: #3b4261; --code-bg: #1f2335;
--radius: 8px;
--dash-grid: 72px 120px 100px 1fr 60px 48px;
}
[data-theme="light"] {
--bg: #f5f5f5; --bg-surface: #ffffff; --bg-highlight: #e8e8ec;
--fg: #1a1a2e; --fg-dim: #4b5563; --fg-bright: #374151;
--accent: #1d4ed8; --green: #15803d; --red: #b91c1c;
--yellow: #92400e; --cyan: #0e7490; --magenta: #7e22ce;
--border: #d1d5db; --code-bg: #eaeaef;
}
html, body { height: 100%; background: var(--bg); color: var(--fg); font-family: 'SF Mono', 'Cascadia Code', 'Fira Code', 'Consolas', monospace; font-size: 14px; }
body { display: flex; flex-direction: column; }
/* Header */
#header { padding: 8px 16px; background: var(--bg-surface); border-bottom: 1px solid var(--border); display: flex; align-items: center; gap: 12px; flex-shrink: 0; }
#header h1 { font-size: 16px; color: var(--accent); font-weight: 600; }
#status-bar { font-size: 12px; color: var(--fg-dim); margin-left: auto; }
.skip-permissions-warning { font-size: 12px; color: var(--yellow); font-weight: 600; padding: 2px 8px; border: 1px solid var(--yellow); border-radius: var(--radius); }
/* Hamburger menu */
#hamburger-wrap { position: relative; }
#hamburger-btn { background:none; border:1px solid var(--border); color:var(--fg); border-radius:var(--radius); width:32px; height:32px; cursor:pointer; display:flex; flex-direction:column; align-items:center; justify-content:center; gap:4px; padding:0; flex-shrink:0; }
#hamburger-btn:hover { background:var(--bg-highlight); border-color:var(--fg-dim); }
#hamburger-btn:focus-visible { outline:2px solid var(--accent); outline-offset:2px; }
#hamburger-btn span { display:block; width:14px; height:2px; background:var(--fg); border-radius:1px; }
#hamburger-menu { display:none; position:absolute; top:calc(100% + 6px); left:0; background:var(--bg-surface); border:1px solid var(--border); border-radius:var(--radius); min-width:180px; box-shadow:0 4px 16px rgba(0,0,0,0.3); z-index:40; overflow:hidden; }
[data-theme="light"] #hamburger-menu { box-shadow:0 4px 16px rgba(0,0,0,0.12); }
#hamburger-menu.open { display:block; }
.hmenu-item { display:flex; align-items:center; gap:10px; width:100%; padding:10px 14px; background:none; border:none; color:var(--fg); font:inherit; font-size:13px; cursor:pointer; text-align:left; white-space:nowrap; }
.hmenu-item:hover { background:var(--bg-highlight); }
.hmenu-item:focus-visible { outline:2px solid var(--accent); outline-offset:-2px; }
.hmenu-item .hmenu-icon { width:16px; text-align:center; font-size:14px; opacity:0.8; }
.hmenu-sep { height:1px; background:var(--border); margin:4px 0; }
@media (max-width:600px) {
#hamburger-btn { width:40px; height:40px; }
.hmenu-item { padding:12px 14px; }
}
/* Tab bar */
#tab-bar { display: flex; align-items: center; gap: 2px; padding: 4px 16px; background: var(--bg-surface); border-bottom: 1px solid var(--border); flex-shrink: 0; overflow-x: auto; }
#tab-bar::-webkit-scrollbar { height: 4px; }
#tab-bar::-webkit-scrollbar-thumb { background: var(--border); border-radius: 2px; }
.ws-tab { padding: 5px 10px; border-radius: 6px 6px 0 0; cursor: pointer; display: flex; align-items: center; gap: 6px; font-size: 12px; background: var(--bg); border: 1px solid var(--border); border-bottom: none; color: var(--fg-dim); white-space: nowrap; user-select: none; position: relative; }
.ws-tab:hover { background: var(--bg-highlight); color: var(--fg-bright); }
.ws-tab.active { background: var(--bg-highlight); color: var(--fg-bright); border-bottom: 2px solid var(--accent); }
.ws-tab .tab-indicator { width: 8px; height: 8px; border-radius: 50%; flex-shrink: 0; }
.ws-tab .tab-indicator[data-state="idle"] { background: var(--fg-dim); opacity: 0.3; }
.ws-tab .tab-indicator[data-state="thinking"] { background: var(--cyan); animation: pulse 1.5s ease-in-out infinite; }
.ws-tab .tab-indicator[data-state="running"] { background: var(--green); border-radius: 2px; animation: pulse 1s ease-in-out infinite; }
.ws-tab .tab-indicator[data-state="attention"] { background: var(--yellow); border-radius: 1px; transform: rotate(45deg); animation: pulse 1s ease-in-out infinite; }
.ws-tab .tab-indicator[data-state="error"] { background: var(--red); }
@keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.3; } }
@media (prefers-reduced-motion: reduce) {
.ws-tab .tab-indicator[data-state="thinking"],
.ws-tab .tab-indicator[data-state="running"],
.ws-tab .tab-indicator[data-state="attention"],
.dash-state-dot[data-state="running"],
.dash-state-dot[data-state="thinking"],
.dash-state-dot[data-state="attention"] { animation: none; opacity: 1; }
.thinking-indicator::after { animation: none; content: '...'; }
}
.ws-tab .tab-close { background: none; border: none; color: var(--fg-dim); font-size: 14px; cursor: pointer; padding: 0 2px; line-height: 1; opacity:0; transition:opacity 0.15s; }
.ws-tab:hover .tab-close, .ws-tab:focus-within .tab-close, .ws-tab .tab-close:focus-visible { opacity:1; }
.ws-tab .tab-close:hover { color: var(--red); }
#new-tab-btn { background: none; border: 1px dashed var(--border); color: var(--fg-dim); border-radius: 6px 6px 0 0; padding: 5px 10px; cursor: pointer; font-family: inherit; font-size: 14px; line-height: 1; }
#new-tab-btn:hover { background: var(--bg-highlight); color: var(--fg-bright); border-color: var(--accent); }
/* Messages */
#messages { flex: 1; overflow-y: auto; padding: 16px; display: flex; flex-direction: column; gap: 12px; }
.msg { padding: 10px 14px; border-radius: var(--radius); max-width: 95%; line-height: 1.55; word-wrap: break-word; overflow-wrap: break-word; }
.msg-user { background: var(--bg-highlight); border: 1px solid var(--border); align-self: flex-end; color: var(--fg-bright); }
.msg-assistant { align-self: flex-start; }
.msg-info { color: var(--cyan); font-size: 13px; padding: 4px 14px; white-space: pre-wrap; font-family: inherit; }
.msg-error { color: var(--red); font-size: 13px; padding: 4px 14px; }
.msg-tool { background: var(--bg-surface); border: 1px solid var(--border); border-left: 3px solid var(--yellow); font-size: 13px; padding: 8px 12px; align-self: flex-start; max-width: 95%; }
.msg-tool .tool-header { color: var(--yellow); font-weight: 600; margin-bottom: 4px; }
.msg-tool .tool-preview { color: var(--fg-dim); white-space: pre-wrap; font-size: 12px; max-height: 300px; overflow-y: auto; }
/* Streaming content */
.reasoning { color: var(--fg-dim); font-style: italic; }
.thinking-indicator { color: var(--fg-dim); font-size: 13px; padding: 6px 14px; }
.thinking-indicator::after { content: '...'; animation: dots 1.5s steps(3, end) infinite; }
@keyframes dots { 0% { content: '.'; } 33% { content: '..'; } 66% { content: '...'; } }
/* Markdown styling */
.msg-assistant h1, .msg-assistant h2, .msg-assistant h3 { color: var(--accent); margin: 8px 0 4px; }
.msg-assistant h1 { font-size: 18px; } .msg-assistant h2 { font-size: 16px; } .msg-assistant h3 { font-size: 14px; }
.msg-assistant strong { color: var(--fg-bright); }
.msg-assistant em { color: var(--magenta); }
.msg-assistant code { background: var(--code-bg); padding: 2px 5px; border-radius: 3px; font-size: 13px; }
.msg-assistant pre { background: var(--code-bg); padding: 10px 12px; border-radius: var(--radius); overflow-x: auto; margin: 6px 0; border: 1px solid var(--border); }
.msg-assistant pre code { background: none; padding: 0; }
.msg-assistant ul, .msg-assistant ol { margin: 4px 0 4px 20px; }
.msg-assistant li { margin: 2px 0; }
.msg-assistant a { color: var(--accent); text-decoration: underline; }
.msg-assistant blockquote { border-left: 3px solid var(--border); padding-left: 10px; color: var(--fg-dim); margin: 4px 0; }
.msg-assistant hr { border: none; border-top: 1px solid var(--border); margin: 8px 0; }
.msg-assistant p { margin: 4px 0; }
/* Input area */
#input-area { padding: 12px 16px; background: var(--bg-surface); border-top: 1px solid var(--border); display: flex; gap: 8px; flex-shrink: 0; }
#input-area textarea { flex: 1; background: var(--bg); color: var(--fg); border: 1px solid var(--border); border-radius: var(--radius); padding: 8px 12px; font-family: inherit; font-size: 14px; resize: none; outline: none; min-height: 40px; max-height: 200px; }
#input-area textarea:focus { border-color: var(--accent); }
#input-area button { background: var(--accent); color: var(--bg); border: none; border-radius: var(--radius); padding: 8px 16px; font-family: inherit; font-size: 14px; cursor: pointer; font-weight: 600; }
#input-area button:hover { opacity: 0.9; }
#input-area button:disabled { opacity: 0.4; cursor: not-allowed; }
/* Inline approval blocks */
.approval-block { background: var(--bg-surface); border: 1px solid var(--border); border-left: 3px solid var(--yellow); border-radius: var(--radius); padding: 0; align-self: flex-start; max-width: 95%; font-size: 13px; }
.approval-block.approved { border-left-color: var(--green); }
.approval-block.denied { border-left-color: var(--red); }
.approval-tool { padding: 6px 12px; border-bottom: 1px solid var(--border); }
.approval-tool:last-of-type { border-bottom: none; }
.approval-tool .tool-name { color: var(--yellow); font-weight: 600; font-size: 12px; margin-bottom: 2px; }
.approval-tool .tool-cmd { color: var(--fg-bright); white-space: pre-wrap; word-break: break-all; }
.approval-tool .tool-cmd .dollar { color: var(--green); }
.approval-tool .tool-diff { white-space: pre-wrap; font-size: 12px; margin-top: 4px; }
.approval-tool .tool-diff .diff-del { color: var(--red); }
.approval-tool .tool-diff .diff-add { color: var(--green); }
.approval-tool .tool-diff .diff-warn { color: var(--yellow); }
.approval-prompt { padding: 8px 12px; font-size: 12px; border-top: 1px solid var(--border); background: var(--bg-highlight); }
.approval-actions { display: flex; gap: 6px; margin-bottom: 6px; }
.approval-btn { background: var(--bg); border: 1px solid var(--border); color: var(--fg-bright); border-radius: 4px; padding: 4px 12px; font-family: inherit; font-size: 12px; cursor: pointer; display: inline-flex; align-items: center; gap: 4px; }
.approval-btn:hover { background: var(--bg-highlight); }
.approval-btn .key { display: inline-block; background: var(--bg-surface); border: 1px solid var(--border); border-radius: 3px; padding: 0 4px; font-size: 11px; color: var(--accent); font-weight: 600; min-width: 18px; text-align: center; }
.btn-approve:hover { border-color: var(--green); color: var(--green); }
.btn-deny:hover { border-color: var(--red); color: var(--red); }
.btn-always:hover { border-color: var(--accent); color: var(--accent); }
.approval-feedback-input { width: 100%; background: var(--bg); color: var(--fg); border: 1px solid var(--border); border-radius: 4px; padding: 4px 8px; font-family: inherit; font-size: 12px; outline: none; }
.approval-feedback-input:focus { border-color: var(--accent); }
.approval-feedback-input::placeholder { color: var(--fg-dim); }
.approval-badge { padding: 6px 12px; font-size: 12px; font-weight: 600; border-top: 1px solid var(--border); }
.approval-badge.badge-approved { color: var(--green); }
.approval-badge.badge-denied { color: var(--red); }
.tool-output { padding: 6px 12px; background: var(--code-bg); border-top: 1px solid var(--border); white-space: pre-wrap; font-size: 12px; color: var(--fg-dim); max-height: 300px; overflow-y: auto; }
.tool-output.collapsed { max-height: 150px; position: relative; }
.tool-output.collapsed::after { content: 'click to expand'; position: absolute; bottom: 0; left: 0; right: 0; text-align: center; padding: 4px; background: linear-gradient(transparent, var(--code-bg) 60%); color: var(--fg-dim); font-size: 11px; cursor: pointer; }
/* Plan review dialog */
#plan-overlay { display: none; position: fixed; inset: 0; background: rgba(0,0,0,0.6); z-index: 100; justify-content: center; align-items: center; }
#plan-overlay.active { display: flex; }
#plan-dialog { background: var(--bg-surface); border: 1px solid var(--border); border-radius: 12px; padding: 20px; max-width: 700px; width: 90%; max-height: 80vh; overflow-y: auto; }
#plan-dialog h3 { color: var(--accent); margin-bottom: 12px; font-size: 15px; }
#plan-content { white-space: pre-wrap; font-size: 13px; color: var(--fg-bright); margin-bottom: 16px; max-height: 50vh; overflow-y: auto; background: var(--code-bg); padding: 12px; border-radius: var(--radius); }
#plan-feedback { width: 100%; background: var(--bg); color: var(--fg); border: 1px solid var(--border); border-radius: var(--radius); padding: 8px; font-family: inherit; font-size: 13px; margin-bottom: 12px; }
#plan-buttons { display: flex; gap: 8px; justify-content: flex-end; }
#plan-buttons button { padding: 8px 20px; border: none; border-radius: var(--radius); font-family: inherit; font-size: 13px; cursor: pointer; font-weight: 600; }
#btn-plan-approve { background: var(--green); color: var(--bg); }
#btn-plan-reject { background: var(--red); color: var(--bg); }
/* Scrollbar */
::-webkit-scrollbar { width: 8px; }
::-webkit-scrollbar-track { background: var(--bg); }
::-webkit-scrollbar-thumb { background: var(--border); border-radius: 4px; }
::-webkit-scrollbar-thumb:hover { background: var(--fg-dim); }
/* Focus indicators */
:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
#input-area textarea:focus-visible { outline: none; border-color: var(--accent); box-shadow: 0 0 0 2px color-mix(in srgb, var(--accent) 30%, transparent); }
.approval-btn:focus-visible { outline-offset: 1px; }
/* Empty state */
.empty-state { color: var(--fg-dim); text-align: center; padding: 48px 16px; font-size: 13px; }
/* Disconnected status */
#status-bar.disconnected { color: var(--red); }
/* Dashboard overlay */
.dashboard-overlay { display:none; position:fixed; inset:0; background:var(--bg); z-index:50; overflow-y:auto; }
.dashboard-overlay.active { display:flex; justify-content:center; align-items:flex-start; }
.dashboard-content { width:100%; max-width:960px; padding:32px 16px 24px; }
.dashboard-input-row { display:flex; gap:8px; margin-bottom:24px; }
.dashboard-input { flex:1; background:var(--bg-surface); color:var(--fg); border:1px solid var(--border); border-radius:var(--radius); padding:12px 16px; font:inherit; font-size:15px; outline:none; }
.dashboard-input:focus { border-color:var(--accent); box-shadow:0 0 0 2px color-mix(in srgb,var(--accent) 30%,transparent); }
.dashboard-input::placeholder { color:var(--fg-dim); }
.dashboard-new-btn { background:var(--accent); color:var(--bg); border:none; border-radius:var(--radius); padding:12px 20px; font:inherit; font-size:14px; font-weight:600; cursor:pointer; white-space:nowrap; }
.dashboard-new-btn:hover { opacity:0.9; }
.dashboard-new-btn:disabled { opacity:0.4; cursor:not-allowed; }
.dashboard-section { margin-bottom:24px; }
.dashboard-section-title { font-size:12px; text-transform:uppercase; letter-spacing:0.05em; color:var(--fg-dim); margin-bottom:10px; font-weight:600; }
.dashboard-cards { display:grid; grid-template-columns:repeat(auto-fill,minmax(200px,1fr)); gap:10px; }
.dashboard-card { background:var(--bg-surface); border:1px solid var(--border); border-radius:var(--radius); padding:12px 14px; cursor:pointer; transition:border-color 0.15s; }
.dashboard-card:hover { border-color:var(--accent); }
.dashboard-card:active { background:var(--bg-highlight); border-color:var(--accent); }
.dashboard-card:focus-visible { outline:2px solid var(--accent); outline-offset:2px; }
.dashboard-card .card-title { font-size:13px; color:var(--fg-bright); font-weight:600; margin-bottom:4px; overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
.dashboard-card .card-meta { font-size:11px; color:var(--fg-dim); }
.dashboard-empty { color:var(--fg-dim); font-size:13px; padding:8px 0; }
/* Dashboard header bar */
.dash-header { display:flex; justify-content:space-between; align-items:center; padding:8px 16px; background:var(--code-bg); border-radius:var(--radius) var(--radius) 0 0; }
.dash-header-title { color:var(--accent); font-size:12px; font-weight:bold; letter-spacing:0.05em; }
.dash-header-summary { color:var(--fg-dim); font-size:11px; }
/* Dashboard column headers */
.dash-colheaders { display:grid; grid-template-columns:var(--dash-grid); padding:4px 16px; background:var(--bg-surface); border-bottom:1px solid var(--border); font-size:11px; color:var(--fg-dim); text-transform:uppercase; letter-spacing:0.03em; }
.dash-col-tokens,.dash-col-ctx { text-align:right; }
/* Dashboard table */
.dash-table { min-height:40px; }
.dash-row { position:relative; border-left:3px solid transparent; cursor:pointer; transition:background 0.15s; }
.dash-row:nth-child(odd) { background:var(--bg); }
.dash-row:nth-child(even) { background:var(--bg-surface); }
.dash-row:hover { background:var(--bg-highlight); box-shadow:inset 0 0 0 1px var(--border); }
.dash-row:focus-visible { outline:2px solid var(--accent); outline-offset:-2px; }
.dash-row[data-state="running"] { border-left-color:var(--green); }
.dash-row[data-state="thinking"] { border-left-color:var(--accent); }
.dash-row[data-state="attention"] { border-left-color:var(--yellow); }
.dash-row[data-state="idle"] { border-left-color:var(--fg-dim); opacity:0.7; }
.dash-row[data-state="error"] { border-left-color:var(--red); }
.dash-row-main { display:grid; grid-template-columns:var(--dash-grid); padding:8px 16px 2px; align-items:center; font-size:12px; }
.dash-row-sub { padding:0 16px 8px 88px; font-size:11px; color:var(--fg-dim); white-space:nowrap; overflow:hidden; text-overflow:ellipsis; }
.dash-row-sub.sub-attention { color:var(--yellow); }
/* State indicator cell */
.dash-cell-state { display:flex; align-items:center; gap:6px; font-size:11px; }
.dash-state-dot { width:6px; height:6px; border-radius:50%; flex-shrink:0; }
.dash-state-dot[data-state="running"] { background:var(--green); border-radius:2px; animation:pulse 2s infinite; }
.dash-state-dot[data-state="thinking"] { background:var(--accent); animation:pulse 2.2s infinite; }
.dash-state-dot[data-state="attention"] { background:var(--yellow); border-radius:1px; transform:rotate(45deg); animation:pulse 1.8s infinite; }
.dash-state-dot[data-state="idle"] { background:var(--fg-dim); }
.dash-state-dot[data-state="error"] { background:var(--red); border-radius:0; }
.dash-state-label { white-space:nowrap; }
.dash-state-label[data-state="running"] { color:var(--green); }
.dash-state-label[data-state="thinking"] { color:var(--accent); }
.dash-state-label[data-state="attention"] { color:var(--yellow); }
.dash-state-label[data-state="idle"] { color:var(--fg-dim); }
.dash-state-label[data-state="error"] { color:var(--red); }
/* Table cells */
.dash-cell-name { font-weight:bold; color:var(--fg-bright); overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
.dash-row[data-state="idle"] .dash-cell-name { color:var(--fg-dim); }
.dash-cell-node { color:var(--fg-dim); font-size:11px; overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
.dash-cell-task { color:var(--fg-bright); overflow:hidden; text-overflow:ellipsis; white-space:nowrap; }
.dash-row[data-state="idle"] .dash-cell-task { color:var(--fg-dim); }
.dash-cell-tokens { text-align:right; color:var(--fg-dim); font-size:11px; }
.dash-cell-ctx { text-align:right; font-size:11px; }
.dash-cell-ctx.ctx-low { color:var(--green); }
.dash-cell-ctx.ctx-mid { color:var(--yellow); }
.dash-cell-ctx.ctx-high { color:var(--red); }
.dash-cell-ctx.ctx-danger { color:var(--red); font-weight:bold; }
.dash-cell-ctx.ctx-idle { color:var(--fg-dim); }
/* Dashboard footer */
.dash-footer { display:flex; justify-content:space-between; align-items:center; padding:8px 16px; background:var(--code-bg); border-top:1px solid var(--border); border-radius:0 0 var(--radius) var(--radius); font-size:11px; margin-bottom:24px; }
.dash-footer-nodes { color:var(--fg-dim); display:flex; align-items:center; gap:6px; }
.dash-footer-node-dot { width:6px; height:6px; border-radius:50%; background:var(--green); display:inline-block; flex-shrink:0; }
.dash-footer-stats { color:var(--fg-dim); }
/* Dashboard responsive */
@media (max-width:700px) {
:root { --dash-grid:68px 110px 1fr 56px 44px; }
.dash-col-node,.dash-cell-node { display:none; }
.dash-row-sub { padding-left:76px; }
}
@media (max-width:600px) {
.dashboard-input-row { flex-direction:column; }
.dashboard-new-btn { width:100%; }
}
@media (max-width:480px) {
.dashboard-content { padding:24px 12px 16px; }
.dashboard-cards { grid-template-columns:1fr; }
:root { --dash-grid:50px 1fr 50px; }
.dash-col-node,.dash-cell-node,.dash-col-task,.dash-cell-task,.dash-col-ctx,.dash-cell-ctx { display:none; }
.dash-row-sub { padding-left:66px; }
}
/* Login overlay */
#login-overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.7); display: flex; align-items: center; justify-content: center; z-index: 1000; }
#login-box { background: var(--bg-surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 32px; width: 320px; max-width: 90vw; box-shadow: 0 8px 32px rgba(0,0,0,0.4); }
#login-box h2 { color: var(--accent); font-size: 16px; margin-bottom: 16px; }
#login-box input { width: 100%; padding: 10px 12px; background: var(--bg); border: 1px solid var(--border); border-radius: 4px; color: var(--fg); font: inherit; font-size: 13px; margin-bottom: 12px; }
#login-box input:focus-visible { border-color: var(--accent); outline: none; box-shadow: 0 0 0 2px color-mix(in srgb, var(--accent) 30%, transparent); }
#login-box input::placeholder { color: var(--fg-dim); }
#login-box button { width: 100%; padding: 12px; background: var(--accent); color: var(--bg); border: none; border-radius: 4px; font: inherit; font-size: 13px; font-weight: 600; cursor: pointer; }
#login-box button:hover { opacity: 0.9; }
#login-box button:focus-visible { outline: 2px solid var(--fg); outline-offset: 2px; }
#login-box button:disabled { opacity: 0.5; cursor: not-allowed; }
#login-error { color: var(--red); font-size: 12px; margin-bottom: 8px; display: none; }
.sr-only { position: absolute; width: 1px; height: 1px; padding: 0; margin: -1px; overflow: hidden; clip: rect(0,0,0,0); white-space: nowrap; border: 0; }
@media (max-width: 380px) { #login-box { padding: 24px 20px; } }
/* Keyboard shortcuts overlay */
#kb-overlay { position: fixed; inset: 0; background: rgba(0,0,0,0.7); display: flex; align-items: center; justify-content: center; z-index: 999; }
#kb-box { background: var(--bg-surface); border: 1px solid var(--border); border-radius: var(--radius); padding: 24px 28px; width: 360px; max-width: 90vw; max-height: 80vh; overflow-y: auto; box-shadow: 0 8px 32px rgba(0,0,0,0.4); }
#kb-box h2 { color: var(--accent); font-size: 14px; margin-bottom: 14px; }
.kb-row { display: flex; justify-content: space-between; padding: 4px 0; font-size: 12px; }
.kb-key { color: var(--fg-bright); background: var(--bg-highlight); border: 1px solid var(--border); border-radius: 3px; padding: 1px 6px; font-family: inherit; font-size: 11px; white-space: nowrap; }
.kb-desc { color: var(--fg-dim); }
.kb-section { color: var(--fg-dim); font-size: 11px; text-transform: uppercase; letter-spacing: 0.05em; margin-top: 12px; margin-bottom: 4px; }
.kb-section:first-child { margin-top: 0; }
#kb-box .kb-hint { color: var(--fg-dim); font-size: 11px; text-align: center; margin-top: 14px; }