mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
Patrick Buckley
9be155b97a
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup - Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files - Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow - Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems - Refresh README and docs with badges, diagram links, and current descriptions - Refactor test_server_live.py with mock streaming helpers for deterministic CI testing - Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2) Console dashboard: - Move state indicators from top cards to fixed bottom status bar with cluster metrics - Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000) - Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent, LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance - Add render cache, stale indicator, active filter highlight, loading states Server web UI: - Apply matching Instrument Panel aesthetic for visual consistency with console - Fix branding (pcode → turnstone), extract inline styles to CSS classes - Rename pcode localStorage keys and history state to turnstone Legacy cleanup: - Remove persona-model-specific --persona flag and /persona slash command - Remove model_identity from chat_template_kwargs (vLLM-specific mechanism) - Refactor plan agent to use standard developer message instead of model_identity - Remove dead code (unused date/has_tools variables, noqa suppressions) * Fix CI typecheck: add mypy overrides for optional sympy/numpy imports The math sandbox optionally imports sympy and numpy at runtime (try/except ImportError). In CI these packages are not installed, so mypy raises import-not-found rather than import-untyped. Add mypy overrides to ignore missing imports for these optional dependencies. * Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity - Change #node-table from role="tree" to role="list" and group elements from role="treeitem" to role="listitem" (proper ARIA semantics) - Include currentView and currentFilter.state in renderStatusBar cache key so active pill highlight updates when switching views - Align pulse animation to 0.35 opacity (already applied in CSS)
205 lines
6.4 KiB
Markdown
205 lines
6.4 KiB
Markdown
# Cluster Simulator
|
||
|
||
The simulator (`turnstone-sim`) creates lightweight simulated nodes that talk to a real Redis instance using the standard turnstone protocol. External observers — `TurnstoneClient`, `turnstone-console`, real bridges — see identical behavior. No LLM backend is needed.
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
pip install turnstone[sim]
|
||
|
||
# 10 nodes, steady load, 60 seconds
|
||
turnstone-sim --nodes 10 --scenario steady --duration 60 --mps 5
|
||
|
||
# 100 nodes via Docker
|
||
docker compose --profile sim up redis console sim
|
||
```
|
||
|
||
## How It Works
|
||
|
||
Each simulated node is an asyncio coroutine (not a thread or process), so 1000 nodes run efficiently on a single event loop. The simulator:
|
||
|
||
1. Registers nodes via Redis heartbeats (same keys as real bridges)
|
||
2. Accepts messages from per-node and shared inbound queues
|
||
3. Simulates LLM responses with configurable latency and token generation
|
||
4. Simulates tool execution with configurable latency and failure rates
|
||
5. Publishes real protocol events (`ContentEvent`, `StateChangeEvent`, `TurnCompleteEvent`, etc.)
|
||
6. Reports latency, throughput, and utilization metrics at completion
|
||
|
||
```
|
||
TurnstoneClient → Redis Queue → SimNode → Redis Pub/Sub → TurnstoneClient
|
||
↓
|
||
turnstone-console (cluster dashboard)
|
||
```
|
||
|
||
## Scenarios
|
||
|
||
| Scenario | Description |
|
||
|----------|-------------|
|
||
| `steady` | Inject messages at a constant rate (`--mps`) for `--duration` seconds |
|
||
| `burst` | Push `--burst-size` messages instantly, then wait for completion |
|
||
| `node_failure` | Steady load + periodically kill nodes to test redistribution |
|
||
| `directed` | Send messages to specific nodes via `target_node` routing |
|
||
| `lifecycle` | Create, use, and close workstreams across nodes |
|
||
|
||
## CLI Reference
|
||
|
||
```
|
||
turnstone-sim [options]
|
||
```
|
||
|
||
### Cluster
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--nodes` | `10` | Number of simulated nodes |
|
||
|
||
### Scenario
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--scenario` | `steady` | Scenario name |
|
||
| `--duration` | `60` | Duration in seconds |
|
||
| `--mps` | `5.0` | Messages per second (steady) |
|
||
| `--burst-size` | `100` | Messages to send (burst) |
|
||
| `--node-kill-interval` | `15` | Seconds between kills (node_failure) |
|
||
| `--node-kill-count` | `1` | Nodes per kill cycle |
|
||
|
||
### Simulation
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--llm-latency` | `2.0` | Mean LLM response latency (seconds) |
|
||
| `--tool-latency` | `0.5` | Mean tool execution latency (seconds) |
|
||
| `--tool-failure-rate` | `0.02` | Tool failure probability (0.0–1.0) |
|
||
| `--seed` | — | Random seed for reproducibility |
|
||
|
||
### Redis
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--redis-host` | `localhost` | Redis host |
|
||
| `--redis-port` | `6379` | Redis port |
|
||
| `--redis-password` | — | Redis password |
|
||
| `--prefix` | `turnstone` | Redis key prefix |
|
||
|
||
### Output
|
||
|
||
| Flag | Default | Description |
|
||
|------|---------|-------------|
|
||
| `--metrics-file` | — | Write JSON report to file |
|
||
| `--log-level` | `INFO` | Log verbosity |
|
||
|
||
## Example: Load Testing
|
||
|
||
```bash
|
||
# 100 nodes, high throughput, 2 minutes
|
||
turnstone-sim --nodes 100 --scenario steady --duration 120 --mps 50
|
||
|
||
# Burst of 500 messages across 50 nodes
|
||
turnstone-sim --nodes 50 --scenario burst --burst-size 500 --duration 60
|
||
|
||
# Node failure resilience (kill 2 nodes every 10 seconds)
|
||
turnstone-sim --nodes 20 --scenario node_failure --duration 120 \
|
||
--node-kill-interval 10 --node-kill-count 2
|
||
|
||
# Fast simulation (low latency, no failures)
|
||
turnstone-sim --nodes 10 --scenario steady --duration 30 \
|
||
--llm-latency 0.1 --tool-latency 0.05 --tool-failure-rate 0 --mps 10
|
||
```
|
||
|
||
## Metrics Report
|
||
|
||
The simulator prints a summary at completion:
|
||
|
||
```
|
||
============================================================
|
||
SIMULATION REPORT
|
||
============================================================
|
||
Scenario: steady
|
||
Nodes: 100
|
||
Duration: 60.2s
|
||
Total turns: 295
|
||
Total errors: 5
|
||
Node kills: 0
|
||
------------------------------------------------------------
|
||
THROUGHPUT
|
||
Messages/sec: 4.97
|
||
Turns/sec: 4.89
|
||
------------------------------------------------------------
|
||
LATENCY (seconds)
|
||
p50: 3.21
|
||
p90: 5.44
|
||
p99: 8.12
|
||
mean: 3.56
|
||
max: 12.1
|
||
------------------------------------------------------------
|
||
UTILIZATION
|
||
Mean ws/node: 2.3
|
||
Max ws/node: 8
|
||
Idle nodes: 12
|
||
============================================================
|
||
```
|
||
|
||
Use `--metrics-file report.json` to write the full report as JSON.
|
||
|
||
## Console Integration
|
||
|
||
The simulator's nodes appear in `turnstone-console` exactly like real nodes. Run them together to see the dashboard populate with simulated workstreams:
|
||
|
||
```bash
|
||
# Terminal 1: start Redis and console
|
||
docker compose up redis console
|
||
|
||
# Terminal 2: run simulator
|
||
docker compose --profile sim up sim
|
||
```
|
||
|
||
Or all at once:
|
||
|
||
```bash
|
||
SIM_NODES=50 SIM_DURATION=120 docker compose --profile sim up redis console sim
|
||
```
|
||
|
||
Open http://localhost:8090 to see simulated nodes, workstream states, token counts, and load bars updating in real time.
|
||
|
||
## Architecture
|
||
|
||
> See also: [Simulator Architecture diagram](diagrams/png/10-simulator-architecture.png)
|
||
|
||
```
|
||
turnstone/sim/
|
||
├── __init__.py # Public API: SimCluster, SimConfig
|
||
├── config.py # SimConfig — all simulation parameters
|
||
├── engine.py # SimEngine — LLM + tool execution simulation
|
||
├── node.py # SimNode + SimWorkstream — protocol-compatible node
|
||
├── cluster.py # SimCluster + InboundDispatcher + PooledBroker
|
||
├── scenario.py # 5 scenario classes
|
||
├── metrics.py # MetricsCollector — latency, throughput, utilization
|
||
└── cli.py # CLI entry point
|
||
```
|
||
|
||
**Key design:** The `InboundDispatcher` batches ~50 node queues into a single Redis `BLPOP` call, keeping connection count bounded at ~20 regardless of node count. All nodes share a single `ConnectionPool(max_connections=64)`.
|
||
|
||
## Programmatic Use
|
||
|
||
```python
|
||
import asyncio
|
||
from turnstone.sim import SimCluster, SimConfig
|
||
|
||
async def main():
|
||
config = SimConfig(
|
||
num_nodes=10,
|
||
scenario="steady",
|
||
duration=30,
|
||
messages_per_second=2.0,
|
||
llm_latency_mean=0.5,
|
||
)
|
||
cluster = SimCluster(config)
|
||
await cluster.start()
|
||
await cluster.run_scenario()
|
||
print(cluster.report())
|
||
await cluster.stop()
|
||
|
||
asyncio.run(main())
|
||
```
|