mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00

Files

T

Patrick Buckley 414eb52d67 feat: raise scaling limits for 1000-node clusters (#129 )

* feat: raise scaling limits for 1000-node clusters

Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.

Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers

Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling

Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster

New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)

Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.

* fix: add image tag to compose anchors to avoid redundant builds

All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.

* fix: address Copilot review feedback on scaling PR

- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor

* fix: add PostgreSQL env vars to cluster bridge anchor

Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.

* fix: address Copilot round 2 + PG connection exhaustion at startup

Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
  per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change

Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
  to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service

* fix readme

* fix: startup resilience for large clusters

Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.

Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.

Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).

Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service

* fix: replace silent error suppression with structured logging

Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.

memory.py (26 changes):
  Every storage operation now logs on failure. Previously the entire
  persistence facade had zero logging — messages, workstream state,
  and structured memories could silently stop being saved.

server.py:
  Usage recording failures now log at warning (was pass).
  Global SSE fan-out errors log at debug (was pass).

console/server.py:
  Config reload notification logs per-node failures at warning.
  Settings read fallbacks log at warning with the default value used.

auth.py:
  User existence check logs at warning (was pass).
  Setup rollback failures log at error (was suppress).
  OIDC state cleanup logs at debug (was suppress).

mcp_client.py:
  DB-managed MCP server list failure logs at warning (was pass).

collector.py:
  Node poll failure upgraded from debug to warning with exc_info.
  Health fetch failure logs at debug with exc_info (was silent).

bridge.py:
  Best-effort plan rejection logs at warning (was suppress).
  Malformed SSE data logs at debug (was suppress).

session.py:
  Tool output UI callback failure logs at debug (was suppress).

* fix: stagger collector poll with deterministic per-node jitter

Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.

Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.

* fix: add startup jitter to bridge heartbeat and health monitor probe

Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.

Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.

Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.

* fix: split collector httpx timeout and raise keepalive pool

Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.

* fix: narrow detect_model return type for CLI and eval callers

detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.

2026-03-19 04:53:11 -07:00

7.0 KiB

Raw Blame History

Docker Deployment

Docker Compose stack for running the full turnstone platform or the simulator.

Quick Start

# Copy and edit environment config
cp .env.example .env

# Full stack (needs an LLM API on the host)
docker compose up

# Simulator only (no LLM needed)
docker compose --profile sim up redis console sim

Console dashboard: http://localhost:8090

See also: Deployment diagram

Services

Service	Port	Profile	Description
`redis`	6379	default	Message broker, pub/sub, node registry
`server`	8080	default	Web UI + chat workstreams + LLM
`bridge`	—	default	Redis-to-HTTP bridge (multi-node routing)
`console`	8090	default	Cluster dashboard
`channel`	—	production	Channel gateway (Discord, Slack, etc.)
`server-1`…`server-10`	—	cluster	10-node server fleet (PostgreSQL required)
`bridge-1`…`bridge-10`	—	cluster	Matching bridge fleet
`sim`	—	sim	Multi-node cluster simulator

Profiles

Default (no flag) — starts redis, server, bridge, console. Requires an OpenAI-compatible LLM API running on the host (default: http://localhost:8000/v1).

docker compose up

Production — adds PostgreSQL and the channel gateway. Requires POSTGRES_PASSWORD and (for Discord) TURNSTONE_DISCORD_TOKEN:

docker compose --profile production up

Cluster — 10-node server/bridge fleet sharing PostgreSQL and Redis. Access all nodes via the console at :8090. Requires POSTGRES_PASSWORD:

docker compose --profile cluster up

Sim — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:

# Sim + console (no LLM needed)
docker compose --profile sim up redis console sim

# Everything including sim
docker compose --profile sim up

Configuration

All configuration is via environment variables in .env (copy from .env.example):

LLM Backend

Variable	Default	Description
`LLM_BASE_URL`	`http://host.docker.internal:8000/v1`	OpenAI-compatible API URL
`OPENAI_API_KEY`	`dummy`	API key (`dummy` for local servers)
`TAVILY_API_KEY`	—	Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search)

Redis

Variable	Default	Description
`REDIS_PASSWORD`	—	Redis auth password (empty = no auth)
`REDIS_PORT`	`6379`	Host port mapping

Server

Variable	Default	Description
`SERVER_PORT`	`8080`	Host port mapping
`SKIP_PERMISSIONS`	—	Set to any value to auto-approve all tools

Console

Variable	Default	Description
`CONSOLE_PORT`	`8090`	Host port mapping
`CONSOLE_POLL_INTERVAL`	`10`	Node polling interval (seconds)

Auth

Variable	Default	Description
`TURNSTONE_AUTH_ENABLED`	—	Set to `1` to require authentication
`TURNSTONE_AUTH_TOKEN`	—	Config-file token for server/bridge/console (backward compat, works alongside JWT)
`TURNSTONE_JWT_SECRET`	—	Secret key for signing JWTs (required when using user identity / JWT auth)

Database

Variable	Default	Description
`TURNSTONE_DB_BACKEND`	`sqlite`	Storage backend: `sqlite` or `postgresql`
`TURNSTONE_DB_URL`	—	Database URL (e.g. `postgresql://user:pass@db:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db`
`TURNSTONE_DB_POOL_SIZE`	`2`	PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max)

The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.

Large clusters: Each turnstone process maintains a small connection pool (5 max). At hundreds of nodes this adds up — use PgBouncer in transaction pooling mode between turnstone and PostgreSQL.

First-time setup: After deploying with auth enabled, create an initial admin user by running turnstone-admin create-user inside the container:
docker compose exec server turnstone-admin create-user --username admin --name "Admin"
You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass --token --scopes read,write,approve to also generate an initial API token.

Channel Gateway

Variable	Default	Description
`TURNSTONE_DISCORD_TOKEN`	—	Discord bot token (required to enable Discord adapter)
`TURNSTONE_DISCORD_GUILD`	`0`	Restrict to a single Discord guild (0 = all guilds)

The channel service runs in the production profile. When TURNSTONE_DISCORD_TOKEN is set, the Discord adapter connects to the Discord Gateway and routes messages through Redis MQ to the bridge and server. See Channel Integrations for full setup instructions including Discord application creation and user account linking.

Simulator

Variable	Default	Description
`SIM_NODES`	`100`	Number of simulated nodes
`SIM_SCENARIO`	`steady`	Scenario: `steady`, `burst`, `node_failure`, `directed`, `lifecycle`
`SIM_DURATION`	`60`	Duration in seconds
`SIM_MPS`	`5.0`	Messages per second (steady scenario)
`SIM_LOG_LEVEL`	`INFO`	Log verbosity
`SIM_SEED`	—	Random seed for reproducibility
`SIM_METRICS_FILE`	—	Write JSON report to file

Scaling

For multi-node testing, use the cluster profile which provides 10 dedicated server+bridge pairs with unique node IDs (node-1 through node-10), resource limits, and shared PostgreSQL:

POSTGRES_PASSWORD=secret docker compose --profile cluster up

The default server and bridge also run alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at :8090.

For production clusters beyond ~50 nodes, add PgBouncer between turnstone services and PostgreSQL. See PgBouncer Connection Pooling for Docker Compose and Helm configuration.

Volumes

Volume	Mount	Purpose
`redis-data`	`/data`	Redis persistence
`turnstone-data`	`/data`	SQLite database (`.turnstone.db`)

Building

The image uses a multi-stage Dockerfile:

# Build all services
docker compose build

# Rebuild without cache
docker compose build --no-cache

All entry points are installed in a single image: turnstone-server, turnstone-bridge, turnstone-console, turnstone-channel, turnstone-admin, turnstone-sim, turnstone-eval.

Cleanup

# Stop and remove containers
docker compose down

# Stop, remove containers and volumes
docker compose down -v

7.0 KiB Raw Blame History