Files
turnstone/docs/docker.md
T
Patrick Buckley a6e929b0a0 Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24)

Bidirectional channel adapter framework connecting external messaging
platforms to turnstone workstreams via Redis MQ. Discord ships as the
first adapter; the protocol supports future Slack/Teams integrations.

Channel framework:
- ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping
- AsyncRedisBroker with single dispatch loop and per-channel ordered workers
- channel_routes table (migration 003) for persistent route storage
- 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD)
- Unified turnstone-channel gateway entry point, loads adapters by config
- Message chunking, approval formatting, plan review formatting

Discord adapter:
- discord.py v2.4+ bot with thread-per-@mention model
- Slash commands: /link (modal), /unlink, /ask, /status, /close
- Persistent button views for tool approval and plan review
- Streaming responses via edit-in-place (1.5s interval)
- Stale route detection and atomic session resume via resume_session field
- SessionResumedEvent confirmation back to channel
- Auto-approve support (blanket + per-tool list)

Atomic session resume:
- resume_session field on CreateWorkstreamMessage for single-request resume
- Server resumes session during POST /v1/api/workstreams/new atomically
- Bridge emits SessionResumedEvent to per-workstream channel
- WorkstreamCreatedEvent extended with resumed/session_id/message_count
- Server UI dashboardResumeSession simplified to single request
- Pruned sessions fall back gracefully to fresh start

Service auth:
- Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET
- Bridge: approve scope (1 week). Console collector: read. Proxy: write.

Console admin:
- Channels tab with per-user view, force-link modal, unlink
- 3 admin API endpoints for channel user management
- Styled confirm modals replacing browser confirm() dialogs

Bug fixes:
- AsyncRedisBroker: replaced per-channel listener tasks with single
  dispatch loop + per-channel queue workers (fixes message stealing race)
- Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates
- Bridge: _active_sends tracked for initial messages (fixes missing
  TurnCompleteEvent and unfinalized streaming messages)
- Bridge: HTTP calls moved outside lock scope in approval handlers
- Bridge: _handle_send cleans up _active_sends on HTTP/server errors
- Formatter: reads server SSE format (func_name/preview) with fallback

Docs, SDK, tests:
- docs/channels.md setup guide, architecture diagram 16
- Updated api-reference.md, architecture.md, console.md, docker.md
- Python SDK: resume_session param on create_workstream (async + sync)
- TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces
- OpenAPI schema: resume_session request, resumed/message_count response
- 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing,
  18 discord, 12 resume flow) — 1120 total passing

* Fix CI lint/typecheck failures and address Copilot review feedback (#24)

Lint: fix import ordering, remove unused imports, use contextlib.suppress.
Mypy: explicit postgresql dialect import, add discord module overrides for
optional-dependency CI environments.
Copilot: fix double-escaping in admin confirm modals, return resolved
session_id from server resume response, fix channel_routes diagram schema,
use atomic setdefault for routing locks, add post-insert race guard in
admin channel create, support SSE format in auto-approve check, update
identity linking note in architecture diagram.

* Fix remaining mypy call-arg errors for discord.py optional dependency

Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class
definitions that fail when discord.py is not installed in CI.
2026-03-04 13:02:58 -08:00

6.0 KiB

Docker Deployment

Docker Compose stack for running the full turnstone platform or the simulator.

Quick Start

# Copy and edit environment config
cp .env.example .env

# Full stack (needs an LLM API on the host)
docker compose up

# Simulator only (no LLM needed)
docker compose --profile sim up redis console sim

Console dashboard: http://localhost:8090

See also: Deployment diagram

Services

Service Port Profile Description
redis 6379 default Message broker, pub/sub, node registry
server 8080 default Web UI + chat workstreams + LLM
bridge default Redis-to-HTTP bridge (multi-node routing)
console 8090 default Cluster dashboard
channel production Channel gateway (Discord, Slack, etc.)
sim sim Multi-node cluster simulator

Profiles

Default (no flag) — starts redis, server, bridge, console. Requires an OpenAI-compatible LLM API running on the host (default: http://localhost:8000/v1).

docker compose up

Production — adds PostgreSQL and the channel gateway. Requires POSTGRES_PASSWORD and (for Discord) TURNSTONE_DISCORD_TOKEN:

docker compose --profile production up

Sim — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:

# Sim + console (no LLM needed)
docker compose --profile sim up redis console sim

# Everything including sim
docker compose --profile sim up

Configuration

All configuration is via environment variables in .env (copy from .env.example):

LLM Backend

Variable Default Description
LLM_BASE_URL http://host.docker.internal:8000/v1 OpenAI-compatible API URL
OPENAI_API_KEY dummy API key (dummy for local servers)
TAVILY_API_KEY Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search)

Redis

Variable Default Description
REDIS_PASSWORD Redis auth password (empty = no auth)
REDIS_PORT 6379 Host port mapping

Server

Variable Default Description
SERVER_PORT 8080 Host port mapping
SKIP_PERMISSIONS Set to any value to auto-approve all tools

Console

Variable Default Description
CONSOLE_PORT 8090 Host port mapping
CONSOLE_POLL_INTERVAL 10 Node polling interval (seconds)

Auth

Variable Default Description
TURNSTONE_AUTH_ENABLED Set to 1 to require authentication
TURNSTONE_AUTH_TOKEN Config-file token for server/bridge/console (backward compat, works alongside JWT)
TURNSTONE_JWT_SECRET Secret key for signing JWTs (required when using user identity / JWT auth)

Database

Variable Default Description
TURNSTONE_DB_BACKEND sqlite Storage backend: sqlite or postgresql
TURNSTONE_DB_URL Database URL (e.g. postgresql://user:pass@db:5432/turnstone). For SQLite, defaults to /data/.turnstone.db

The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.

First-time setup: After deploying with auth enabled, create an initial admin user by running turnstone-admin create-user inside the container:

docker compose exec server turnstone-admin create-user --username admin --name "Admin"

You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass --token --scopes read,write,approve to also generate an initial API token.

Channel Gateway

Variable Default Description
TURNSTONE_DISCORD_TOKEN Discord bot token (required to enable Discord adapter)
TURNSTONE_DISCORD_GUILD 0 Restrict to a single Discord guild (0 = all guilds)

The channel service runs in the production profile. When TURNSTONE_DISCORD_TOKEN is set, the Discord adapter connects to the Discord Gateway and routes messages through Redis MQ to the bridge and server. See Channel Integrations for full setup instructions including Discord application creation and user account linking.

Simulator

Variable Default Description
SIM_NODES 100 Number of simulated nodes
SIM_SCENARIO steady Scenario: steady, burst, node_failure, directed, lifecycle
SIM_DURATION 60 Duration in seconds
SIM_MPS 5.0 Messages per second (steady scenario)
SIM_LOG_LEVEL INFO Log verbosity
SIM_SEED Random seed for reproducibility
SIM_METRICS_FILE Write JSON report to file

Scaling

Scale to multiple server/bridge pairs:

docker compose up --scale server=3 --scale bridge=3

Each bridge auto-generates a unique node ID from its container hostname. When scaling server, remove the host port mapping (or use a reverse proxy) to avoid port conflicts.

Volumes

Volume Mount Purpose
redis-data /data Redis persistence
turnstone-data /data SQLite database (.turnstone.db)

Building

The image uses a multi-stage Dockerfile:

# Build all services
docker compose build

# Rebuild without cache
docker compose build --no-cache

All entry points are installed in a single image: turnstone-server, turnstone-bridge, turnstone-console, turnstone-channel, turnstone-admin, turnstone-sim, turnstone-eval.

Cleanup

# Stop and remove containers
docker compose down

# Stop, remove containers and volumes
docker compose down -v