Compare commits

...

167 Commits

Author SHA1 Message Date
Patrick Buckley 52d59cf7b7 chore: bump version to 0.8.2 2026-03-17 02:20:44 -07:00
Patrick Buckley 2dc885ab4d fix: output guard detects single secret-bearing env lines (#115)
* fix: output guard detects single secret-bearing env lines

The credential leak check required 3+ env-style lines before flagging.
A single AWS_SECRET_ACCESS_KEY=... line was missed. Now flags whenever
any env line has a secret-bearing key name (SECRET, KEY, TOKEN,
PASSWORD, CREDENTIAL), regardless of how many total env lines exist.

* fix: tighten env secret key matching, add tests

Tighten _RE_ENV_SECRET_KEY to word-boundary segments so MONKEY/TURKEY
don't false-positive. Use any() for short-circuit. Add test for single
secret line detection and substring false-positive prevention.
2026-03-17 02:19:22 -07:00
Patrick Buckley 14488f43e0 feat: metacognitive nudge on tool error — search memories for guidance
Add tool_error nudge type that fires when a tool returns an error,
prompting the model to search memories for prior feedback about the
tool or error pattern before retrying.

- Gated on nudges config (respects nudges=false)
- Only fires when memories exist (no noise on fresh workstreams)
- Broad error detection: Error*, *error:*, Command timed out, Unknown tool
- Nudge wording aligned to memory(action='search') convention
- Respects existing cooldown (5 min) and rate limiting
- 4 new tests
2026-03-17 02:06:10 -07:00
Patrick Buckley 90f2070146 chore: bump version to 0.8.1 2026-03-17 01:28:14 -07:00
Patrick Buckley 1e551830ea fix: allow deleting installed (readonly) skills
Readonly guard should prevent editing content, not uninstalling.
Remove readonly check from admin_delete_skill so batch-installed
skills can be individually deleted. Enable delete button in UI
for all skills regardless of readonly flag.
2026-03-17 01:26:44 -07:00
Patrick Buckley 84cc212ecd ui: tighten category and risk columns (100px -> 80px) 2026-03-17 01:26:44 -07:00
Patrick Buckley da4025d338 ui: skills table — category first, risk column, remove variables
- Move category column before name
- Remove variables column (rarely useful in table view)
- Add dedicated RISK column with scan badge, unicode shape indicators
  (checkmark/triangle/diamond/warning), and multi-line tooltip showing
  composite score and flagged axes from scan report
- Risk badge is keyboard-focusable (tabindex=0) with aria-label
- Unscanned skills show em-dash placeholder at 40% opacity
- Balanced grid: 100px 1.5fr 100px 120px
- Risk + category hidden on mobile (<700px)
2026-03-17 01:26:44 -07:00
Patrick Buckley 88085c29ff fix: normalize install response + review fixes
Address 5 Copilot review items + code review findings:

- Normalize install endpoint to always return envelope response:
  {installed: [...], skipped: [...], total: N} — eliminates dual
  response shape (single SkillInfo vs batch). Breaking change to
  install endpoint response, SDKs and OpenAPI spec updated.
- Add SkillInstallResponse + SkillInstallSkipped Pydantic models
- POST /resources spec now correctly documents response_code=201
- SQLite count_skill_resources_bulk chunks IN clause at 900 to stay
  under SQLITE_MAX_VARIABLE_NUMBER (999)
- Fix installDiscoveredSkill() JS handler for envelope response
- Add error key to 409 duplicate response for error handler compat
- Update Python SDK install_skill return type (dict, not SkillInfo)
- Add TypeScript SkillInstallResponse + SkillInstallSkipped types
- Regenerate openapi-console.json
- Update all install tests for envelope response shape
2026-03-17 01:26:44 -07:00
Patrick Buckley 3152667a0c fix: update test_skill_sources for 5-tuple _parse_github_url
_parse_github_url now returns (owner, repo, branch, path, branch_explicit).
Update all test unpackings and add assertions for branch_explicit.
2026-03-17 01:26:44 -07:00
Patrick Buckley 4b44d88401 fix: harden batch skill install — 7 review items + OpenAPI snapshot
- Race condition: wrap create_prompt_template in try/except, append
  to skipped on conflict instead of crashing
- HTTP timeout: per-request timeout (10s+5s connect) instead of shared
  15s pool; parallelize SKILL.md and resource fetches with semaphore
  (5 concurrent)
- Branch detection: return branch_explicit from _parse_github_url(),
  eliminate duplicated regex matching and type: ignore comments
- Content-length: check len(resp.content) after fetch instead of
  unreliable content-length header; add size check in batch path
- Rate limits: _check_rate_limit() inspects x-ratelimit-remaining,
  raises actionable error on 403, warns when remaining < 10
- Root resources: fix _find_resource_files skipping root-level
  resources like scripts/foo.sh for root SKILL.md
- resource_count: pass accurate count in update and install responses
- Regenerate openapi-console.json with new resource endpoints
2026-03-17 01:26:44 -07:00
Patrick Buckley 8957b9ce0e feat: batch install skills from multi-skill GitHub repos
When a GitHub repo URL has no root SKILL.md (monorepo pattern like
anthropics/skills), automatically scan the repo tree for all SKILL.md
files and install every discovered skill in one operation.

- Add fetch_skills_from_github_repo() — scans recursive tree, parses
  each SKILL.md, collects per-skill resources via shared helpers
- Extract _find_resource_files() and _fetch_resource_contents() to
  eliminate duplication between single and batch fetch paths
- Extend admin_skill_install to fall back to batch scanning when
  single-skill fetch returns 404
- Each skill gets a specific source_url pointing to its subdirectory
- Backward compatible: single-skill repos return same response shape
- Frontend handles both shapes with contextual toast messages
- Filter tree scan to URL path subtree when path is provided
- Cap at 50 skills per repo scan

Also addresses review feedback:
- Fix path prefix check (scripts/ not scriptsX/)
- Use count_skill_resources_bulk for single skill GET
- Add content field to SkillResourceInfo schema
- Fix OpenAPI spec paths ({path} not {path:path})
- Check r.ok on resource upload promises
- Preserve / in URL-encoded paths (split/map/join pattern)
- URL-encode path in Python SDK delete method
- Fix test_install_not_found to mock batch fallback
- Fix test_search_empty_results for required q param
- Narrow except clause to ValueError in batch parser
2026-03-17 01:26:44 -07:00
Patrick Buckley 28a6b0dd33 feat: skill resources — API, admin UI, runtime injection, and SDK
Complete the resource surface for skills (scripts/, references/, assets/):

- 4 admin API endpoints: list, get, create, delete skill resources
- Storage: delete_skill_resource_by_path + count_skill_resources_bulk
- Admin UI: resource count badge in skills table, resource sections in
  create/edit modals with add/delete, readonly guard for installed skills
- Runtime: _load_skills populates skill resources, _init_system_messages
  injects <skill-resources> catalog (inlined if <8KB)
- Python SDK: list/create/delete_skill_resource (async + sync)
- TypeScript SDK: listSkillResources, createSkillResource, deleteSkillResource
- Path traversal protection (normpath + .. rejection + null byte check)
- Block empty skill discover searches (frontend toast + backend 400)
- Rename MCP "Registry" tab to "Discover" for consistency with skills
- Move Skills + MCP Servers into new "Extensions" sidebar group
- 25 tests (7 storage, 16 API + 2 security)
2026-03-17 01:26:44 -07:00
Patrick Buckley 7bc17cc072 fix: populate func_args for all tools in intent judge evaluation
The heuristic engine was seeing empty {} for web_fetch, web_search,
watch, notify, task, and load_skill — only bash, file ops, and MCP
tools had their arguments forwarded. The judge could not pattern-match
on URLs, queries, commands, or messages for these tools.
2026-03-16 19:33:16 -07:00
Patrick Buckley 10a1800492 chore: bump version to 0.8.0 2026-03-16 19:22:17 -07:00
Patrick Buckley 1010f163f0 feat: load_skill built-in tool — model-driven skill discovery and act… (#112)
* feat: load_skill built-in tool — model-driven skill discovery and activation

Two-action tool: 'search' finds skills by multi-word query with substring
matching on name/description/tags/category (auto-approved, read-only);
'load' activates a skill by name via set_skill() (requires approval).

Guards: filters disabled skills from search + load; short-circuits when
skill is already active; approval_label includes skill name for granular
tool policies (load_skill__<name>); main session only (excluded from
sub-agents). Logs storage errors in search path.

25 tests covering registration, preparer validation, executor logic,
disabled/already-active edge cases, multi-word queries, approval labels.

* refactor: use BM25 relevance ranking for load_skill search

Replace substring matching with BM25Index from turnstone/core/bm25.py,
matching the pattern used by memory relevance and tool search. Handles
multi-word queries, term frequency, and document length normalization.

* fix: address copilot review — BM25 tags parsing, primary_key, test cleanup

- Parse JSON tags into space-separated text before BM25 indexing so
  individual tag terms match queries (was passing raw '["foo","bar"]')
- Add primary_key: "name" to load_skill.json for PRIMARY_KEY_MAP
- Remove dead resolve_workstream patch from test helper
- Update diagram: "substring match" → "BM25 ranking"
2026-03-16 19:20:19 -07:00
Patrick Buckley c28bfc1e58 feat: skill discovery — search and install skills from external sources (#111)
* feat: skill discovery — search and install skills from external sources

Add discovery UI and API for finding and installing skills from
skills.sh registries and GitHub repositories with one-click install,
SKILL.md frontmatter parsing, and security scan integration.

Core modules:
- skill_parser.py: ParsedSkill dataclass, parse_skill_md() with YAML
  frontmatter support (Anthropic + Hermes tag formats), name validation
- skill_sources.py: SkillsShClient (async search + resolve),
  fetch_skill_from_github (SKILL.md + bundled resource fetching with
  256KB cap, text extension filter, GitHub API tree traversal)

API:
- GET /v1/api/admin/skills/discover — search with installed annotation
  and scan_status for installed skills
- POST /v1/api/admin/skills/install — fetch, parse, duplicate check,
  create with origin="source" readonly=true, store resources, audit

Also fixes pre-existing bug where _skill_to_response omitted scan_status,
scan_report, scan_version fields — scan tier badges in the installed
skills table were silently empty despite data existing in storage.

Admin UI: pill toggle (Installed/Discover), discovery cards with scan
tier badges, GitHub import modal with proper focus trap/Escape/backdrop,
scoped selectors preventing MCP↔Skills cross-tab state corruption.

SDK: discover_skills() + install_skill() on Python (async+sync) and
TypeScript console clients.

48 new tests across 3 test files. All 2632 tests pass.

* fix: address copilot review — 404 vs 502, O(n) lookups, branch fallback

- SkillNotFoundError subclass: install returns 404 when SKILL.md is
  missing, 502 only for connectivity/upstream errors
- get_skill_by_source_url() + list_installed_skill_urls(): indexed
  storage lookups replace O(n) full-table scans with content blobs
- Default branch fallback: tries main then master when URL doesn't
  specify a branch
- Path normalization: strip trailing slash once, remove redundant
  candidate
- SDK install_skill() returns typed SkillInfo with response_model
- Tree size guard: skip resource tree if response >2MB
2026-03-16 18:42:32 -07:00
Patrick Buckley e71ea38953 feat: output guard data pipeline — persist assessments, SSE events, a… (#110)
* feat: output guard data pipeline — persist assessments, SSE events, admin UI

Complete the output guard pipeline: persist assessments for v2 calibration,
surface warnings in every UI layer, and add scan badges to admin skills tab.

Storage: migration 022 adds output_assessments table (flags, risk_level,
annotations, output_length, redacted — raw output never stored) and
scan_version column on prompt_templates. Three new protocol methods with
SQLite + PostgreSQL implementations.

Server/CLI: on_output_warning now persists assessments fire-and-forget.
CLI shows flags, annotations, and redaction notice. Session emits on_info
warning when high/critical scan_status skill is loaded.

MQ: OutputWarningEvent dataclass + bridge SSE forwarding.

Web UI: output_warning SSE handler with inline warning rendering
(role="alert" for accessibility), semantic risk colors.

Console admin: scan badges on skills list (dedicated scope-scan-* CSS with
green/yellow/red risk vocabulary), scan report breakdown in edit modal with
4-axis scores, POST /admin/skills/{id}/rescan endpoint, GET
/admin/output-assessments endpoint with date-filtered pagination.

Security fixes: ReDoS in connection string regex ([^@\s]+ → [^:@\s]+),
negative limit bypass in all admin endpoints (max(1, ...)), to_dict()
excludes sanitized output by default.

False-positive fixes: credentials pattern anchored to path context,
env secret key check restricted to key portion only.

* fix: address PR #110 review — list redaction, test fixture, OpenAPI snapshot

Per-part redaction: evaluate each text part independently in structured
output instead of joining all parts and replacing only the first one.

Fix test annotations default from "{}" to "[]" matching schema.

Regenerate TypeScript OpenAPI snapshots for new admin endpoints.
2026-03-16 17:39:24 -07:00
Patrick Buckley 5378b33641 feat: output guard — evaluate tool results before they enter context (#109)
* feat: output guard — evaluate tool results before they enter context

Add turnstone/core/output_guard.py — a time-budgeted heuristic that
evaluates tool execution results after execution but before they
enter the conversation context window.

Priority-ordered detection (5s budget, highest priority first):
1. Prompt injection: override phrases, role injection, instruction
   override markers, meta-injection patterns
2. Credential leakage: API keys (OpenAI/GitHub/AWS/Google), PEM
   private key blocks, connection strings, .env secret format
3. Encoded payloads: script data URIs, hex shellcode sequences
4. Adversarial URLs: cloud metadata endpoints, credential query params
5. System info disclosure: private IPs, sensitive file paths

Annotates and optionally redacts (credentials → [REDACTED:<type>]).
Does NOT gate — surfaces warnings via on_output_warning callback.

Integration:
- Wired into session.py tool result loop via _evaluate_output()
- JudgeConfig gains output_guard + redact_secrets fields (both default true)
- SessionUI protocol gains on_output_warning callback
- 25 compiled regex patterns, pure function, no I/O

29 tests covering all detection categories, benign output false
positive checks, credential redaction, and time budget behavior.

* fix: address PR #109 review — protocol, config, and guard fixes

Copilot review feedback:
- Replace _CLEAN singleton with _clean() factory to prevent mutable
  shared state (OutputAssessment has list fields)
- Remove redundant second _CREDENTIAL_PATTERNS loop in _check_credentials
- Evaluate text parts of list outputs (images) not just string outputs
- Wire output_guard + redact_secrets through ConfigStore settings
  registry and _build_judge_config() so operators can configure via
  admin Settings tab
- Remove --no-output-guard CLI flag claim from docs (use Settings tab)

Typecheck fix:
- Add on_output_warning to all SessionUI implementations: NullUI
  (eval, 5 test files), WebUI (server — emits SSE event), TerminalUI
  (CLI — ANSI colored warning), RecordingUI, FakeUI
2026-03-16 16:22:10 -07:00
Patrick Buckley 9b605f81a3 feat: skill scanner — evaluate SKILL.md content at install time
Add turnstone/core/skill_scanner.py — a production content scanner
that evaluates skill risk across four axes:

1. Content risk: command execution, external downloads, credential
   handling, data exfiltration, eval/exec, sudo, browser automation
2. Supply chain risk: pipe-to-shell, transitive installs, obfuscation,
   download-exec chains, executable URLs from untrusted domains
3. Vulnerability risk: prompt injection (E004), insecure credential
   handling (W007), third-party content exposure (W011)
4. Declared capability risk: parsed from allowed_tools field —
   Bash(*) is high, Bash(git:*) is low, read-only tools are safe

Composite score with equal 25% weights per axis. Floor rule: any
single axis at critical forces composite to at least medium tier.

Wired into both SQLite and PostgreSQL storage backends:
- scan_skill() runs at create_prompt_template time
- Re-scan triggers on update when content or allowed_tools change
- Results populate the existing scan_status and scan_report columns
- Silent failure on scanner errors (never blocks skill creation)

Scanner helper factored into _utils.py (shared across backends).
23 unit tests covering tier classification, capability scoring,
negation filtering, floor rule, serialization, and trusted domains.
2026-03-16 15:45:21 -07:00
Patrick Buckley f05e6bddad feat(judge): enrich heuristic rules from 23 to 36 (#107)
* feat(judge): enrich heuristic rules from 23 to 36

Add 13 new pattern-based rules to the intent validation heuristic,
calibrated from analysis of 25K public agent skill security audits
across three independent auditors.

New critical: download-then-execute chains.
New high: browser+data export, transitive installs from untrusted
sources, control plane mutations (crontab, systemctl).
New medium: content ingestion pipelines (curl|python3), interpreter
execution (python3 script.py), cloud CLI mutations (az/gcloud/aws/
kubectl/terraform create/delete/destroy).
New low: tool_search, read_resource, web_search.

Fixes: crontab -l no longer false-positives, systemctl stop/disable
now flagged, az/gcloud subcommand patterns work correctly.

* fix(judge): address PR #107 review feedback

- content-ingestion: narrow second pattern to specific interpreters/
  processors (python3, node, ruby, perl, php, jq) instead of any word.
  Prevents false positives on read-only downstream (wget -O - | head).
- cloud-infra-mutation: split kubectl into its own pattern with specific
  verbs (apply, create, delete, scale, rollout, drain, cordon) to avoid
  false positive on resource types (kubectl get deploy).
- cloud-infra-mutation: split terraform/pulumi to specific verbs only
  (apply, destroy, import) — terraform plan no longer matches.
- control-plane-mutation: exclude -h and -V flags from crontab pattern
  alongside existing -l exclusion.
- Add 35 heuristic rule tests covering all 13 new rules with positive
  matches and negative (false-positive prevention) cases.
2026-03-16 15:09:36 -07:00
Patrick Buckley 75eda9a096 feat: unified skills system — merge prompt templates + workstream tem… (#106)
* feat: unified skills system — merge prompt templates + workstream templates

Evolves prompt_templates into a first-class skills entity and merges
workstream templates into the same model, collapsing two concepts into
one.

Migration 021: 21 new columns on prompt_templates (skills metadata,
security scan fields, session config from WS templates), skill_resources
table for bundled files, skill_versions table for auto-snapshot version
history. Data migration converts existing WS templates into skills with
name collision handling, migrates version history, renames workstreams
and scheduled_tasks columns, cleans orphaned permissions, drops old
tables.

Key changes:
- All public interfaces renamed: templates → skills (API, CLI, SDK, UI)
- Session config (model, temperature, token_budget, auto_approve, etc.)
  now lives on the skill and is applied at workstream creation
- /skill slash command, set_skill() API, --skill CLI flag
- BM25 skill search via SkillSearchManager for activation="search" skills
- Admin UI: Skills tab with collapsible Session Config section,
  description subtitles, activation/origin/MCP badges, pagination
- Shared validation helper (_parse_skill_session_config) for DRY CRUD
- Version history with auto-snapshot on every edit + API endpoint
- Cascade delete (resources + versions) on skill removal
- Security: range validation, activation allowlist, fail-closed enabled
  check, duplicate name 409, readonly guard, JSON validation
- 77 new tests across storage, runtime, search, API integration, and
  migration behavior verification (2521 total)

* fix: address Copilot review + rename admin.templates → admin.skills

- Skip skill lookup when resume_ws is set (avoids spurious 400)
- Fix _applied_skill_version mismatch (1 in both workstreams table and session)
- Remove stale template field from MQ protocol diagram
- Rename admin.templates permission to admin.skills everywhere (runtime,
  frontend, tests, docs) with migration step for persisted role data
- Fix stale /api/templates references in docs and diagrams
- Update docstrings/comments for skills terminology

* fix: address Copilot round 2 — skill version lineage + stale doc refs

- Compute actual skill version from skill_versions count (not hardcoded 1)
- Use same version in both workstreams table and session metadata
- Fix response payload example: "templates" → "skills" key
- Fix "Each template summary" → "Each skill summary"
2026-03-16 14:47:06 -07:00
renovate[bot] 4c00d71150 chore(deps): lock file maintenance (#105)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-16 02:18:02 -07:00
Patrick Buckley 80e1924d7f feat: enable prompt caching for Anthropic and OpenAI providers (#104)
* feat: enable prompt caching for Anthropic and OpenAI providers

Activate automatic prompt caching on both LLM providers to reduce input
token costs on multi-turn conversations. Anthropic gets cache_control:
ephemeral (90% savings on cache hits), OpenAI GPT-5.x gets 24h extended
cache retention (free). Cache metrics flow end-to-end through the entire
data pipeline: provider → session → server SSE → MQ protocol → storage →
Prometheus metrics → admin Usage tab.

- AnthropicProvider: top-level cache_control on all requests, extract
  cache_creation_input_tokens and cache_read_input_tokens from streaming
  and non-streaming responses
- OpenAIProvider: prompt_cache_retention=24h for GPT-5.x, extract
  cached_tokens from usage.prompt_tokens_details
- UsageInfo: new cache_creation_tokens and cache_read_tokens fields
- Migration 020: add cache columns to usage_events table
- Storage: record_usage_event and query_usage updated (sqlite + pg)
- Metrics: turnstone_tokens_total{type="cache_creation|cache_read"}
- Server: on_status passes cache tokens to SSE, storage, and metrics
- MQ: StatusEvent carries cache fields through bridge
- SDKs: Python and TypeScript StatusEvent types updated
- OpenAPI: UsageBreakdownItem schema includes cache fields
- Console UI: Usage tab shows cache write/read as secondary readouts
  with visual separator, dimmed when zero
- 16 new tests, docs and 3 diagrams updated

* fix: address Copilot review feedback

- Fix MQ protocol diagram clipping by switching to vertical package
  layout (inbound on top, outbound below) with package aliases
- Regenerate OpenAPI snapshots to include cache_creation_tokens and
  cache_read_tokens on UsageBreakdownItem
- Replace fragile MagicMock(spec=[]) + del pattern with
  types.SimpleNamespace in cache metrics missing-attributes test
2026-03-16 01:59:55 -07:00
Patrick Buckley 471bf89c8b Merge pull request #101 from turnstonelabs/feat/mcp-registry
feat: MCP Registry integration — discover and install servers from th…
2026-03-15 22:01:25 -07:00
Patrick Buckley 35785d3a0e fix: address round 2 Copilot feedback
- Pass table_name to op.drop_index in migration 019 downgrade for
  dialect portability
- Fix dedup comment accuracy (first occurrence wins, not highest version)
- Re-render registry cards on install failure to reset stuck
  "Installing..." button state
2026-03-15 21:53:07 -07:00
Patrick Buckley 2ff0cd8240 Merge pull request #103 from turnstonelabs/renovate/lock-file-maintenance
chore(deps): lock file maintenance
2026-03-15 21:47:29 -07:00
Patrick Buckley f9f0ff0b53 Merge pull request #102 from turnstonelabs/renovate/github-actions
chore(deps): update softprops/action-gh-release digest to 153bb8e
2026-03-15 21:47:26 -07:00
Patrick Buckley df8a36ced4 fix: add timeout to MCP server disconnect to prevent hung removals
stack.aclose() on a stuck streamable-http transport hangs indefinitely,
causing 50% CPU on all nodes when removing a broken remote server via
reconcile_sync. Wrap with asyncio.wait_for(timeout=10s) so cleanup
proceeds even if the transport refuses to close cleanly.
2026-03-15 21:42:05 -07:00
Patrick Buckley ef6cac6428 fix: address review feedback and add sync-pending indicator
Review fixes:
- Rename query param from `q` to `search` across endpoint, frontend,
  SDKs, OpenAPI spec, docs, and tests to match upstream registry API
- Validate variables/env/headers are dicts in install endpoint (400 on
  malformed input instead of 500)
- Block javascript: and unsafe URL schemes on repo and website links
  rendered from registry data (XSS prevention)
- Add roving tabindex to Servers/Registry pill toggle for correct
  keyboard focus behavior
- Add noreferrer to website link in detail modal

Sync-pending indicator:
- "Sync to Nodes" button pulses yellow after create/edit/delete/import
  to alert admin that nodes have unseen changes
- Clears after successful sync
- Reduced-motion safe
2026-03-15 21:41:02 -07:00
renovate[bot] f4c3b3a9c4 chore(deps): lock file maintenance 2026-03-16 04:11:10 +00:00
renovate[bot] d06db3feee chore(deps): update softprops/action-gh-release digest to 153bb8e 2026-03-16 04:10:39 +00:00
Patrick Buckley 50544c0d1b feat: MCP Registry integration — discover and install servers from the official registry
Backend: standalone MCPRegistryClient (httpx async) queries the official MCP
Registry API (registry.modelcontextprotocol.io, v0.1). Two new console admin
endpoints: GET /v1/api/admin/mcp-registry/search (proxy with installed-status
annotation, dedup, uninstallable server filtering) and POST
/v1/api/admin/mcp-registry/install (auto-reloads all cluster nodes). Migration
019 adds registry_name/version/meta columns to mcp_servers with partial unique
index. Configurable registry URL via mcp.registry_url setting for
enterprise/private registries. resolve_install_config() handles both remote
(streamable-http) and package (npm→npx, pypi→uvx) installs. Pydantic models,
OpenAPI spec, Python + TypeScript SDK methods.

Frontend: unified MCP admin tab with Servers/Registry pill toggle (ARIA
tablist). Servers view: tri-state source badges (CONFIG/MANUAL/REGISTRY).
Registry view: search bar with type filter (remote/npm/pypi), auto-browse on
tab switch, result cards with source-type badges and repo links, one-click
install for zero-config remotes, install modal with dynamic form for servers
needing env vars/headers/URL variables. Package install warning banner.
Post-install status polling with connection/error feedback toasts. Trust
notice banner linking to the official registry.

Safety: 30s connect timeout on streamablehttp_client and session.initialize()
prevents hung connections from blocking the MCP event loop indefinitely.
Required-only headers in install config prevents empty auth headers from
causing silent 401s.

71 new tests (registry client, API endpoints, storage columns). Docs:
dedicated docs/mcp-registry.md, updated api-reference, architecture, console,
sdk, settings docs. Updated MCP architecture diagram.
2026-03-15 21:09:27 -07:00
Patrick Buckley d1c484737f fix(ci): regenerate lockfile for v0.7.0 and exclude local package from pip-audit
uv lock --check fails after version bump because the lockfile is stale.
pip-audit --strict fails because turnstone 0.7.0 isn't on PyPI yet.

Fix: regenerate uv.lock, and audit only third-party deps via
uv export --no-emit-project piped to pip-audit -r.
2026-03-15 17:28:13 -07:00
Patrick Buckley c2c6689a8e chore: bump version to 0.7.0 2026-03-15 17:15:27 -07:00
Patrick Buckley f5af4875ba fix: surface MCP server errors in admin UI instead of silent logging (#100)
* fix: surface MCP server errors in admin UI instead of silent logging

get_server_status() hardcoded error="" — connection and refresh failures
were logged but never surfaced to the admin panel.

Added _last_error dict to MCPClientManager: set on failure (connect,
refresh, periodic refresh, notification handler), cleared on success,
cleaned up on remove. Read in get_server_status().

Admin UI: error tooltip on list row status span, error text in red
in detail modal per-node list. Schema already had the field.

6 new tests for error tracking lifecycle.

* feat: add turnstone_mcp_server_errors Prometheus gauge

Exposes the count of MCP servers currently in error state via
/metrics for alerting and reliability tracking.

* fix: address copilot review — sanitize error strings, clear on notification success

- Add _set_error() helper: strips newlines, truncates to 256 chars
- All error-setting sites now use _set_error() for consistent sanitization
- Notification handler clears _last_error on successful refresh (fixes
  stale error for push-notification servers that skip _periodic_refresh)
2026-03-15 17:12:13 -07:00
Patrick Buckley 0d77d65266 test: add OIDC handler integration tests (22 tests) (#99)
TestClient-based integration tests for the 4 OIDC HTTP endpoints: authorize, callback, admin list identities, admin delete identity.

Uses real SQLite storage with mocked external OIDC calls (exchange_code, validate_id_token, provision_oidc_user) to exercise the full handler→module→storage contract. Covers happy paths, error flows, rate limiting, JWKS key rotation retry, and state expiration.
2026-03-15 16:44:44 -07:00
Patrick Buckley c11991819e fix: apt-get upgrade in Dockerfile to resolve CVE-2026-0861
Trivy scan fails on HIGH for libc-bin/libc6 (2.41-12+deb13u1).
The fix (2.41-12+deb13u2) is available in Debian repos but the
base python:3.14-slim image hasn't been rebuilt yet. Adding
apt-get upgrade pulls in all pending security patches at build time.
2026-03-15 16:40:03 -07:00
Patrick Buckley fc948d711d fix: use pgautoupgrade for seamless postgres major version upgrades
Replaces postgres:18-alpine with pgautoupgrade/pgautoupgrade:18-alpine
in compose.yaml. Sets PGDATA=/var/lib/postgresql/data so pgautoupgrade
detects existing pg17 data and runs pg_upgrade automatically on first
start. No manual migration needed.

Also increases healthcheck start_period to 30s to accommodate the
one-time upgrade process.
2026-03-15 16:26:19 -07:00
renovate[bot] d9722f3578 chore(config): migrate config .github/renovate.json (#97)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 16:24:39 -07:00
renovate[bot] f494553020 chore(deps): update helm release redis to v25 (#93)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:41 -07:00
renovate[bot] c766c81f25 chore(deps): update helm release postgresql to v18 (#92)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:38 -07:00
renovate[bot] 3b746fb28d chore(deps): update docker images (#90)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:34 -07:00
renovate[bot] b82fa4923c chore(deps): lock file maintenance (#94)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:18 -07:00
renovate[bot] 4ac316dc0e chore(deps): update github actions (#91)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:16 -07:00
renovate[bot] 387ef06da7 chore(deps): update helm release redis to ~20.13.0 (#89)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:13 -07:00
renovate[bot] e4e2200c33 chore(deps): update helm release postgresql to ~16.7.0 (#88)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:11 -07:00
renovate[bot] 8b94d553e4 chore(deps): update docker images (#87)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:55:53 -07:00
renovate[bot] cf16724137 chore(deps): pin dependencies (#86)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:53:03 -07:00
Patrick Buckley 22402e89de feat: add dependency management with Renovate, uv.lock, and security … (#83)
* feat: add dependency management with Renovate, uv.lock, and security scanning

Adds automated dependency update detection and vulnerability scanning
across all dependency layers (Python, vendored JS, TypeScript SDK, Docker,
GitHub Actions).

- Renovate config with 10 package groups and custom regex managers for
  vendored JS (KaTeX, Highlight.js, Mermaid) tracking via npm registry
- uv.lock for reproducible builds (80 packages)
- Dockerfile switched to uv sync --frozen with layer caching
- CI: pip-audit (via lock file), npm audit, lock-check jobs
- CI: lint job uses pre-commit for ruff version consistency
- Docker security scan workflow (weekly Trivy, HIGH/CRITICAL)
- Helper script for vendored JS library updates

* fix: resolve CI failures and address review feedback

- Update pre-commit hooks: ruff v0.9.10 -> v0.15.6 (fixes deprecated
  UP038 rule), mypy v1.14.1 -> v1.19.1
- Add per-file-ignore for N802 on sandbox.py (ast visitor convention)
- Fix pip-audit: install into uv venv so uv run can find it
- Pin uv-version in CI to match lock file generator (0.9.18)
- Upgrade vitest ^2.0 -> ^4.1 to fix esbuild GHSA-67mh-4wv8-2f99
- Vendored JS script: use grep -rl for auto-discovery of version refs
  (catches docs/architecture.md), fix LICENSE comment, portable grep
2026-03-15 15:46:48 -07:00
Patrick Buckley e7743fd079 feat: per-tool "Always" approve instead of blanket auto-approve (#82)
* feat: per-tool "Always" approve instead of blanket auto-approve

Interactive "Always" button now adds specific tool names to
auto_approve_tools instead of setting blanket auto_approve=True.
Only the tool types in the current batch are auto-approved going
forward — new tool types still prompt for approval.

Server uses approval_label (with func_name fallback) matching the
existing approve_tools() lookup. CLI and bridge use func_name.
Budget override excluded from all paths.

UI: dashed border on Always button signals persistent action,
dynamic tooltip/badge show tool names, aria-label for screen
readers, focus-visible outline fix, overflow-wrap on badge.

Bridge: seeds with DEFAULT_SAFE_TOOLS on first "always" to avoid
losing existing safe-tool auto-approvals.

16 new tests (10 unit + 6 TestClient integration). Updated tool
pipeline diagram and docs.

* fix: address copilot review — filter errored items, hide Always on budget-only

- Server/bridge/JS: add `not it.get("error")` filter so policy-denied
  items aren't added to auto_approve_tools
- Hide Always button when no eligible tools (budget-override-only batch)
- Docs: clarify CLI/bridge use func_name (coarser MCP granularity)
2026-03-15 15:25:40 -07:00
Patrick Buckley 27349e1c13 refactor: move bridge content buffer to server-side single source of truth
Eliminate dual accumulation by piggybacking assistant response text on
the server's ws_state:idle SSE event.  The bridge no longer maintains
its own _ws_content_buffer — it reads content directly from the idle
event and passes it through to TurnCompleteEvent unchanged.

Server-side: WebUI accumulates tokens in on_content_token(), joins and
includes in the idle broadcast, then resets (with 256 KB cap).

Downstream consumers (Discord bidi DM forwarding, catch-up) are
unaffected — TurnCompleteEvent.content is still populated.
2026-03-15 15:04:22 -07:00
Patrick Buckley 37a48bb30d fix: validate scope_id requires scope in memory API (#80)
* fix: validate scope_id requires scope in memory API

Prevent misleading scope_id usage: reject scope_id with global scope,
require scope when scope_id is provided, require scope_id for
workstream/user scopes on writes. Belt-and-suspenders guard in storage
backends ignores scope_id when scope is empty.

* fix: strip whitespace in scope validation, relax user scope_id requirement

Address Copilot review: .strip() whitespace-only values in all three
validation helpers; SaveMemoryRequest no longer requires scope_id for
user scope since the server auto-resolves it from auth context.
2026-03-15 14:28:03 -07:00
Patrick Buckley 730f5704ff fix: inject prompt template guardrails into plan agent system message (#79)
* fix: inject prompt template guardrails into plan agent system message

Safety/behavioral templates were silently bypassed by the plan agent,
which only used _PLAN_IDENTITY. Now _plan_system_content() prepends
_template_content (when present) so admin-configured guardrails apply
to both _exec_plan and _refine_plan, matching the task agent pattern.

* fix: address Copilot review — log truncation, comment clarity, test robustness

- Log warning on template truncation in _plan_system_content() for
  consistency with _init_system_messages()
- Clarify comment that prior plan pairs (not general history) are forwarded
- Use ChatSession._PLAN_IDENTITY for index assertions instead of substring
2026-03-15 14:24:10 -07:00
Patrick Buckley 9b3b1c1ddd fix: reorder new-workstream modal so Task is the primary field (#77)
* fix: reorder new-workstream modal so Task is the primary field

Users were typing their prompt into the Name field (first text input,
auto-focused) and leaving Task empty, creating idle workstreams. Move
Task textarea to the top of the form, auto-focus it, and add
Ctrl/Cmd+Enter submit shortcut. Accessibility fixes: cancel button
focus-visible, label-hint contrast raised to WCAG AA, platform-aware
keyboard hint, Ctrl+Enter added to shortcuts overlay.

* fix: Enter on Cancel button no longer triggers submit

Copilot review caught that pressing Enter while focused on the Cancel
button bypassed native click and called submitNewWs(). Skip the
Enter-to-submit handler for BUTTON elements so native activation fires.
Also make keyboard shortcuts overlay platform-aware (Ctrl vs ⌘).
2026-03-15 14:23:57 -07:00
Patrick Buckley a9ed8a954b fix: convert _pending_nudge from single-slot to list for defensive correctness
Every append site immediately drains via _init_system_messages(), so this
is defensive — ensures multiple nudges survive if the drain flow is ever
refactored to batch calls.
2026-03-15 14:17:34 -07:00
Patrick Buckley 1efcbcf2ba perf: parallelize _collect_mcp_status and _notify_nodes_mcp_reload wi… (#73)
* perf: parallelize _collect_mcp_status and _notify_nodes_mcp_reload with asyncio.gather

Both functions queried cluster nodes sequentially, making latency
O(N × timeout). Use asyncio.gather to query all nodes concurrently,
matching the existing admin_list_watches pattern. Also reuse the
shared proxy_client instead of creating throwaway httpx clients per
node, and add debug logging on MCP status fetch failures.

* perf: bound node fan-out concurrency and improve debug logging

Add _NODE_FAN_OUT_LIMIT (50) semaphore to all three gather fan-out
sites (_collect_mcp_status, _notify_nodes_mcp_reload, admin_list_watches)
to cap concurrent outbound connections below the httpx pool limit,
leaving headroom for other proxy traffic at 1000-node scale.

Add exc_info=True to all debug log calls for actionable diagnostics.

* test: add unit tests for _collect_mcp_status and _notify_nodes_mcp_reload

11 tests covering success, non-200, missing URL, exceptions, empty
cluster, and mixed multi-node scenarios for both fan-out helpers.
2026-03-15 13:54:49 -07:00
Patrick Buckley 1d36d80fe5 fix: reduce metacognition false positives with strong/weak pattern tiers (#76)
* fix: reduce metacognition false positives with strong/weak pattern tiers

Correction detection: split "no" handling — "no," and "no." are strong
(always fire), "no <word>" uses an allowlist of correction-context words
(pronouns, demonstratives, verbs) instead of a blocklist. Phrases like
"no problem", "no worries", "no rush" are excluded automatically.

Completion detection: move most patterns to weak tier, gated by message
length (<80 chars) and absence of continuation markers ("?", "can you",
"but", "now", "please", etc.). "thanks for X" excluded at regex level.
Strong tier (always fire): "that's all", "lgtm".

* fix: align allowlist comment with implementation (include articles)
2026-03-15 13:53:39 -07:00
Patrick Buckley e603a6a7d1 fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var (#74)
* fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var

OIDC redirect_uri was derived from the request Host header, which is
unreliable behind reverse proxies. Add TURNSTONE_OIDC_REDIRECT_BASE
(env var / config.toml) to pin the externally-reachable origin.

Extract _build_oidc_redirect_uri() helper to deduplicate the authorize
and callback handlers. Validate redirect_base at load time (must be
scheme://host[:port], rejects paths/query strings/invalid schemes).

* fix(oidc): reject redirect_base with missing hostname

Addresses Copilot review: values like `https://` or `https://:443`
passed validation but would produce invalid redirect URIs.

* fix(oidc): reject redirect_base with userinfo or invalid port

Addresses Copilot round 2: urlparse silently accepts user:pass@host
and non-numeric ports. Now explicitly rejects both.
2026-03-15 13:52:05 -07:00
Patrick Buckley 2e95f2ac73 test: add scope coverage for internal MCP/config reload endpoints (#75)
* test: add scope coverage for internal MCP/config reload endpoints

Verify required_scope() returns "approve" for _internal endpoints
across all access patterns (bare, /v1/-prefixed, console proxy with
and without /v1/), plus a GET negative test confirming only POST is
elevated. Closes the "internal endpoints accept read scope" item in
PROGRESS.md — the endpoints were already in APPROVE_PATHS.

* test: add config-reload v1/proxy scope tests per review feedback

Add /v1/-prefixed and console proxy variants for config-reload to
match the mcp-reload coverage, as flagged by Copilot review.
2026-03-15 13:45:35 -07:00
Patrick Buckley 5f27ed9fca feat: OIDC identity management inline in Users admin tab (#72)
Expandable user rows in the console Users tab reveal OIDC identities
linked to each user. Issuer badge, truncated subject, email, relative
last-login time, and unlink action with confirmation modal + audit trail.

Keyboard accessible (tabindex, Enter/Space, aria-expanded, focus-visible).
In-place refresh after unlink (no close/reopen flicker). Audit captures
user_id before delete. Mobile responsive (3-column at <700px).
Reduced-motion support. 2 new admin API endpoints reusing admin.users
permission and existing storage methods.
2026-03-15 03:52:42 -07:00
Patrick Buckley 20df7b3034 feat: OIDC SSO authentication with PKCE, auto-provisioning, and role … (#71)
* feat: OIDC SSO authentication with PKCE, auto-provisioning, and role mapping

Add OpenID Connect as a fourth authentication method, enabling single sign-on
via any OIDC provider (Okta, Azure AD, Google, Keycloak). Opt-in via env vars
(TURNSTONE_OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET).

Security:
- Authorization Code Flow with PKCE (S256)
- State/nonce parameters with database-backed pending store (multi-node safe)
- JWKS signature validation with async fetch + key rotation retry
- Algorithm allowlist from JWKS key (not token header) prevents confusion
- Identity matching exclusively by (issuer, sub) — prevents account takeover
- password_enabled=false enforced server-side, not just UI
- Rate limiting on both authorize and callback endpoints
- OIDC users get "!oidc" password sentinel (bcrypt rejects naturally)
- ID token validated for iss, aud, exp, nonce

Features:
- Auto-provisioning with username deduplication on first login
- Claim-based role mapping with IdP demotion propagation (revokes stale roles)
- "Continue with [Provider]" SSO button on login page
- OIDC-only mode hides password form
- Setup wizard required before OIDC login (admin bootstrap)

Storage: migration 018 (oidc_identities + oidc_pending_states tables),
8 new protocol methods on both SQLite and PostgreSQL backends.
66 new tests (2273 total).

* fix: address PR #71 review feedback (18 items)

Bugs fixed:
- OIDC success redirect now fetches permissions via new /auth/whoami
  endpoint before completing login (fixes permission-gating in UI)
- Remove double decodeURIComponent on oidc_error (URLSearchParams
  already decodes; extra call throws on stray %)
- Authorize rate limiter returns redirect instead of JSON 429
  (endpoint reached via browser navigation, not fetch)
- Lazy JWKS fetch in callback when startup discovery failed (IdP
  recovery without restart)
- Startup exception handlers now log with exc_info=True
- PostgreSQL pop_oidc_pending_state uses DELETE...RETURNING for
  true atomicity (eliminates TOCTOU)

Behavior:
- New OIDC users without role mapping get builtin-viewer by default
  (assigned_by="oidc-default", not revoked by role sync)

Documentation fixes:
- Role mapping: sync semantics (add + revoke stale), not "additive only"
- PASSWORD_ENABLED=false blocks ALL password logins including admin
- Algorithm: asymmetric allowlist, not per-key derivation
- PlantUML diagram updated for role revocation

API spec fixes:
- Removed error_codes=[302] from callback (302 is success redirect)
- Added /auth/whoami to both server + console specs
- Regenerated TypeScript SDK OpenAPI snapshots (23 + 51 paths)

* fix: address PR #71 round 2 review feedback (10 items)

Rate limiting:
- Authorize endpoint now calls record() after check() so the rate
  limiter actually counts attempts (was a no-op before)

OIDC resilience:
- Split startup try/except: discovery failure disables OIDC, JWKS
  prefetch failure leaves OIDC enabled for lazy retry on first login
- JWKS unavailable message changed to "temporarily unavailable"
  (was misleadingly "not configured")
- create_oidc_pending_state raises on collision instead of OR IGNORE
  (prevents silent insert drop on state collision)
- SQLite pop_oidc_pending_state uses BEGIN IMMEDIATE for write lock
  (eliminates TOCTOU race)

Frontend:
- OIDC error display deferred 300ms so showLogin()'s async status
  fetch doesn't clear it via _switchMode → _clearError

API spec:
- OIDC authorize/callback endpoints now declare response_code=302
- Added AuthWhoamiResponse Pydantic model for /auth/whoami
- Regenerated TypeScript SDK OpenAPI snapshots

Documentation:
- Diagram: JWKS "cached at startup, refreshed on-demand" (was "hourly")
- Added TODO(tech-debt) comments on Host header redirect_uri sites
2026-03-15 03:44:18 -07:00
Patrick Buckley 68c991fbdd fix: restore safe HTML element rendering and suppress plantuml warning (#70)
* fix: restore safe HTML element rendering and suppress plantuml warning

- Add safe HTML tag allowlist in inlineMarkdown: br, hr, kbd, mark,
  sub, sup, ins, wbr, details, summary, abbr, small, u, s
  (attribute-free only — XSS safe, tags with attributes stay escaped)
- Add <details>/<summary> block-level protection pass with recursive
  markdown rendering of inner content
- Add plantuml to _NO_HIGHLIGHT_LANGS (suppresses highlight.js warning
  for unsupported language)
- CSS for details (collapsible, overflow hidden), kbd (mono font,
  key style), mark (yellow-glow token for theme adaptation)

* fix: restrict safe tags to inline-only, broaden details regex

- Remove hr, details, summary from inline _SAFE_TAGS allowlist (they
  are block-level and produce invalid HTML inside <p> wrappers)
- Make <details> regex newline-optional so same-line
  <details><summary>Title</summary> patterns are captured
2026-03-15 02:34:03 -07:00
Patrick Buckley 376da3d084 feat: prompt template tech debt — tests, read-only endpoints, double-… (#67)
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal

Close test coverage gaps for prompt templates:
- Resume with deleted template: verifies graceful degradation (template_content=None, warning logged)
- Threading safety: concurrent set_template/init_system_messages with no race conditions
- Factory passthrough: template kwarg propagation through WorkstreamManager.create()

Add read-only template listing endpoints (read scope, no content exposed):
- GET /v1/api/templates — prompt template summaries (name, category, is_default, origin)
- GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model)
- Available on both server and console; Python + TypeScript SDK methods added
- Console creation modal switched from admin endpoint to read-scope endpoint

Eliminate double-load inefficiency in workstream creation:
- Template validation moved before mgr.create() (no create-then-rollback on invalid template)
- template kwarg plumbed through WorkstreamManager.create() and session factory
- _SessionFactory Protocol added for proper mypy typing

Add workstream creation modal to server web UI:
- Name, model, template dropdown, ws_template/profile dropdown
- Instrument panel aesthetic: gradient top border, blur backdrop, amber accent
- Focus trap, Escape/Enter keyboard handling, loading state, error display
- WCAG AA contrast compliance, reduced-motion support

* fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots

Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates()
to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint.
Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types.
Regenerate openapi-server.json and openapi-console.json snapshots.

Addresses Copilot review feedback on PR #67.

* fix: skip template pre-validation when resuming a workstream

When resume_ws is set, the request's template field is irrelevant —
resume() restores the template from workstream_config. Pre-validating
a stale template name would incorrectly return 400 before the resume
even runs.

Addresses Copilot review feedback on PR #67.
2026-03-15 02:09:31 -07:00
Patrick Buckley e2a199c9c3 feat: mermaid diagram rendering with lazy loading and theme integration (#69)
* feat: mermaid diagram rendering with lazy loading and theme integration

Integrate mermaid.js 11.13.0 (self-hosted, MIT, ~2.9MB) for rendering
```mermaid code blocks as inline SVG diagrams. Covers flowcharts,
sequence, class, ER, state, gantt, pie, timeline, and mindmap.

- Lazy-loaded via dynamic script injection on first mermaid block
  detection (not eagerly loaded on every page view)
- 3-state loader (idle/loading/ready) with callback queue
- Serialized rendering to avoid mermaid internal state corruption
- Theme integration via getComputedStyle reading CSS design tokens;
  re-renders all diagrams on dark/light theme toggle
- Source preserved in data-mermaid-source for theme re-rendering
- Error handling with source code fallback display
- securityLevel: "strict" (DOMPurify) for SVG XSS prevention
- THIRD-PARTY-NOTICES updated with mermaid MIT license

* fix: mermaid render fixes from Copilot review

- Call result.bindFunctions(container) after SVG insertion for
  interactive diagram elements (click handlers, links, tooltips)
- Clear mermaid-error class on successful render (fixes stale error
  styling after theme toggle re-render)
- Clear mermaid-error in reRenderAllMermaid before re-render sequence
- Restructure postRenderMarkdown so mermaid rendering runs even when
  highlight.js is unavailable (hljs guard changed from early return
  to conditional block)
2026-03-15 02:09:10 -07:00
Patrick Buckley 4152ea2352 fix: widen code fence regex and skip auto-detect on unlabeled blocks
- Regex changed from (\w*) to ([^\s`]*) to capture language names with
  special chars (c++, c#, objective-c, shell-session)
- Alias map normalizes c++ → cpp, c# → csharp, f# → fsharp for CSS
  class names
- Empty language no longer emits class="language-", preventing
  highlight.js auto-detect across all 37 bundled languages on
  unlabeled code blocks (performance fix for large blocks)
2026-03-15 02:04:58 -07:00
Patrick Buckley 44cc14b46f feat: syntax highlighting via highlight.js with code block variants (#68)
Integrate highlight.js 11.11.1 (self-hosted, BSD-3-Clause, ~125KB) for
language-aware syntax highlighting on fenced code blocks.

- postRenderMarkdown() hook applies highlighting at stream_end and
  history load — not during streaming (innerHTML replaced per token)
- Custom theme using CSS design tokens (auto-adapts dark/light)
- Code block variants: diff (green/red line coloring), bash/shell
  (terminal left-border), ascii/text/plaintext (no highlighting)
- Class prefix changed from lang- to language- (CommonMark standard)
- Graceful degradation when highlight.js unavailable
- THIRD-PARTY-NOTICES file for bundled dependency attribution
- pyproject.toml package-data glob for vendored hljs directory
2026-03-15 01:36:57 -07:00
Patrick Buckley 83b0cde32f feat: GFM extended syntax renderers (callouts, footnotes, definition … (#66)
* feat: GFM extended syntax renderers (callouts, footnotes, definition lists)

Add three GFM extended syntax features to the server web UI markdown
renderer, with no external library dependencies (pure JS/CSS):

- Callouts/Alerts: > [!NOTE], [!TIP], [!IMPORTANT], [!WARNING], [!CAUTION]
  with color-coded left borders, icons, and recursive markdown body
- Definition Lists: Term + `: Definition` pattern with multi-term support
- Footnotes: [^id] inline superscript references, [^id]: definitions
  collected into a numbered section with bidirectional navigation

Design review fixes: scoped footnote IDs (prevent collisions across
messages), aria-hidden on callout icons, aria-label on callout containers,
focus-visible on footnote links, smooth-scroll footnote navigation.

* fix: use getElementById for footnote scroll to handle special chars in IDs

querySelector throws on fragment IDs containing &, . or : characters
(produced by escapeHtml on footnote labels). getElementById accepts any
string and is the correct API for ID-based element lookup.
2026-03-15 01:33:54 -07:00
Patrick Buckley 2ef8a8711b feat: rich markdown renderer with LaTeX support for server web UI (#65)
* feat: rich markdown renderer with LaTeX support for server web UI

Extract markdown rendering from app.js into dedicated renderer.js with
full GFM support: tables (alignment, hover, striping), nested lists,
task list checkboxes, nested blockquotes, images (click-to-load for
privacy), and inline/display LaTeX math via self-hosted KaTeX 0.16.38.

Security: escape image/link URLs to prevent attribute injection, block
javascript: scheme in links, add rel="noopener noreferrer", images
require explicit click to load (no automatic external requests).

Accessibility: scope="col" on table headers, tabindex on scrollable
table containers, aria-labels on task checkboxes and image placeholders,
KaTeX error color override for WCAG AA contrast, reduced-motion support.

* fix: address code review — XSS hardening and list type splitting

- Escape all text through escapeHtml() at start of inlineMarkdown()
  so only renderer-generated tags appear in innerHTML (prevents raw
  HTML/script injection from LLM output)
- Replace inline onclick handler on image placeholders with data-*
  attributes and delegated DOM event listeners (prevents entity
  decoding XSS in event handler attributes)
- Split list blocks into separate <ul>/<ol> when marker type changes
  at the same indent level (mixed ordered/unordered sequences)
2026-03-15 00:04:01 -07:00
Patrick Buckley 3658b77de8 feat: Discord content catch-up + bidirectional notification replies (… (#64)
* feat: Discord content catch-up + bidirectional notification replies (#64)

Two improvements to the Discord channel adapter:

1. Fix intermittent dropped responses caused by a race between the
   bridge's two independent SSE connections (global SSE detects idle
   before per-ws SSE delivers all content tokens). The bridge now
   accumulates content in _ws_content_buffer and attaches it to
   TurnCompleteEvent.content. The Discord bot uses this as a catch-up
   when streaming events were missed.

2. Bidirectional notification replies — when the notify tool sends a DM,
   the message is tracked with the originating ws_id. Users can reply to
   the DM and the reply is routed to the workstream. The response is
   forwarded back to the DM, with the response itself tracked for
   multi-turn conversations. Includes user identity verification,
   stale notification feedback, and FIFO-capped tracking (100 entries).

* fix: address Copilot review — re-insert on unlinked user, deque buffer

- Re-insert _notify_ws_map entry when resolve_user returns None so the
  user can retry after linking (same pattern as user-mismatch re-insert)
- Rename _MAX_CONTENT_BUFFER_BYTES → _MAX_CONTENT_BUFFER_CHARS (len()
  returns characters, not bytes)
- Use deque + running total for O(1) popleft instead of list.pop(0)
2026-03-14 23:58:27 -07:00
Patrick Buckley 83577739e0 chore: bump version to 0.6.2
- MCP admin tab: database-backed server management, hot-reload,
  reconcile, unified config view, paste-based import
- Catch-up migration for builtin-admin permissions (017)
- `[all]` optional dependency group (@Burhan-Q)
2026-03-14 17:25:58 -07:00
Burhan 71d13936fe add "all" optional dep (#61) 2026-03-14 17:05:49 -07:00
Patrick Buckley 0cd061196c fix: catch-up migration ensuring builtin-admin has all 20 permissions (#63)
Migrations 011-016 each appended a permission to the builtin-admin role
via conditional UPDATE, but on some deployments these never applied.
Migration 017 idempotently sets the complete permission string rather
than appending incrementally.

Must be merged after feat/admin-mcp-servers (migration 016).
2026-03-14 17:03:21 -07:00
Patrick Buckley 19abc0cc65 feat: admin MCP Servers tab — database-backed MCP server management w… (#62)
* feat: admin MCP Servers tab — database-backed MCP server management with live status

Add MCP Servers admin tab (14th tab, System group) for managing MCP server
definitions via the database instead of static JSON config files.

Storage: `mcp_servers` table (migration 016), 6 CRUD methods on both SQLite
and PostgreSQL backends, `MCP_SERVER_MUTABLE` field allowlist.

Config priority chain: DB rows (if any enabled) → CLI `--mcp-config` →
`mcp.config_path` setting → none. Nodes auto-load from DB on startup via
`load_mcp_config(storage=)`.

Hot-reload: `reconcile_sync(storage)` diffs running servers against DB —
adds missing, removes stale, reconnects changed. `_db_managed` set tracks
DB-sourced servers so config-file servers (MCP_CONFIG env) are never removed
by reconcile. Per-server `AsyncExitStack` for clean teardown.

Reload pattern: console writes to DB then signals nodes via
`POST /_internal/mcp-reload` (update by reference, no config payload).

Console admin API: 7 endpoints under `/v1/api/admin/mcp-servers` (CRUD +
reload + import), `admin.mcp` permission, secret masking (env/headers
replaced with *** unless ?reveal=true), audit log sanitization.

Unified view: tab merges DB-managed servers with config-sourced servers
detected on nodes. Config servers shown as read-only rows with "config"
badge — no edit/delete.

Admin UI: 7-column grid with magenta status dots, transport badges,
single-column create/edit modal, paste-based JSON import (mcpServers format),
detail modal with per-node status. Mobile 3-column collapse, reduced-motion
support, backdrop-click dismiss, focus trapping.

SDKs: 7 methods on Python (async+sync) and TypeScript SDKs.

Also fixes: Settings tab permission gate (admin.users → admin.settings),
_ALL_PERMISSIONS list in governance.js (5 missing permissions added),
_internal/mcp-reload added to APPROVE_PATHS.

Docs: architecture.md (14 tabs), api-reference.md (7 endpoints),
20-mcp-architecture.puml updated with admin-driven lifecycle.

66 new tests (2232 total).

* fix: address Copilot review feedback on MCP admin PR

- Docs: fix "merges both sources" → "first-match-wins priority" (architecture.md)
- Validation: require command for stdio, url for streamable-http transport
- Validation: check args/headers/env types in import handler before storing
- Schema: add transport/command/url to McpServerStatus, source to McpServerDetail
- Thread safety: move all remove_server_sync mutations onto MCP event loop thread
- Regenerate OpenAPI JSON snapshots for TypeScript SDK
2026-03-14 17:02:50 -07:00
Patrick Buckley c5cdfc8f44 chore: bump version to 0.6.1 2026-03-14 13:19:40 -07:00
Patrick Buckley 8895bf07eb feat: admin Settings tab — form-based editor replacing "coming soon" … (#60)
* feat: admin Settings tab — form-based editor replacing "coming soon" stub

Section-grouped layout with collapsible headers for all ~40 ConfigStore
settings (model, session, tools, server, mcp, ratelimit, health, judge,
memory). Type-appropriate inputs: CSS toggle for bools, number with
min/max/step, select for choices, text for strings. Secret fields shown
read-only. Source badge (storage/default), amber restart indicator.

Inline save per field with dirty detection, row flash on success, reset
to default via styled confirm modal. Full WCAG keyboard accessibility
(Enter/Space on section headers, aria-labels, focus-visible). Mobile
responsive single-column at <700px. Reduced-motion safe.

* fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes

Settings UI:
- Help tooltips: ? button on ~25 settings with plain-English explanations
  and optional reference links (arXiv, Fowler, MCP spec). Click to toggle
  popover, Escape to dismiss, aria-expanded for accessibility.
- Sections start collapsed for scannable overview.
- Restart badge: hidden by default, shows when dirty, persists after save
  with amber glow. Positioned left of source badge.
- Secret row alignment fixed (transparent border matches input box model).
- Docs link in toolbar → Swagger UI Settings section.
- Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard,
  numeric dirty detection (0.1 vs 0.10 no longer false positive).
- Secret reset button enabled when source=storage (clear legacy overrides).
- Space key repeat guard on section headers.
- Sidebar: sticky + max-height:100vh, no longer stretches with content.

Backend:
- context_window default changed from 131072 to 0 (auto-detect). Fallback
  lowered from 131K to 32K (realistic for local models when detection fails).
  Session normalizes 0→32768 defensively.
- Settings registry: help + reference_url fields on SettingDef, richer
  descriptions for model/session/tools/judge/memory settings.
- Schema API includes help + reference_url.
- Bootstrap system prompt: added Runtime Settings section.

Docs: tab counts updated to 13 across README, architecture, console, governance.
2026-03-14 13:12:40 -07:00
Patrick Buckley 101afd84da feat: database-backed settings (ConfigStore) with admin API (#59)
* feat: database-backed settings (ConfigStore) with admin API

Replace config.toml for non-bootstrap settings on the server with a
database-backed ConfigStore.  ~40 settings across model, session,
tools, server, mcp, ratelimit, health, judge, and memory sections are
now managed via the admin Settings API.  CLI flags for these settings
removed from the server entry point (CLI standalone tool unchanged).

Storage: system_settings table (migration 015) with composite PK
(key, node_id) for per-node overrides.  ON CONFLICT upsert in both
SQLite and PostgreSQL.  admin.settings permission granted to
builtin-admin role.

Settings registry (settings_registry.py): code-defined catalog of
all known settings with types, defaults, validation, descriptions.
Registry defaults aligned with previous argparse defaults.

ConfigStore (config_store.py): thread-safe in-memory cache loaded
from storage on init.  Lock-free reads via dict snapshot swap.
reload() for hot-reload via internal endpoint.

Secret settings (judge.api_key) blocked from write via admin API
(403) — must be configured via config.toml or env vars.

warn_migrated_settings() logs warnings for config.toml keys that
overlap with ConfigStore-managed settings.

Console admin API: GET /v1/api/admin/settings (list with effective
values), GET .../schema (registry catalog), PUT .../{key} (update),
DELETE .../{key} (reset to default).  Audit trail on mutations.

MQ: ConfigChangeEvent for cross-node cache invalidation (emission
from console deferred to bridge integration).

Python + TypeScript SDK methods.  63 new tests.  Feature docs at
docs/settings.md, PlantUML diagram 24-settings-architecture.

* fix: address PR review — config-reload scope, registry defaults, doc alignment

- config-reload endpoint requires approve scope (was write)
- config-reload handler is sync def (avoids blocking event loop)
- reasoning_effort passes empty string through (removes `or "medium"`)
- ratelimit.requests_per_second changed to float (matches RateLimiter)
- session.retention_days allows 0 (disable pruning)
- ratelimit.trusted_proxies added to registry + wired in server
- admin_list_settings filters to global settings only (no node_id ambiguity)
- admin_update_setting validates "value" key presence (400 if missing)
- Console config change fans out reload to nodes directly (no MQ dep)
- Docs aligned with actual API response shapes and masking ("***")
2026-03-14 11:42:18 -07:00
Patrick Buckley efd98712e9 feat: [memory] admin panel Memories tab — browse, search, inspect, de… (#57)
* feat: [memory] admin panel Memories tab — browse, search, inspect, delete

Add 13th admin tab in the Observe group for cluster-wide memory
management.  List view with type/scope filter dropdowns and debounced
search input.  Detail modal shows full metadata grid and scrollable
content block.  Delete from both list row and detail modal with
confirmation and audit trail.

Permission-gated behind admin.memories.  Escape key, backdrop click,
and focus trap wired for the detail modal.  Mobile responsive: hides
description and updated columns below 700px.

* fix: memory detail modal — focus, delete safety, CSS shorthand order

Address Copilot review feedback: move focus to close button on modal
open for keyboard accessibility, disable delete button and clear stale
handler during loading/error states to prevent wrong-memory deletion,
and fix font shorthand/font-size ordering in toolbar filter styles.
2026-03-14 02:42:43 -07:00
Patrick Buckley 67f43a7ee0 feat: [memory] REST API endpoints + SDK methods + docs (#56)
* feat: [memory] REST API endpoints + SDK methods + docs

Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope

Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit

Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.

Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.

TypeScript SDK: matching methods + types on both clients.

Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.

Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.

Also fixes stray `total: int` on CreateChannelUserRequest.

* fix: [memory] address PR review — cross-user scope, schema types, snapshots

Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity.  Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.

Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.

Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.

6 new security tests for user-scope access control.
2026-03-14 02:28:47 -07:00
Patrick Buckley 2888e8ce0a feat: MCP cluster-ops example — reference MCP server + SDK implementa… (#55)
* feat: MCP cluster-ops example — reference MCP server + SDK implementation

Standalone MCP server under examples/mcp-cluster-ops/ that exposes
tools for executing commands across a Turnstone cluster via the MQ
client SDK. Serves as a reference implementation for both MCP server
patterns (FastMCP, lifespan, tool handlers) and TurnstoneClient usage.

4 tools: list_nodes, run_on_node, run_on_nodes, run_on_all_nodes.
Parallel dispatch via asyncio.gather, raw ToolResultEvent output
capture, UTF-8 safe truncation, input validation, concurrency caps.

35 tests, ruff clean, mypy --strict clean.

* fix: address review feedback on MCP cluster-ops example

- Remove REDIS_SSL support (RedisBroker doesn't accept ssl kwarg)
- Move max-nodes check from _dispatch_parallel into tool handlers
  for consistent error shape (always returns {"error": ...} object)
- Propagate KeyboardInterrupt/SystemExit from asyncio.gather instead
  of swallowing them as per-node failures
- Fix _truncate omitted bytes count to reflect actual bytes dropped
  after multi-byte boundary adjustment
- Apply strip/dedup to node IDs in run_on_all_nodes (matching
  run_on_nodes behavior)
- Add __name__ guard to __main__.py
- Fix misleading UTF-8 byte count comment in tests
2026-03-14 02:23:43 -07:00
Patrick Buckley d1a248b413 feat: [memory] config section — configurable relevance_k, fetch_limit… (#54)
* feat: [memory] config section — configurable relevance_k, fetch_limit, max_content, nudge_cooldown, nudges

MemoryConfig dataclass in memory_relevance.py, constructed from
config.toml [memory] section via argparse defaults. Replaces
hardcoded constants in session.py. Master nudges=false switch
disables all metacognitive prompting.

* fix: wire memory config into apply_config and correct error wording

Add "memory" to apply_config sections so [memory] config.toml values
actually propagate. Fix "byte limit" → "character limit" since
len(content) measures characters.
2026-03-14 00:51:27 -07:00
Patrick Buckley 723cad24bb feat: structured memory system — typed/scoped memories with BM25 rele… (#53)
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting

Replace flat key-value memories table with structured_memories (migration 014).
Four memory types (user/project/feedback/reference), three scopes
(global/workstream/user). Consolidate remember/recall/forget into two tools:
memory (action-based: save/search/delete/list) and recall (conversation
history only).

BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5
memories for system message injection based on conversation context.
Metacognitive prompting injects ephemeral nudges after corrections, tool
denials, workstream resume, and completion signals.

Scope isolation enforced: system message injection and nudge counts filtered
to visible memories only (global + current workstream + authenticated user).
User scope requires authentication. Content capped at 32KB. ILIKE/LIKE
metacharacters escaped in both backends.

113 new tests (2053 total).

* fix: CI failure + copilot review feedback

- Fix time.monotonic() cooldown: use None sentinel instead of 0.0
  default (monotonic clock starts at boot, not epoch — fresh CI
  runners have uptime < 300s so cooldown check always triggered)
- Catch sa.exc.IntegrityError specifically in upsert instead of
  broad Exception (copilot review)
- Preserve existing description/type on upsert when caller doesn't
  explicitly set them (copilot review)
- Add last_accessed + access_count columns to schema/migration for
  future LRU/LFU eviction support
2026-03-13 21:21:09 -07:00
Patrick Buckley 73cacc8ad6 feat: admin panel — right-aligned sidebar navigation with two-column … (#52)
* feat: admin panel — right-aligned sidebar navigation with two-column modals

Replace the horizontal tab bar (11 tabs, overflowing on standard monitors)
with a grouped sidebar on the right side, matching the admin button's
position in the header for natural spatial flow.

Sidebar: 5 groups (Identity, Automation, Governance, Observe, System) with
12 nav items including new Settings stub. Always visible on desktop (180px),
off-canvas drawer on mobile (<700px) sliding from right with backdrop.

Admin button: toggle behavior (click again to return to overview), active
state with amber highlight + top accent line, aria-expanded management.

Breadcrumb: shows active tab ("Admin / Users", "Admin / Audit", etc).

Modals: WS Template and Schedule create/edit forms restructured into
two-column grid (820px) with "Identity"/"Model Config" and
"Schedule"/"Execution" column headings. All modals gain max-height: 85vh
+ overflow-y: auto safety net. Modal z-index bumped to 600 (above sidebar).

Also: "Tokens" renamed to "API Tokens", redundant "Server default"
placeholders removed from model config fields, view fade-in transition,
comprehensive ARIA (grouped sidebar, aria-hidden on mobile, focus return
on drawer close), reduced-motion support.

* fix: address Copilot review — aria-orientation, settings permission gate, inert sidebar

- Add aria-orientation="vertical" to sidebar tablist for assistive tech
- Gate Settings tab behind admin.users permission so empty-state logic
  works correctly when user has no admin permissions
- Use inert attribute on mobile sidebar when closed to prevent keyboard
  focus from reaching off-canvas controls
- Add resize listener to sync aria-hidden/inert when crossing the
  700px mobile breakpoint
2026-03-13 19:50:35 -07:00
Patrick Buckley ccd1c1a9ad chore: bump version to 0.6.0
Workstream templates (#49), intent validation (#50), conversation schema redesign (#51).
2026-03-13 14:43:01 -07:00
Patrick Buckley 1295919613 fix: simplify conversation storage — atomic assistant rows with tool_… (#51)
* fix: simplify conversation storage — atomic assistant rows with tool_calls JSON

Replace the denormalized storage model (separate rows for assistant
content, tool_call, tool_result) with atomic assistant rows carrying
tool_calls as a JSON column. Eliminates the 100-line heuristic
reconstruct_messages function and its cross-turn merge bug.

Schema: add tool_calls TEXT column to conversations (migration 013).
Migration backfills existing data — merges tool_call rows into their
parent assistant row as JSON, renames tool_result to tool, deletes
consumed tool_call rows.

Session save path: assistant content + tool_calls saved in one
save_message call before tool execution (crash resilient). Tool
results saved as role="tool".

Extract shared storage utilities to _utils.py: row_to_dict, mutable
field frozensets, reconstruct_messages. Both backends import from
_utils — PostgreSQL no longer depends on _sqlite.py.

Includes denied/blocked tool call badge fix on resume: _build_history
detects denied results and propagates flag to parent assistant entry.
Frontend uses flag for correct badge-denied rendering. Denied tools
visually muted. role="status" on badges for accessibility.

Net -45 lines. 8 new tests for reconstruction, all 1914 tests pass.

* fix: migration 013 uses parameterized deletes and ordered downgrade

- DELETE of consumed tool_call rows now uses parameterized batches
  (chunks of 500) instead of string interpolation
- Downgrade rebuilds via temp table to preserve chronological id
  ordering when re-inserting tool_call rows
2026-03-13 14:26:05 -07:00
Patrick Buckley 09ea3d164d feat: intent validation v1 — advisory LLM judge for tool approvals (#50) (#50)
* feat: intent validation v1 — advisory LLM judge for tool approvals (#50)

Two-tier evaluation pipeline for non-auto-approved tool calls:
- Heuristic tier (instant): 23 pattern-based rules across 4 severity
  levels (critical/high/medium/low) with first-match-wins priority
- LLM judge tier (async): multi-turn evaluation with read_file/
  list_directory tool access, security-hardened path blocking, forcing
  message on final turn, four-stage JSON parsing with retry nudge

Progressive UI: heuristic verdict badge + judge spinner, LLM verdict
upgrade via intent_verdict SSE event, glow on action buttons. Verdict
persisted to intent_verdicts table for audit. Prometheus metrics for
verdict counts and LLM latency. Enabled by default (--no-judge to opt
out). 132 new tests (1938 total).

Integration: session, server/WebUI, CLI, MQ bridge, console admin API,
Discord channel adapter. Config via [judge] in config.toml or CLI flags.

* fix: address PR #50 Copilot review feedback

- Fix double JSON encoding of func_args in both heuristic and LLM
  verdict persistence paths — use pre-serialized string from verdict
- Fix confidence 0.0 treated as falsy in channel verdict formatter
- Fix timestamp format inconsistency in storage backends (isoformat
  vs strftime) — now uses strftime consistently
- Add on_intent_verdict to eval.py NullUI (mypy fix)
- Fix late verdict after approval resolved — store last decision and
  apply immediately to late-arriving verdicts
- Add permission rollback to migration 012 downgrade
- Update docs to reflect judge enabled by default
- Document confidence_threshold as reserved for v2

* fix: judge per-call timeout and credential recon heuristic

- Wrap create_completion() in ThreadPoolExecutor with per-call timeout
  to prevent indefinite hangs on slow local models. On timeout, replace
  the executor so subsequent batch items don't queue behind lingering
  API calls
- Add IntentJudge.shutdown() and wire into session.close() for cleanup
- Add credential-recon heuristic rule: /etc/passwd, /etc/shadow,
  /etc/master.passwd access flagged as HIGH/review (reconnaissance
  pattern even though the command itself is read-only)
- 3 new tests for credential file access patterns

* fix: denied/blocked tool calls show correct badge on resume

- _build_history() detects denied results ("Denied by user") and
  blocked results ("Blocked") and propagates denied flag to parent
  assistant entry for frontend consumption
- Frontend history replay uses denied flag for badge-denied class
  instead of hardcoding badge-approved for all historical tool calls
- Denial feedback always prefixed with "Denied by user:" so content
  detection works with custom user feedback
- Denied tools visually muted (opacity 0.55, muted tool name)
- role="status" on all approval badge elements (accessibility)
- Broadened "Blocked" prefix match (catches "Blocked by tool policy")
2026-03-13 04:12:46 -07:00
Patrick Buckley 02d9c5c797 feat: workstream templates — behavioral profiles for workstream creation (#49)
* feat: workstream templates — behavioral profiles for workstream creation

Workstream templates define the complete configuration for workstream
creation: system prompt, model, auto-approve policy, per-tool
auto-approve, temperature, reasoning effort, max tokens, agent max
turns, token budget, and completion notifications. Applied once at
creation time (snapshot, not live binding). Auto-versioning captures
pre-update state on every edit.

Schema & storage:
- workstream_templates + workstream_template_versions tables (migration 011)
- ws_template_id/ws_template_version columns on workstreams table
- ws_template column on scheduled_tasks table
- Full CRUD + versioning on SQLite and PostgreSQL backends
- prompt_template_hash (SHA-256) for drift detection

Runtime:
- Template resolution before mgr.create() for model override
- Post-creation settings application (prompt, temperature, approval, budget)
- Token budget enforcement in session.send() — 80% warning, approval gate
  at 100% via __budget_override__ synthetic tool
- WebUI.auto_approve_tools server-side per-tool auto-approve
- Prompt template drift detection (hash comparison, log warning on mismatch)

Integration:
- ws_template field on CreateWorkstreamMessage, bridge, channel router,
  scheduler dispatch, MQ client
- Console admin "WS Templates" tab (11th) with CRUD, version history modal
- Profile dropdown on workstream creation modal
- WS template dropdown on scheduler create/edit modals
- Prompt template name validation on ws_template create/update
- 7 console admin API endpoints + read-only summary endpoint
- Full OpenAPI spec entries in console_spec.py
- Python SDK (sync + async) and TypeScript SDK methods
- Pydantic schemas for all request/response models

Docs & diagrams:
- New 21-ws-template-architecture.puml sequence diagram
- Updated governance, storage, MQ protocol diagrams + PNGs
- Updated architecture.md, governance.md, api-reference.md, console.md, sdk.md

48 new tests (1788 total). mypy clean. ruff clean.

* fix: address PR #49 review feedback

- auto_approve_tools uses approval_label (not just func_name) for
  consistency with tool policy evaluation
- inline system_prompt from ws_template persisted as
  _ws_template_system_prompt in workstream_config, restored on resume
  (previously lost because _template_content wasn't persisted)
- budget gate (__budget_override__) no longer bypassed by blanket
  auto_approve — requires explicit approval or tool policy allow
- diagram 21 field list corrected (removed tool_search/threshold,
  added prompt_template_hash/notify_on_complete)

* fix: address PR #49 review feedback (round 2)

- Grant admin.ws_templates permission in migration 011 (tab was hidden)
- Center WS template modals and fix radio button alignment
- Skip template validation when ws_template overrides prompt
- Guard against empty version snapshots on no-op updates
- Replace setTimeout race with Promise chain in schedule ws_template select
- Validate numeric fields in admin create/update handlers (400 not 500)
- Add ws_template to TypeScript OpenAPI specs
- Use typed Pydantic response models in SDK ws_template methods
2026-03-12 21:06:51 -07:00
Patrick Buckley f1f448277f chore: bump version to 0.5.6
Prompt template runtime wiring, security hardening, scheduler/channel/MQ
template support, migration 010.
2026-03-12 16:58:20 -07:00
Patrick Buckley 2f7f70825b feat: wire prompt templates into session startup with full creation-p… (#47)
* feat: wire prompt templates into session startup with full creation-path support

Prompt templates (prompt_templates table) now have runtime effect:

- is_default=true templates auto-apply as system message content,
  concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
  field on POST /v1/api/workstreams/new, console creation modal dropdown,
  scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
  regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession

Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
  server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion

Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.

Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.

Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.

Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.

* fix: address PR #47 review feedback

- Defer template validation until after resume_ws — a bad template name
  no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
  openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
  reject unknown template names with 400 instead of allowing schedules
  that would silently fail at dispatch time
2026-03-12 16:57:31 -07:00
Patrick Buckley 4866c9873c feat: ddgCluster compose profile with DuckDuckGo Search MCP sidecar (#48)
Add ddgCluster profile extending the 10-node cluster with a DuckDuckGo
Search MCP sidecar. All cluster nodes connect via streamable-http and
gain duckduckgo_web_search + duckduckgo_fetch_content tools. No API
key required.

Key implementation details learned during testing:
- MCP SDK DNS rebinding protection must be disabled for Docker
  internal networking (Host header uses container names)
- FastMCP server binds to 127.0.0.1 by default; must set
  mcp.settings.host='0.0.0.0' for cross-container access
- DDG CLI lacks --host/--port flags; settings configured via Python
  entry point that patches FastMCP.settings directly
- Safe search disabled by default

Also adds MCP_CONFIG env var support to all server commands (shell
conditional, no-op when empty) and moves default server/bridge to
production profile for cleaner profile separation.
2026-03-12 16:51:53 -07:00
Patrick Buckley 8b2e2130fc fix: MCP resource template URI expansion via prefix matching (#46)
* fix: MCP resource template URI expansion via prefix matching

Resource templates (RFC 6570 URI patterns like `db://tables/{table}/rows/{id}`)
were discovered from MCP servers but non-functional — `read_resource_sync()`
only accepted exact URIs from `_resource_map`, which excludes templates.

Add prefix-based fallback: extract the static prefix from each template
(everything before the first `{`), store a prefix→server mapping, and
fall back to longest-prefix matching when exact URI lookup fails. MCP
servers handle URI routing internally so we just need to route the
expanded URI to the correct server.

Also surface templates in the system message catalog and `/mcp` command
so the model knows they exist and can construct expanded URIs.

* fix: address PR #46 review feedback

- Template prefix collision now keeps more specific (longer) template
  URI instead of blindly overriding
- Fix _match_template docstring to accurately describe startswith
  matching on static prefixes (not full template matching)
- Add missing loop.close() in integration test finally block
- Rewrite test_template_longest_prefix_wins with genuinely different
  prefix lengths to avoid brittle collision-order dependency
2026-03-12 15:46:00 -07:00
Patrick Buckley f81c06761d chore: remove dead code, add MCP integration + collector tests (#45)
* chore: remove dead code, add MCP integration + collector tests

Remove unused delete_prompt_templates_by_server from protocol and
both storage backends (sync uses per-template deletion).

Add 10 MCP integration tests exercising full lifecycle: rebuild
resources/prompts, read_resource_sync/get_prompt_sync with real
asyncio loop, governance sync to real SQLite, shutdown cleanup,
listener notification isolation.

Add 3 console collector MCP aggregation tests: multi-node sums,
absent when zero, mixed nodes with/without MCP.

* fix: close event loops and SQLite backend in MCP integration tests
2026-03-12 15:16:51 -07:00
Patrick Buckley be165c1971 feat: MCP resource and prompt discovery with read_resource tool (#44)
* feat: MCP resource and prompt discovery with read_resource tool

Extends MCPClientManager with resource and prompt discovery alongside
existing tool support. Resources and prompts are discovered on connect,
cached per-server with copy-on-write rebuilds, and refreshed via push
notifications, periodic polling, or manual /mcp refresh.

New read_resource built-in tool reads MCP resources by URI. Requires
user approval (same as MCP tool calls) since resources are served by
external MCP servers. Resource catalog injected into system message
with XML delimiters. Error messages sanitized to prevent leaking
server internals to the model.

Prompt discovery stores prefixed names (mcp__server__prompt) and
exposes get_prompt_sync() for future use_prompt tool (Chunk D).

/mcp command now shows tools, resources, and prompts. Docs and
diagrams updated.

* feat: MCP prompt governance sync with origin tracking and readonly guards

Migration 009 adds origin, mcp_server, and readonly columns to
prompt_templates. MCP prompts discovered by MCPClientManager are
automatically synced into the governance table as read-only templates
with origin="mcp".

Sync engine handles: create on connect, update on prompt refresh,
delete when prompts are removed from server. Manual templates take
precedence on name collision (MCP prompt skipped with warning).

Admin API returns 403 on update/delete of readonly templates. Console
UI shows MCP origin badge and disables edit/delete buttons. Storage
backends gain get_prompt_template_by_name, list_prompt_templates_by_origin,
and delete_prompt_templates_by_server methods.

Also addresses PR #44 review feedback: concurrent.futures.TimeoutError
handling in sync dispatch, XML-escape resource catalog descriptions,
resource template entries excluded from _resource_map, URI collision
warnings, needs_periodic capability-aware computation, malformed JSON
primary key fallback for read_resource.

* feat: use_prompt tool, prompt catalog, and PR review hardening

New use_prompt built-in tool invokes MCP prompt templates by name,
expanding them into messages. Requires user approval (external MCP
servers). Prompt catalog injected into system message with XML
delimiters (up to 30 prompts, HTML-escaped).

Prompt listener registered in session for catalog rebuild on changes.

Addresses PR #44 review feedback:
- _init_system_messages() now uses copy-on-write (build locally,
  assign atomically) so background thread callbacks never see
  partial system messages
- sync_prompts_to_storage() serialized behind _sync_lock to prevent
  races between set_storage() (main thread) and MCP background thread
- shutdown() clears listener lists to release callback references

Docs and diagrams updated for 18 built-in tools.

* feat: granular tool policies for MCP resources, prompts, and tools

Policy evaluation now uses approval_label (falling back to func_name)
for fnmatch pattern matching, enabling fine-grained per-URI and
per-server policies:
- read_resource: mcp_resource__{normalized_uri}
- use_prompt: mcp__{server}__{prompt} (prefixed name)
- MCP tools: mcp__{server}__{tool} (was static "mcp_tool")

URI normalization resolves .. path segments to prevent traversal
bypasses in policy matching. Resource templates filtered from system
message catalog (not directly readable). use_prompt arguments
validated as dict with string coercion.

TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly
fields. Governance docs updated with MCP policy patterns.

* feat: MCP visibility in server and console UIs

Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts
counts. Server UI status bar shows magenta MCP indicator with tooltip.
Console cluster status bar shows MCP metrics with magenta LED dot.
Console node detail view shows per-node MCP summary. Console collector
aggregates MCP counts across nodes in overview.

Uses var(--magenta) design token with new --magenta-glow for theme
adaptation. ARIA roles on MCP status elements. Tooltips on console
MCP metric labels. Node MCP summary hidden on mobile (< 700px).

New diagram: 20-mcp-architecture.puml covering full MCP lifecycle
(connection, discovery, refresh, governance sync, policy, UI).

* fix: McpStatus in health schema, count properties, catalog name fidelity

Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so
typed clients see the mcp field from /health.

Addresses Copilot review feedback:
- resource_count/prompt_count properties avoid list allocation on
  /health and /metrics polls
- get_tools/resources/prompts return shallow-copied dicts to prevent
  callers from mutating internal cache
- Prompt names and arg names in system message catalog are NOT
  HTML-escaped (model must use exact strings in use_prompt calls);
  only descriptions are escaped

* fix: OpenAPI spec McpStatus + diagram approval column accuracy

Adds McpStatus schema and optional mcp field to HealthResponse in
openapi-server.json, matching the Python schema and TypeScript types.

Fixes tool pipeline diagram: math, web_fetch, web_search correctly
shown as auto-approve (not "Yes" for approval).
2026-03-12 14:49:58 -07:00
Patrick Buckley 3264fdefca fix: channel bidirectional routing — emit TurnCompleteEvent on all id… (#43)
* fix: channel bidirectional routing — emit TurnCompleteEvent on all idle transitions

Bridge previously only emitted TurnCompleteEvent for MQ-initiated turns
(those with a correlation_id in _active_sends). Server-UI-initiated turns
went idle without emitting TurnCompleteEvent, so the Discord bot's
StreamingMessage never finalized — content accumulated in the buffer and
collided with the next Discord-triggered response.

Now TurnCompleteEvent is emitted unconditionally on every idle transition.
correlation_id is empty for non-MQ turns; SDK client filters by
correlation_id so existing consumers are unaffected.

* fix: remove unused variable flagged by ruff
2026-03-12 11:57:15 -07:00
Patrick Buckley 28cb3a5c51 fix: approval timeout UI state and content flush before tool calls (#42)
* fix: approval timeout UI state and content flush before tool calls

Two bug fixes:

1. Approval timeout now shows denied state in UI — resolve_approval()
   emits an approval_resolved SSE event so the browser transitions
   from pending to denied (red border + badge). Also fixes the cancel-
   during-approval path. Frontend resolveInlineApproval() gains a
   skipPost parameter to avoid redundant POST when server-initiated.
   ApprovalResolvedEvent added to Python and TypeScript SDKs.

2. Content streaming flushes pending buffer before tool call deltas —
   _stream_response() held up to 13 trailing chars in the pending
   buffer (for <think> tag detection) when transitioning to tool calls.
   Now flushed eagerly when tool_call_deltas arrive, before clearing
   in_think so reasoning text is correctly categorized.

* fix: address Copilot review feedback on PR #42

Patch _execute_tools in stream flush test to prevent real bash execution,
simplify confusing nested comprehension, and update resolve_approval()
docstring to reflect cancel/timeout call paths.
2026-03-12 11:43:06 -07:00
Patrick Buckley 8b11e0a6f9 fix: bridge retries node_id fetch indefinitely with capped backoff 2026-03-12 11:12:03 -07:00
Patrick Buckley 648ba477e1 refactor: list_user_roles uses _row_to_dict instead of positional row mapping 2026-03-11 21:14:58 -07:00
Patrick Buckley 7960784786 fix: usage events now record per-request tool_calls delta, not cumulative total 2026-03-11 21:12:03 -07:00
Patrick Buckley e06554d1ec feat: add channel admin endpoints to console OpenAPI spec 2026-03-11 21:08:28 -07:00
Patrick Buckley 8eb8722346 Bump version to 0.5.5 2026-03-11 20:24:12 -07:00
Patrick Buckley a2e2ffacd8 feat: robust plan quality gate, iterative refinement, and amend UX (#41)
* feat: robust plan quality gate, iterative refinement, and amend UX

Plan agent output from weak models often produced garbage (11-char plans
that echo the prompt). Two fixes:

1. Quality validation (_validate_plan) checks length, section structure,
   echo detection, and refusal patterns. Fails trigger one automatic
   retry with a coaching message injected into the agent's existing
   conversation, preserving all prior exploration context.

2. Iterative feedback loop — user feedback at plan review re-runs the
   plan agent via _refine_plan() instead of appending text to the tool
   result. Up to 5 refinement rounds. The plan file path is always
   included in the tool result so the outer model knows where it lives.

UI improvements:
- Web: Reject button dynamically becomes "Amend" (amber) when feedback
  is typed. Key hint badges (Esc/Enter) on plan buttons. Main input
  disabled during review. Light-theme contrast fix via --on-color var.
- CLI: Prompt shows all three actions (approve/amend/reject).
- Bridge: Race condition fix — clear pending entry before HTTP POST so
  sequential plan reviews from the refinement loop aren't skipped.

15 new tests covering validation, retry, and refinement.

* fix: address PR 41 review feedback

- Escape key in plan dialog now mirrors the Amend button: if feedback is
  typed, Esc sends the feedback (amend); if empty, Esc rejects. Previously
  Esc always hard-coded "reject", discarding typed feedback.

- Coaching message for plan retry now says "should include at least two of"
  instead of "MUST include these", matching the actual validation rule
  (_MIN_PLAN_SECTIONS = 2).

* feat: render plan inline in chat after approval

After the plan review dialog closes, the plan content is now rendered
as a collapsible inline block in the chat stream — styled with a
status header (approved/rejected/amending), markdown-rendered body,
and feedback note when amending. Uses the same makeCollapsible pattern
as tool output blocks.

* fix: prevent plan approval hang when inline render fails

The authFetch call that unblocks the server must fire before the
cosmetic inline plan rendering. Previously _addInlinePlan ran first
and any JS error (e.g. from renderMarkdown) prevented the API call,
leaving the session thread blocked forever.

- Move authFetch before _addInlinePlan
- Wrap _addInlinePlan in try-catch
- Guard against empty content
- Only auto-collapse plans longer than 12 lines

* fix: address PR 41 review feedback (round 2)

- Max refinement rounds no longer implicitly approve: the loop now
  shows the final plan for explicit approve/reject before proceeding.
  Previously exhausting 5 rounds silently accepted the last revision.

- Plan inline block: correct aria-label from "Tool output" to
  "Plan content" when makeCollapsible is applied.

- XSS concern (not applicable): renderMarkdown is used for all
  assistant messages — plan content follows the same trust model.

- Test loop concern (acknowledged): refinement tests verify component
  logic; full _execute_tools integration would require extensive
  mocking for marginal coverage gain.

* feat: thinking spinner + inline plan hardening

* fix lint
2026-03-11 20:22:41 -07:00
Patrick Buckley c6ba8d59b0 feat: bootstrap wizard — LLM-guided interactive setup for deployments
Add `turnstone-bootstrap`, a new entry point that uses any LLM (OpenAI,
Anthropic, or local/vLLM) to conversationally walk users through
configuring a Turnstone deployment. Generates .env files, setup.sh
scripts, and optional docker-compose overrides.

- Fully interactive startup (zero CLI args) with provider/model selection
- Auto-detects available models on local OpenAI-compatible endpoints
- 7 tools: read_file, write_file, generate_secret, check_port,
  validate_api_key, check_docker, finish
- Path traversal protection on file read/write
- Duplicate write detection (skips identical content)
- Bounded retry loop (3 attempts) on LLM errors
- Anthropic message conversion with consecutive-role merging
2026-03-11 01:58:39 -07:00
Patrick Buckley 087f5b49f6 Bump version to 0.5.4 2026-03-10 20:47:40 -07:00
Patrick Buckley fd507c6a3c feat: generation cancellation — stop button, cancel API, cooperative … (#40)
* feat: generation cancellation — stop button, cancel API, cooperative cancel

Add cooperative cancellation via threading.Event on ChatSession. The cancel
signal is set from outside the worker thread (HTTP handler, MQ bridge, or
Escape key) and checked at defined checkpoints: per streaming chunk, before
tool execution, inside bash commands, and at each sub-agent turn.

Core: GenerationCancelled(BaseException) exception, cancel()/_check_cancelled()
methods, partial content preservation in _stream_response, clean rollback in
send() with idle state emission (no re-raise).

Server: POST /v1/api/cancel endpoint, CancelledEvent SSE emission, worker
thread safety net.

Frontend: Stop button (■ Stop) with send/stop swap via setBusy(), Escape key
shortcut, cancelled event handler. Accessible: aria-label, focus-visible
override, light theme contrast, non-color differentiation.

MQ: CancelMessage inbound type, bridge _handle_cancel routed handler.

SDK: cancel() on Python async+sync clients, CancelledEvent in Python+TypeScript
event registries, isCancelledEvent type guard.

OpenAPI: CancelRequest schema + endpoint spec.

Docs: API reference, architecture, SDK docs updated. Diagrams: conversation
turn, tool pipeline, MQ protocol, workstream states, SDK architecture.

* fix: address PR #40 review feedback

- setBusy() now resets stopBtn.disabled so stop button is re-enabled on
  next generation after a successful cancel
- Gate cancel side effects (resolve_approval, resolve_plan, cancelled SSE
  event) on worker_thread.is_alive() to avoid spurious events when idle
- Add /v1/api/cancel endpoint and CancelRequest schema to TypeScript
  openapi-server.json to keep it in sync with Python-generated spec
2026-03-10 20:43:52 -07:00
Patrick Buckley 562c3c8ab7 docs: add governance section and missing diagrams to README 2026-03-10 19:50:07 -07:00
Patrick Buckley 4773535bb8 docs: add governance architecture diagram PNG 2026-03-10 19:41:35 -07:00
Patrick Buckley 7492816ab2 feat: governance — RBAC, tool policies, prompt templates, usage track… (#39)
* feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging

Add comprehensive governance layer for the admin console:

- RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer),
  custom role CRUD, user-role assignment with privilege escalation prevention
- Tool policies with glob pattern matching, priority-ordered evaluation
  (allow/deny/ask), enforced before auto-approve in WebUI.approve_tools()
- Prompt templates with variable substitution, categories, default flag
- Usage tracking: per-LLM-request token/tool metrics, aggregated queries
  (group by day/model/user), automatic 90-day pruning via scheduler
- Audit logging: append-only event trail for all admin mutations,
  filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For
  aware IP extraction
- require_permission() enforced on all 35+ admin endpoints (users, tokens,
  channels, schedules, watches, roles, orgs, policies, templates, usage, audit)
- Field allowlists on storage update methods prevent mass-assignment bugs
- Self-deletion guard on admin_delete_user, delete_user cascades user_roles
- _row_to_dict helper eliminates ~400 lines of fragile positional row mapping
- _audit_context helper deduplicates 18 instances of audit boilerplate
- Migration 008: 7 new tables, 3 builtin roles, org_id on users
- Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit)
  with permission-gated visibility, 7 modal dialogs, full keyboard accessibility
- Python + TypeScript SDK methods for all governance endpoints
- 120+ new tests (1554 total)

* fix: address PR #39 review feedback

- Rebuild serialized items after policy evaluation so denied/allowed
  verdicts are reflected in tool_info/approve_request SSE payloads
- Make `since` query param optional in usage OpenAPI spec (handler
  already defaults to last 7 days)
- Add response_model=StatusResponse to DELETE role/policy/template
  and POST/DELETE role assignment endpoints in OpenAPI spec
- Add missing org_id/created/updated fields to UserRoleInfo schema
- Add missing created field to AuditEventInfo schema
- Show "no permissions" empty state instead of loading inaccessible
  tab when all admin tabs are permission-gated
- Fix "13 permissions" → "15 permissions" in architecture.md and
  security.md
- Fix import sorting in test_audit.py and test_tool_policy.py

* fix: address PR #39 round 2 review feedback

- Clear stale permissions from sessionStorage on config-token login
  (auth.js _storePermissions)
- Only trust X-Forwarded-For when behind a proxy that sets
  X-Forwarded-Proto (conditional on is_secure_request trust model)
- Thread user_id from auth into WebUI.on_status for usage events
- Add created field to TS AuditEventInfo type
- Return typed Pydantic models from all SDK governance methods instead
  of dict[str, Any] — both async and sync clients
- Validate group_by param against allowed enum in admin_usage handler
- Add deterministic secondary sort (event_id DESC) to
  list_audit_events in both SQLite and PostgreSQL backends
2026-03-10 19:32:37 -07:00
Patrick Buckley d6ba1d5e25 fix: mypy no-any-return in agent context overflow handler 2026-03-10 14:11:09 -07:00
Patrick Buckley 41d1b27d34 Bump version to 0.5.3 2026-03-10 13:46:50 -07:00
Patrick Buckley 8bc284c60e fix: agent context overflow — truncate tool output, catch context errors
Agent tool outputs are now truncated to 16k chars to prevent search
results (14M+ chars observed) from blowing past the model's context
limit. On context-exceeded API errors, the agent returns its last
content instead of crashing.
2026-03-10 13:43:01 -07:00
Patrick Buckley a322d6b1d1 fix: sub-agent context — clean plan, merged task, no Qwen template error
Plan agent: own identity only (no base system prompt needed).
Task agent: base system prompt merged with task identity into a single
system message (needs tool patterns for tool execution).
Neither agent receives conversation history.

Fixes Jinja template error on Qwen models that reject system messages
appearing after non-system messages.
2026-03-10 13:36:23 -07:00
Patrick Buckley 70d495aa5b fix: per-workstream SSE fan-out — multiple consumers no longer steal … (#38)
* fix: per-workstream SSE fan-out — multiple consumers no longer steal each other's tokens

After 4d665a5 removed the single-consumer SSE lock, the shared
_event_queue let concurrent consumers (browser, bridge, console proxy)
race on Queue.get(), each receiving ~1/N of content tokens and producing
garbled streaming text.

Replace the single queue with per-client fan-out: each SSE connection
registers its own bounded queue (maxsize=500) on WebUI._listeners, and
_enqueue() copies every event to all registered queues. On eviction or
close, a ws_closed sentinel is injected so SSE generators exit promptly.

* fix: address CI failures and Copilot review feedback

- Handle ws_closed sentinel in events_sse generator (break on close)
- Guarantee sentinel delivery by evicting one item when queue is full
- Clear listeners list after injecting sentinels on cleanup
- Fix test_slow_consumer to fill only slow queue directly
- Fix ruff SIM117 (nested with), unused import, mypy unused-ignore
2026-03-10 13:25:16 -07:00
Patrick Buckley de64535221 Feat/eval improvements (#37)
* ci: add GitHub Release creation on tag push

* refactor: rename plan tool to create_plan

Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns

* feat: eval harness improvements inspired by autoresearch patterns

Major enhancements to turnstone-eval:

- Per-test timeout (--test-timeout, default 300s) and suite timeout
  (--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
  models and providers (--optimizer-model, --observer-model, etc.)
  with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
  failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)

* fix: address Copilot review feedback

- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
2026-03-10 13:06:39 -07:00
Patrick Buckley 187d004033 feat: watch tool — periodic command polling within workstreams (#36)
* feat: watch tool — periodic command polling within workstreams

Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.

Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)

* fix: address Copilot review — condition errors, restore deadlock, docs

- Condition eval errors now deactivate the watch immediately instead
  of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
  approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
  detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
2026-03-10 08:18:28 -07:00
Patrick Buckley 7ea150fa71 Bump version to 0.5.2 2026-03-09 13:42:28 -07:00
Patrick Buckley 4d665a5f62 fix: SSE reconnect loop — remove _sse_generation single-consumer lock
The _sse_generation mechanism assumed one SSE consumer per workstream,
but the bridge also maintains an SSE connection to each workstream.
When a new client connected (browser, proxy, or test), it incremented
the generation counter, killing the bridge's connection. The bridge
reconnected, killing the new client's connection — creating a
mutual-kill cascade that closed every SSE connection after one ping
cycle (5s).

Fix: remove _sse_generation entirely. sse-starlette handles disconnect
detection via its own ASGI task. Also remove the redundant
request.is_disconnected() check which raced with sse-starlette's
disconnect listener in Starlette 0.52.

Root cause confirmed via raw socket test: the server was sending
a zero-length chunked terminator (0\r\n\r\n) at exactly 5s,
cleanly ending the HTTP response body.
2026-03-09 13:40:36 -07:00
Patrick Buckley 3bc3250869 fix: recovered workstreams invisible in console UI (#35)
* fix: recovered workstreams invisible in console UI

Bridge startup recovery (_recover_workstreams) re-registered workstream
ownership but never published WorkstreamCreatedEvent to the cluster
channel. The collector's poll loop would pick up the workstream in its
internal state, but _apply_poll never fanned out SSE events to connected
browsers. Combined, this made channel-resumed workstreams invisible in
the console while remaining accessible through the proxied node UI.

- Bridge: emit WorkstreamCreatedEvent for each recovered workstream
- Collector: diff poll results and fan out synthetic ws_created/ws_closed
  events for workstream additions and removals
- Skip workstreams with empty IDs in poll processing
- Add 4 tests for poll-diff fanout behavior
- Update console data-flow diagram and architecture docs

* fix: address PR review — filter empty ws IDs, stable event ordering

- Filter empty-string keys from old_ids to avoid phantom ws_closed
  events if a previous poll inserted a workstream under key "".
- Sort set diffs before iterating so ws_created/ws_closed fanout
  order is deterministic across poll cycles.
2026-03-09 13:39:47 -07:00
Patrick Buckley db937486cf Bump version to 0.5.1 2026-03-09 01:48:18 -07:00
Patrick Buckley 554257ac4d fix: SSE proxy Firefox reconnect — Connection: keep-alive header 2026-03-09 01:46:35 -07:00
Patrick Buckley 5f0004dc91 feat: add ClusterSnapshot for instant console UI state rebuild (#34)
* feat: add ClusterSnapshot for instant console UI state rebuild

The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.

Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).

Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.

SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.

* fix: address review feedback and SSE proxy reconnect bug

Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
  both state and listener locks, eliminating the event gap between
  snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
  to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
  console schema models (ClusterNodeInfo, NodeDetailResponse,
  ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
  and ClusterOverviewResponse types instead of Record<string, unknown>.

SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
  when no upstream data arrives, preventing the browser EventSource
  from dropping idle connections. The raw byte passthrough refactor
  (4d11078) removed the proxy's independent keepalive — this restores
  it without reverting to EventSourceResponse.
2026-03-09 01:13:35 -07:00
Patrick Buckley 6cc1b3a5bd feat: add vision/image support to read_file tool (#33)
* feat: add vision/image support to read_file tool

read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO)
and returns base64-encoded content parts for vision-capable models.
Non-vision models receive a text description instead. A new
supports_vision flag on ModelCapabilities gates the feature, with
config.toml [models.*.capabilities] overrides for local models
(vLLM, llama.cpp, NIM).

* fix: address PR review feedback

- Discard _read_files on no-vision OSError path, include exception detail
- Discard _read_files on oversized image error (not a successful read)
- Validate capabilities type from config.toml (reject non-dict)
- Clarify tool description re: vision behavior and offset/limit scope
- Remove unused os import in tests, fix import sort order
- Handle list content (image tool results) in eval.py tool result loop
2026-03-08 23:43:42 -07:00
Patrick Buckley cc9afe94cd get title in collector for console 2026-03-08 22:32:52 -07:00
Patrick Buckley 136b75fdef Bump version to 0.5.0 2026-03-08 04:47:10 -07:00
Patrick Buckley 4d1107839b refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing

- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service

* Refactor SSE proxy to raw byte passthrough

- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
  httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
  events, and comments now flow through verbatim. Added per-request timeout
  override (read=None, pool=None) for long-lived SSE streams.

- tests/test_console.py: Add 3 new tests for SSE proxy:
  - Ping and event preservation
  - Upstream error status handling
  - Client disconnect handling

- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
  approach.
2026-03-08 04:46:34 -07:00
Patrick Buckley 7d66bc2159 Bump version to 0.4.6 2026-03-08 03:44:22 -07:00
Patrick Buckley 165cbb2d29 Bump version to 0.4.5 2026-03-08 03:29:44 -07:00
Patrick Buckley c79c47b940 Add MCP dynamic tool refresh with push notifications and periodic pol… (#31)
* Add MCP dynamic tool refresh with push notifications and periodic polling

MCP tool lists now stay up-to-date without restart via three mechanisms:
push notifications (ToolListChangedNotification) for servers that support
it, staggered periodic polling for servers that don't, and manual
/mcp refresh [server] command. MCPClientManager tracks tools per-server
with copy-on-write rebuild, notifies ChatSession listeners which rebuild
tool lists and ToolSearchManager (preserving expanded tools).

* Address Copilot review feedback on MCP refresh PR

- Fix /mcp refresh typo matching (startswith → exact token check)
- Validate --mcp-refresh-interval >= 0 at parse time via shared
  nonneg_float in config.py (deduplicated from cli.py + server.py)
- Clamp negative refresh_interval to 0 in MCPClientManager constructor
- Fix periodic refresh first poll timing (was initial_delay + interval,
  now initial_delay then immediate first poll)
- Clarify _on_mcp_tools_changed docstring re: O(n) BM25 build cost
2026-03-08 03:28:38 -07:00
Patrick Buckley 660c273e8e remove old demo.svg 2026-03-08 01:47:34 -08:00
Patrick Buckley c7586abd0a Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI

When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:

- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool

New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.

Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.

43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.

* Fix Copilot review feedback on tool search

- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
2026-03-08 01:43:38 -08:00
Patrick Buckley 14d57176ce Bump version to 0.4.4 2026-03-07 15:43:56 -08:00
Patrick Buckley 96084ca5f3 Fix PostgreSQL migration race condition with advisory lock
Multiple containers starting simultaneously race on Alembic migrations
against shared PostgreSQL. Use pg_advisory_lock so they wait in line.
Also update SQLite bootstrap to detect post-migration databases.
2026-03-07 15:41:29 -08:00
Patrick Buckley 8b92302247 Bump version to 0.4.3 2026-03-07 13:11:42 -08:00
Patrick Buckley e195ca54a6 new high level arch abstract 2026-03-07 13:01:19 -08:00
Patrick Buckley 50277cd4de Add 10-node cluster profile to Docker Compose 2026-03-07 12:54:41 -08:00
Patrick Buckley fb190f8977 Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity

Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).

Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id

Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days

Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.

* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec

- docs/channels.md: "resumes the session" → "resumes the workstream",
  "Session resumed:" → "Resumed:", "old session was pruned" → "old
  workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
  object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
  models — removes all stale session_id properties from WorkstreamInfo,
  DashboardWorkstream, CreateWorkstreamResponse schemas
2026-03-07 12:49:07 -08:00
Patrick Buckley 25b5e32089 Bump version to 0.4.2 2026-03-05 20:18:46 -08:00
Patrick Buckley 339981a258 Add GPT-5.3, GPT-5.4, and pro model capabilities (#28)
* Add GPT-5.3, GPT-5.4, and pro model capabilities

Add capability entries for gpt-5-pro (272k output, high-only reasoning),
gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro.

* Validate reasoning_effort against model capabilities

_apply_model_params now falls back to caps.default_reasoning_effort when
the requested value is not in caps.reasoning_effort_values. Prevents
sending unsupported effort levels to models like gpt-5-pro (high only).
2026-03-05 20:17:03 -08:00
Patrick Buckley 06de9ff83b Bump version to 0.4.1 2026-03-05 18:13:42 -08:00
Patrick Buckley 924b976f1f Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
2026-03-05 18:11:36 -08:00
Patrick Buckley d5db817391 Fix channel gateway Docker networking and console proxy approval scope
Two runtime bugs:

1. Channel gateway advertised http://127.0.0.1:8091 which is
   unreachable from other Docker containers. Add
   TURNSTONE_CHANNEL_ADVERTISE_URL env var override for Docker/K8s
   environments, set to http://channel:8091 in compose.yaml, and
   pass --http-host=0.0.0.0 so the gateway listens on all interfaces.

2. Console proxy service JWT had only "write" scope but the approval
   endpoint requires "approve". Tool approval buttons in the server
   web UI silently failed when accessed through the console proxy.
   Changed proxy token scopes to read+write+approve.
2026-03-05 18:01:52 -08:00
Patrick Buckley fc8ceb4c72 Update README: 3-node cluster diagram, remove directory tree
Replace single-node Mermaid diagram with a 3-node cluster layout
showing bridge+server pairs per node, shared Redis MQ, console, and
channel gateway. Remove the verbose directory tree listing.
2026-03-05 17:44:33 -08:00
Patrick Buckley 07234dec4d Replace ASCII architecture diagram with Mermaid in README
GitHub renders Mermaid natively as an interactive SVG. The new diagram
shows all client entry points (CLI, browser, SDK, Discord), the full
cluster topology including the channel gateway and notify path, and
the LLM provider layer.
2026-03-05 17:34:42 -08:00
Patrick Buckley dd4cc0b30d Add PROGRESS.md and .coverage to .gitignore 2026-03-05 17:31:55 -08:00
Patrick Buckley e7fe8fca9d Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening

Implements the `notify` tool allowing the LLM to send notifications to
Discord channels/users via the channel gateway. Includes fixes for 11
review findings: JWT auth on the gateway endpoint, first-healthy gateway
delivery with retry+backoff, rate limiting only on success, SSRF URL
scheme validation, Discord mention sanitization, SQLite ON CONFLICT
upsert preserving created timestamps, advertise URL resolution for
0.0.0.0 bind, randomized service IDs, generic error messages to prevent
internal state leakage, and partial direct-target validation.

Service registry with heartbeat-based health filtering (migration 005).
Channel gateway registers on startup, heartbeats every 30s, deregisters
on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint
auth (static + JWT), storage CRUD, and retry behavior.

* Add notify documentation, diagrams, and review fixes

Documentation:
- New sequence diagram 17-notify-flow.puml showing end-to-end delivery
- Updated 16-channel-architecture.puml with services table, notify HTTP
  path, and Notification Flow note
- channels.md: Notifications section (targeting, delivery flow, service
  registry, security) and new config table entries
- tools.md: notify tool reference, updated counts/tables (14→15 tools)
- security.md: channel gateway row in service-to-service auth table
- architecture.md: notification subsystem paragraph

Review fixes (copilot):
- _http.py: fail closed when auth unconfigured (401 instead of pass-
  through), strip whitespace on message/title, generic error messages
  for user-not-found vs no-linked-channels
- session.py: parse gateway response JSON and require at least one
  result with status=="sent" before counting as success
- _postgresql.py: use index_elements instead of constraint for upsert
2026-03-05 17:24:00 -08:00
Patrick Buckley 42b9f89988 Add scheduled task system with cron/at scheduling (#26)
* Add scheduled task system with cron/at scheduling, admin API, and console UI

Console-integrated background scheduler dispatches workstreams on recurring
cron expressions or one-shot ISO8601 timestamps. Four target modes: auto
(best node by headroom), pool (shared queue), all (fan-out), or specific
node. Redis distributed lock with unique owner + Lua conditional release
prevents duplicate dispatch in multi-console deployments.

Storage: scheduled_tasks + scheduled_task_runs tables (migration 004),
9 protocol methods on both SQLite and PostgreSQL backends, field allowlist
on updates, run history auto-pruned at 90 days.

API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter
validation, ISO8601 future-time checks, field length bounds, schedule
count cap (200), and OpenAPI spec entries with Pydantic models.

UI: Schedules tab in admin panel with create/edit/delete modals, run
history modal, cron/at type toggle, target mode select, status dots for
accessibility, responsive grid, keyboard navigation, and focus management.

Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches
logged at WARNING with created_by attribution, user_id propagated in
CreateWorkstreamMessage for audit trail.

46 tests across storage, scheduler engine, and API endpoints.

* Add croniter to test extras for CI compatibility

CI installs [test] extras but not [console], so croniter was missing
when schedule API tests import console/server.py validation functions.

* Address Copilot review: timezone validation, focus trap, enabled flag

- Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z)
- UI appends +00:00 to datetime-local values for explicit UTC
- Fix datetime-local normalization: check length before appending seconds
- Add textarea to modal focus trap selector (prevents focus escape)
- Fix _normalize_task_dict not called in update response
- Persist enabled=false on create (storage defaults to enabled=1)
- Validate at_time is still in future when re-enabling a one-shot task
- broker._redis coupling acknowledged as tracked tech debt
2026-03-05 16:05:00 -08:00
Patrick Buckley 77c0a7736b Bump version to 0.4.0 and update security docs
- Version bump in __init__.py, pyproject.toml, api-reference.md
- security.md: document JWT aud/iss claims, login rate limiting,
  secure cookie defaults (24h, Secure flag), CORS restriction,
  service JWT auto-rotation, secret strength validation, and
  proxy auth forwarding via service tokens (not user JWT forwarding)
2026-03-04 20:45:00 -08:00
Patrick Buckley 872e1770e6 Feature/code dedup (#25)
* Add JWT auth security hardening (6 fixes)

- Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h
- Login brute-force protection: LoginRateLimiter (5 attempts/5min per key)
- JWT aud/iss claims: create_jwt/validate_jwt support audience validation
- Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh
- CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var
- JWT secret strength: warning on secrets shorter than 32 chars
- Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing

* Refactor duplicated code into shared utilities and fix 3 UI bugs

Code deduplication (~235 net lines removed):
- Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py
- Create core/web_helpers.py (require_storage_or_503, read_json_or_400,
  parse_cors_origins, cors_middleware)
- Extract add_redis_args/broker_from_args to mq/broker.py
- Extract add_log_args/configure_logging_from_args to core/log.py
- Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper,
  unused required_role(), duplicate detect_model() wrapper

Bug fixes:
- Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server
  nodes (use ServiceTokenManager with JWT_AUD_SERVER instead)
- Fix login form autofill: wrap inputs in <form>, add name attributes,
  set type=submit on button
- Fix SSE reconnecting flash: add onopen handler to clear status
  immediately on connection (not waiting for first message)
- Fix chat scroll: add min-height:0 to flex containers, overflow:hidden
  on body to constrain viewport height

* Address CI typecheck failure and Copilot review feedback

- Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary)
- Bridge SSE loops: use event_hooks for auth header refresh on reconnect
  instead of static headers that go stale after token rotation
- Login form: remove javascript:void(0) action (CSP anti-pattern)
- Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals
  in middleware builder calls to prevent drift
2026-03-04 20:35:21 -08:00
Patrick Buckley a6e929b0a0 Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24)

Bidirectional channel adapter framework connecting external messaging
platforms to turnstone workstreams via Redis MQ. Discord ships as the
first adapter; the protocol supports future Slack/Teams integrations.

Channel framework:
- ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping
- AsyncRedisBroker with single dispatch loop and per-channel ordered workers
- channel_routes table (migration 003) for persistent route storage
- 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD)
- Unified turnstone-channel gateway entry point, loads adapters by config
- Message chunking, approval formatting, plan review formatting

Discord adapter:
- discord.py v2.4+ bot with thread-per-@mention model
- Slash commands: /link (modal), /unlink, /ask, /status, /close
- Persistent button views for tool approval and plan review
- Streaming responses via edit-in-place (1.5s interval)
- Stale route detection and atomic session resume via resume_session field
- SessionResumedEvent confirmation back to channel
- Auto-approve support (blanket + per-tool list)

Atomic session resume:
- resume_session field on CreateWorkstreamMessage for single-request resume
- Server resumes session during POST /v1/api/workstreams/new atomically
- Bridge emits SessionResumedEvent to per-workstream channel
- WorkstreamCreatedEvent extended with resumed/session_id/message_count
- Server UI dashboardResumeSession simplified to single request
- Pruned sessions fall back gracefully to fresh start

Service auth:
- Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET
- Bridge: approve scope (1 week). Console collector: read. Proxy: write.

Console admin:
- Channels tab with per-user view, force-link modal, unlink
- 3 admin API endpoints for channel user management
- Styled confirm modals replacing browser confirm() dialogs

Bug fixes:
- AsyncRedisBroker: replaced per-channel listener tasks with single
  dispatch loop + per-channel queue workers (fixes message stealing race)
- Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates
- Bridge: _active_sends tracked for initial messages (fixes missing
  TurnCompleteEvent and unfinalized streaming messages)
- Bridge: HTTP calls moved outside lock scope in approval handlers
- Bridge: _handle_send cleans up _active_sends on HTTP/server errors
- Formatter: reads server SSE format (func_name/preview) with fallback

Docs, SDK, tests:
- docs/channels.md setup guide, architecture diagram 16
- Updated api-reference.md, architecture.md, console.md, docker.md
- Python SDK: resume_session param on create_workstream (async + sync)
- TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces
- OpenAPI schema: resume_session request, resumed/message_count response
- 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing,
  18 discord, 12 resume flow) — 1120 total passing

* Fix CI lint/typecheck failures and address Copilot review feedback (#24)

Lint: fix import ordering, remove unused imports, use contextlib.suppress.
Mypy: explicit postgresql dialect import, add discord module overrides for
optional-dependency CI environments.
Copilot: fix double-escaping in admin confirm modals, return resolved
session_id from server resume response, fix channel_routes diagram schema,
use atomic setdefault for routing locks, add post-insert race guard in
admin channel create, support SSE format in auto-approve check, update
identity linking note in architecture diagram.

* Fix remaining mypy call-arg errors for discord.py optional dependency

Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class
definitions that fail when discord.py is not installed in CI.
2026-03-04 13:02:58 -08:00
Patrick Buckley 047680d669 Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23)

JWT-based authentication with three token types: config-file (hmac,
backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs
(HS256, 24h expiry). Username:password login via bcrypt. Hierarchical
scopes: read < write < approve.

New tables: users (username, password_hash), api_tokens (token_hash,
scopes, expires), channel_users (future channel integrations). user_id
column added to sessions and workstreams for attribution.

Console owns admin CRUD (6 endpoints under /api/admin/). Server
validates JWTs locally with shared signing secret. Public /api/auth/setup
endpoint for first-time admin creation (atomic, only works with zero
users). turnstone-admin CLI for user/token management.

Admin console UI: Users and Tokens tabs with full CRUD modals, scope
badges, token show-once with clipboard copy, keyboard accessibility
(focus traps, Escape, arrow key tabs, ARIA roles).

Login UI redesigned: username:password primary, token toggle for legacy,
setup wizard auto-detected via /api/auth/status. Python + TypeScript
SDKs updated with login(username, password), authStatus(), setup().

New docs/security.md + diagram 15-auth-architecture.puml. All existing
docs updated. OpenAPI specs include all new endpoints. 64 new tests
(1023 total). Dependencies: PyJWT, bcrypt.

* Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review

Address Copilot review feedback: escape double quotes in escapeHtml()
to prevent XSS in HTML attributes, add JWT validation fallback so
config tokens containing dots still work, add user_id to
AuthLoginResponse schema, return created field from admin_create_user,
and correct five documentation files to match actual API behavior.
2026-03-04 09:12:18 -08:00
Patrick Buckley 0fd0ad3b2d Add structured logging with structlog and context propagation
Replace ad-hoc logging.basicConfig() calls across all 6 entry points with
a centralized configure_logging() function backed by structlog. JSON output
when stderr is not a TTY (production/Docker), colored console output otherwise.

- New turnstone/core/log.py: configure_logging(), get_logger(), contextvars
  for node_id/ws_id/user_id/request_id auto-injected into every log event
- All entry points (server, bridge, console, sim, cli, migrate) call
  configure_logging() with --log-level and --log-format CLI flags
- Server operational print() calls replaced with structured log.info()
- LogContextMiddleware sets request_id + ws_id per HTTP request with
  token-based reset to prevent context leaking across requests
- Bridge _run_in_context() helper propagates ctx_node_id to child threads
- Env var overrides: TURNSTONE_LOG_LEVEL, TURNSTONE_LOG_FORMAT
- 18 new tests (959 total passing)
2026-03-04 06:47:30 -08:00
Patrick Buckley f3dba836dd Add Git LFS requirement note to README, Diagram PNGs are stored in LFS; git-lfs must be installed for cloning them. 2026-03-04 06:14:27 -08:00
Patrick Buckley 7adda343fc Update docs and diagrams for cluster-scale schema changes
- StorageBackend protocol: document 5 new workstream methods (26 total)
- Session ID: 12-char hex → 32-char full UUID in API reference
- /health endpoint: add node_id field to response docs
- sessions table: document node_id and ws_id columns
- Bridge node_id: document server-owned identity with /health retrieval
- Regenerate storage architecture PNG from updated PlantUML
2026-03-04 06:12:33 -08:00
Patrick Buckley a20a058c59 Add cluster-scale schema, fix console proxy UX, harden SDK sync runner (#22)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner

Schema redesign for multi-node deployments:
- New `workstreams` table with node_id, state, lifecycle tracking
- Add node_id + ws_id columns to sessions table with indexes
- Full UUID (32 hex) for session_id and ws_id (was truncated 12/8)
- Server generates and owns node_id, bridge retrieves via /health
- Bridge retries with exponential backoff, fatal on auth errors
- WorkstreamManager persists workstreams and state changes to storage
- /health endpoint exposes node_id for bridge discovery

Console proxy UX fixes:
- Remove duplicate turnstone branding from proxy banner
- Same-tab navigation for Open Node UI and workstream deep links

SDK _SyncRunner fix:
- Sentinel pattern for StopAsyncIteration across thread boundary

Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir).

* Address PR #22 review feedback

- Fix CLI session_factory signature (ws_id param) — CI typecheck failure
- First-phase eviction in create() now calls _cleanup_ui + record_eviction
- close() persists "closed" state to storage via update_workstream_state
- Fix noqa comment in test to pragma: no cover
2026-03-04 06:04:59 -08:00
Patrick Buckley 498f23c19e Bump version to 0.3.5 2026-03-04 00:21:28 -08:00
Patrick Buckley e057c364b8 Fix console proxy regressions and add workstream task field (#21) (#21)
* Fix console proxy regressions and add workstream task field (#21)

Bug fixes:
- Fix collector polling unversioned /api/dashboard (404 after API
  versioning PR) — nodes showed red/unreachable, no workstreams
- Fix SSE proxy dropping all data events — upstream sends \r\n line
  endings but proxy split on \n\n only; normalize before parsing
- Fix workstream state stuck on idle — on_state_change() only
  broadcasted via SSE but never updated ws.state on the Workstream
  object; dashboard polling now sees correct attention/running states
- Fix deep-link switchTab early return — when ?ws_id matched the
  only workstream, switchTab bailed (wsId === currentWsId) before
  establishing SSE connection; inline init instead of delegating
- Fix console banner covering dashboard overlay — inject <style>
  offsetting .dashboard-overlay below the 32px banner

Enhancements:
- Add turnstone branding to console proxy banner (turnstone │ Console │ node-id)
- Add initial_message field to CreateWorkstreamMessage protocol and
  console "New Workstream" modal (Task textarea, sent as first message)
- Refactor SSE proxy to use shared httpx client with 30s read timeout
  instead of per-request client creation
- Increase approval timeout default from 300s to 3600s (1 hour)

Updated: Python SDK, TypeScript SDK, OpenAPI specs, MQ client,
API schemas, MQ protocol diagram, SDK docs.

* Address PR #21 review feedback (4 items)

- Log unknown state strings in on_state_change instead of silently
  swallowing; remove unnecessary KeyError catch
- Wrap initial_message POST in _handle_create_ws with error handling
  so workstream creation success isn't masked by send failure
- Strip all \r from SSE chunks instead of replacing \r\n, fixing
  chunk-boundary split edge case
- Add tests for initial_message wiring in directed and pool targeting

* Refactor SSE proxy to use httpx-sse aconnect_sse

Replace manual SSE chunk buffering/parsing with httpx_sse.aconnect_sse()
which handles line endings, event types, and all SSE spec edge cases.
Eliminates the \r\n chunk-boundary bug class entirely. Event types are
now always forwarded (sse.event defaults to "message" per spec).
2026-03-04 00:18:48 -08:00
Patrick Buckley e785a94539 Fix Alembic migration auth failure with PostgreSQL
str(engine.url) in SQLAlchemy 2.x masks the password as '***',
causing SCRAM-SHA-256 authentication to fail when Alembic creates
its own engine from the config URL.
2026-03-03 23:09:31 -08:00
Patrick Buckley 2b58c127b1 Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)
* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging

Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core
schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic
migrations, singleton registry. memory.py reduced to thin facade. Session.py
open_db() calls replaced with generic KV methods. [database] config section
with env var support.

Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with
postgres extras and migration entrypoint, Helm chart with bitnami subcharts,
Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB.

39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated.

* Address PR #20 review feedback (16 items)

- Backends only call create_all() when Alembic migrations are disabled
- Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL
  constructed via env expansion with secret reference instead of ConfigMap
- Migration errors fail fast for PostgreSQL (only non-fatal for SQLite)
- save_memory/delete_memory wrapped in exception handling like other facade fns
- pool_size passed through from config/env to init_storage() in cli + server
- Terraform: DB URL moved to Secrets Manager, auth enabled flag set,
  optional TLS listeners with certificate_arn, Redis transit encryption on
- Docker entrypoint no longer suppresses migration output
- Diagram fixes: removed StaticPool claim, removed non-existent migration ref
- compose.yaml/README: clarified production profile requires DB env vars
2026-03-03 22:57:34 -08:00
Patrick Buckley 5ee539c983 Add Python and TypeScript client SDKs for server and console APIs (#19)
* Add Python and TypeScript client SDKs for server and console APIs

Python SDK (turnstone/sdk/) with sync + async clients for both server
and console APIs. Returns Pydantic models directly, streams SSE events
as typed dataclasses. 27 event types with registry-based deserialization.
High-level send_and_wait() for request-response patterns.

TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses
fetch + ReadableStream for SSE parsing. Discriminated union event types
with type guards. Same API surface as Python SDK.

63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at
docs/sdk.md with SDK architecture diagram.

* Address PR #19 review feedback + fix lint

- Fix consume_task leak in send_and_wait when send() raises (try/finally)
- Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout
- Add signal param to TS streamSSE for cancellation support
- Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF
- Fix generate-types.py sys.path (parents[3] not parents[2])
- Document token ignored when httpx_client provided
- Document TS timeout units as milliseconds
- Fix stale docstring in test_sdk_sse.py
- Fix import sorting (ruff I001)
2026-03-03 21:22:24 -08:00
Patrick Buckley 62a4ceac96 Dev/api versioning openapi (#18)
* Add API versioning under /v1/ prefix with OpenAPI 3.1 spec

All API endpoints move to /v1/api/* (clean break, no unversioned
aliases). Non-API routes (/, /health, /metrics, /static, /shared,
/node proxy) stay unversioned.

New turnstone/api/ package:
- Pydantic v2 models for all request/response schemas (server +
  console) used for OpenAPI spec generation
- Programmatic OpenAPI 3.1 spec builder with EndpointSpec catalog
- /openapi.json serves machine-readable spec, /docs serves Swagger UI

Route changes:
- Both servers use Mount("/v1", routes=[...API routes...])
- Auth middleware strips /v1/ prefix before path classification
  (PUBLIC_PATHS/WRITE_PATHS stay unversioned internally)
- Console proxy handles /node/{id}/v1/api/ upstream forwarding
- Bridge and CLI HTTP clients updated to /v1/api/ paths
- /openapi.json and /docs added to PUBLIC_PATHS and rate limiter
  EXEMPT_PATHS

Security fix from review: required_role() now correctly handles
/node/{id}/v1/api/{path} proxy routes (previously the v1 segment
caused write-path detection to fail, allowing read-only token
escalation).

42 new tests (830 total). All frontend JS, docs, and diagrams updated.

* Fix mypy type errors in turnstone/api/ package

- Add generic type params to dict fields in console_schemas.py
- Add return type annotations to docs.py handler factories
- Move type-only imports (BaseModel, Callable, Awaitable) into
  TYPE_CHECKING blocks to satisfy TC002/TC003 ruff rules

* Address PR #18 review feedback + fix mypy errors

Review fixes:
- Add pydantic>=2.0 as explicit dependency in pyproject.toml
  (was only transitively available via openai/mcp)
- Auto-detect path parameters from {param} segments in OpenAPI
  spec builder (fixes missing required path params)
- Use startswith() with concrete prefix for proxy version
  detection instead of fragile substring check
- Make Swagger UI base URL configurable via swagger_ui_base_url
  parameter for air-gapped deployments

Mypy fixes:
- Add generic type params to dict fields in console_schemas
- Add return type annotations to docs.py handler factories
- Move type-only imports into TYPE_CHECKING blocks
2026-03-03 20:28:49 -08:00
Patrick Buckley 29c00c0cdf Extract shared frontend design system into turnstone/shared_static/ (#17)
* Extract shared frontend design system into turnstone/shared_static/

The server UI and console UI had ~60% CSS overlap and significant JS
duplication. Extract shared assets into a new turnstone/shared_static/
package mounted at /shared/ in both servers:

- base.css: design tokens, reset, typography, login/toast/kb overlays,
  dashboard table, state dots, health bar, scrollbar, reduced motion
- auth.js: authFetch, login overlay with focus trap, logout (hooks for
  page-specific post-login/logout callbacks)
- theme.js: dark/light toggle with system preference detection
- toast.js: notification queue with configurable timeout
- utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount
- kb.js: keyboard shortcuts overlay with configurable content, focus
  management, and focus restore on dismiss

Console proxy updated: JS shim injection moved from proxy_static (app.js
prepend) to proxy_index (inline <script> in HTML) so it runs before any
external scripts. New /shared/ path rewriting and proxy_shared_static
route added. ~1540 lines removed from page-specific files, 775 lines in
shared package. 13 new tests (788 total).

* Fix /shared/ auth and remove __init__.py from shared_static

Address PR #17 review feedback:

1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS
   loads before authentication (required for login overlay to render)

2. Remove turnstone/shared_static/__init__.py to prevent exposing
   Python package internals (__init__.py, __pycache__) via the
   StaticFiles mount. Not needed for packaging since pyproject.toml
   uses explicit glob includes.

3 new auth tests for /shared/ public path access.
2026-03-03 20:11:49 -08:00
Patrick Buckley b6e0f0fcca Add node version tracking and drift detection to console dashboard (#16)
* Add node version tracking and drift detection to console dashboard

Surface the version field from each node's /health endpoint in the
console dashboard. Collector extracts version into get_overview()
(version_drift + versions fields), promotes it to top-level in
get_nodes(), and adds get_version_info() for per-node detail. Console
/health endpoint includes drift fields.

Frontend adds a VER column to the 7-column node table grid, shows
per-node version strings, tracks versions per group with "mixed" +
yellow drift badge when nodes disagree, and displays a DRIFT warning
or single version in the status bar. Column hidden on mobile (<700px).
ARIA labels include version info for accessibility.

10 new tests (745 total). Docs and diagram updated.

* Fix drift tooltip text: show 'Versions detected' not 'Nodes running'
2026-03-03 18:57:30 -08:00
Patrick Buckley 206e37e73e Fix circuit breaker, rate limiter, and Anthropic web search correctne… (#15)
* Fix circuit breaker, rate limiter, and Anthropic web search correctness (#15)

Three tech debt items addressing correctness and security gaps:

Circuit breaker HALF_OPEN single-request permit:
- Rename should_allow_request property to acquire_request_permit() method
  to make the side-effecting, non-idempotent nature explicit
- Add _half_open_permit flag: exactly one probe request in HALF_OPEN,
  subsequent callers blocked until probe completes
- Explicitly reset permit on all state transitions (record_success,
  record_failure) for clean state machine invariants
- Session uses BaseException catch to ensure record_failure always fires,
  preventing permanent circuit deadlock on probe crash

Rate limiter X-Forwarded-For support:
- Add resolve_client_ip() with rightmost-untrusted XFF parsing
- Configurable trusted_proxies via --ratelimit-trusted-proxies CLI flag
  and [ratelimit] trusted_proxies config (comma-separated CIDRs)
- IPv4-mapped IPv6 normalization (::ffff:x.x.x.x → IPv4) for dual-stack
- Clientless requests (request.client is None) pass through instead of
  sharing a single "unknown" bucket
- Log warning for invalid CIDR entries in trusted_proxies config
- Show trusted proxies in startup log when enabled

Anthropic web search multi-turn encrypted content:
- Capture raw provider content blocks during streaming via _block_to_dict()
  using model_dump(exclude_none=True) to avoid Anthropic API rejection
- Accumulate thinking_delta into raw_blocks (was silently empty on replay)
- Store _provider_content on assistant messages, pass through verbatim in
  _convert_messages() so encrypted_content/encrypted_index survive turns
- Persist to SQLite via new provider_data column (auto-migrated)
- Add thinking/signature to _block_to_dict fallback attribute list

23 new tests (735 total), ruff + mypy clean.

* Fix Copilot PR #15 review issues: provider data, circuit breaker, IP normalization

- Persist assistant message when provider_data exists even if text
  content is empty — prevents losing Anthropic web search encrypted
  content needed for multi-turn replay (session.py)
- Re-raise KeyboardInterrupt/SystemExit immediately after recording
  failure instead of attempting fallback models (session.py)
- Consume HALF_OPEN permit for the transition caller — prevents two
  concurrent probe requests when only one should be allowed
  (healthcheck.py)
- Normalize IPv4-mapped IPv6 addresses consistently in
  resolve_client_ip() — prevents duplicate rate-limit buckets for
  ::ffff:x.x.x.x vs x.x.x.x (ratelimit.py)
2026-03-03 18:28:39 -08:00
Patrick Buckley 6c5441435b Add console workstream creation + server reverse proxy (#14)
* Add console workstream creation + server reverse proxy (#14)

Enable the console dashboard to create workstreams and proxy server UIs,
so users only need network access to the console port.

Workstream creation via MQ:
- POST /api/cluster/workstreams/new with three targeting modes:
  specific node (directed queue), auto (best node by capacity),
  or general pool (shared queue, any bridge picks up)
- Console pushes CreateWorkstreamMessage to Redis; bridge handles
  the rest (server creation, ownership registration, SSE events)

Reverse proxy for server UIs:
- /node/{node_id}/ serves the server's HTML with static path rewriting
  and a console-return banner injected after <body>
- JS proxy shim prepended to app.js overrides fetch() and EventSource()
  to route root-relative URLs through /node/{id}/api/...
- SSE streams proxied via httpx.AsyncClient(timeout=None) with per-
  connection clients for long-lived streams
- GET/POST API requests forwarded with body and auth token

Security:
- Proxy write paths checked against WRITE_PATHS to prevent read-token
  escalation (read tokens cannot POST /api/send through proxy)
- html.escape() on node_id in banner HTML to prevent XSS
- String length limits on name/model inputs

Frontend:
- "+ new" button in header opens creation modal with node dropdown
  (Auto / General pool / specific nodes with capacity display)
- Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling
- Workstream rows and node links deep-link via proxy paths
- Custom select arrow, Instrument Panel modal styling

Documentation:
- docs/console.md rewritten with proxy and creation API docs
- docs/architecture.md console section updated
- PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered
- README.md updated

28 new tests (741 total), ruff + mypy clean.

* Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness

- Normalize trailing slashes in required_role() to prevent write-role
  bypass via /api/send/ or /node/{id}/api/send/ (auth.py)
- Validate node_id format in proxy handlers (alphanumeric, dot, dash,
  underscore only) to prevent injection vectors
- Use json.dumps() for JS proxy shim prefix to prevent script injection
- URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static)
- Check upstream status in _proxy_sse() — emit error event on non-200
  instead of keeping a dead SSE connection open
- Check upstream status in proxy_index() — propagate non-2xx errors
- Forward query string in _proxy_post() (consistency with _proxy_get)
- Handle JSON null values in create_workstream() — treat null as empty,
  reject non-string types with 400
- Fix docs/diagram LPUSH → RPUSH to match actual broker implementation
2026-03-03 18:25:18 -08:00
Patrick Buckley f02972c11d Add provider-native web search with Tavily fallback (#13)
* Add provider-native web search with Tavily fallback

Replace client-side Tavily web search with provider-native implementations:

- Anthropic: inject web_search_20250305 server-side tool, handle
  server_tool_use / web_search_tool_result streaming blocks, emit
  info_delta for search status display
- OpenAI: inject web_search_options for gpt-5-search-api, format
  url_citation annotations as footnote sources
- Local/vLLM: preserve existing Tavily-based web_search tool as fallback

Add supports_web_search to ModelCapabilities and info_delta to StreamChunk.
Remove end-of-life GPT-4o model entries from capability tables.
Update docs, diagrams, and README. 88 provider tests (32 new).

* Fix Copilot PR #13 review: capture streaming url_citation annotations

Accumulate url_citation annotations during OpenAI streaming and emit
formatted citations as a final info_delta chunk after the stream ends.
Previously annotations were only captured in non-streaming mode, so
search model users in the interactive path never saw citation sources.
2026-03-03 17:12:49 -08:00
Patrick Buckley d28879208f Add multi-provider LLM adapter with model capability flags (#12)
* Add multi-provider LLM adapter with model capability flags

Introduce a provider abstraction layer between ChatSession and LLM SDK
clients, enabling native support for Anthropic alongside OpenAI-compatible
APIs. Each provider translates at the API boundary while the internal
message format remains OpenAI-like throughout session history and persistence.

- LLMProvider protocol with StreamChunk/CompletionResult normalized types
- OpenAIProvider: GPT-4o, GPT-5.x, O-series capability tables with
  conditional temperature, reasoning_effort, and token param handling
- AnthropicProvider: native streaming, message/tool format conversion,
  adaptive vs manual thinking modes, effort parameter for 4.6 models
- ModelCapabilities per-model flags: temperature support, token param name,
  thinking mode, effort levels, context window, max output
- Smart auto-detect: latest Opus for Anthropic, latest base GPT for OpenAI
- --provider CLI flag for both turnstone and turnstone-server
- anthropic SDK as optional dependency (pip install turnstone[anthropic])
- 56 new provider tests, 672 total passing
- Updated architecture docs and 4 PlantUML diagrams

* Fix Copilot PR #12 review: reasoning_effort gating, Anthropic thinking, provider factory

- Default reasoning_effort_values to () so unknown/local models don't
  receive unsupported top-level reasoning_effort param. Models that need
  it (GPT-5.x, search models) have explicit capability declarations.
- Fix Anthropic _reasoning_params: "none" and "" effort now return {}
  instead of enabling thinking with 4096 budget.
- Use create_provider("openai") singleton instead of OpenAIProvider()
  in ChatSession fallback for consistency with registry path.
- Add 8 parameter gating tests: unknown model no reasoning_effort,
  GPT-5 no temperature, GPT-5.1 conditional temperature, O-series
  no temperature, Anthropic none/empty/low effort.
2026-03-03 16:52:02 -08:00
Patrick Buckley a1f00092f5 Migrate HTTP servers from stdlib to Starlette/ASGI + uvicorn (#11)
* Migrate HTTP servers from stdlib to Starlette/ASGI + uvicorn

Replace Python stdlib http.server (ThreadedHTTPServer, BaseHTTPRequestHandler)
with Starlette ASGI applications served by uvicorn across all three HTTP
entry points. SSE endpoints use sse-starlette EventSourceResponse with
async generators that bridge sync queue.Queue via run_in_executor().
Bridge SSE parser replaced with httpx-sse EventSource.

- turnstone/server.py: Starlette app factory with create_app(), pure ASGI
  middleware (auth, rate limit, metrics, CORS), async route handlers,
  lifespan context manager for startup/shutdown. WebUI and ChatSession
  remain fully synchronous — worker threads unchanged.
- turnstone/console/server.py: Same pattern, simpler (no ChatSession).
  Path params replace manual string slicing for node detail route.
- turnstone/mq/bridge.py: _iter_sse_data() uses httpx_sse.EventSource
  instead of hand-rolled line parser.
- Tests: All ThreadedHTTPServer fixtures replaced with
  starlette.testclient.TestClient via create_app() factories.
- Docs: Updated architecture.md, api-reference.md, README.md, and
  PlantUML diagrams (03, 11) + regenerated PNGs.

* Fix Copilot PR #11 review: TestClient cleanup, JSON error handling, SSE timeout

- Close TestClient in teardown for TestConsoleAuth and TestConsoleLogin to avoid lifespan/resource leaks
- Close TestClient via yield/finally in TestConsoleHTTPEndpoints fixture
- Add _read_json() helper for safe JSON body parsing (returns {} on invalid JSON instead of 500, matching old stdlib handler behavior)
- Apply same try/except pattern to console auth_login endpoint
- Increase SSE queue.get timeout from 1s to 5s to align with sse-starlette ping interval, reducing executor task churn
2026-03-02 23:34:22 -08:00
Patrick Buckley bb3b735d69 Refactor detect_model into shared function, auto-detect context window
Move detect_model() to turnstone.core.model_registry as a single
implementation replacing duplicates in cli.py, server.py, and eval.py.
Auto-detects context window from backend metadata (meta.n_ctx_train)
when available, falling back to the 131072 default otherwise.

Also replace vLLM-specific references in help text and comments with
generic "OpenAI-compatible API" / "model server" language.
2026-03-02 23:28:01 -08:00
Patrick Buckley 87ffad18c2 Use system role instead of developer for broader model compatibility
The developer message role is not supported by all chat templates
(e.g. Qwen). Switch to the standard system role which is universally
supported by OpenAI-compatible APIs.
2026-03-02 23:22:30 -08:00
393 changed files with 116832 additions and 5975 deletions
+9
View File
@@ -12,3 +12,12 @@ venv/
.mypy_cache/
.ruff_cache/
.hypothesis/
deploy/
docs/
tests/
sdk/
*.md
!README.md
!LICENSE
.coverage
.swp
+21 -77
View File
@@ -1,85 +1,29 @@
# =============================================================================
# Turnstone Docker Compose — Environment Configuration
# Copy to .env and fill in your values: cp .env.example .env
# Turnstone Environment Variables
# Copy to .env and adjust values for your deployment
# =============================================================================
# ---------------------------------------------------------------------------
# LLM Backend
# ---------------------------------------------------------------------------
# OpenAI-compatible API URL (vLLM, llama.cpp, OpenAI, etc.)
# -- LLM Backend --------------------------------------------------------------
LLM_BASE_URL=http://host.docker.internal:8000/v1
OPENAI_API_KEY=sk-...
# ANTHROPIC_API_KEY=sk-ant-... # Set instead for Anthropic provider
# TAVILY_API_KEY=tvly-... # For web search fallback (local models only)
# API key for the LLM backend ("dummy" for local servers without auth)
OPENAI_API_KEY=dummy
# -- Database (production profile) --------------------------------------------
# DB_BACKEND=postgresql
# POSTGRES_USER=turnstone
# POSTGRES_PASSWORD=changeme
# DATABASE_URL=postgresql+psycopg://turnstone:changeme@postgres:5432/turnstone
# Tavily API key for web_search tool (optional)
TAVILY_API_KEY=
# -- Redis ---------------------------------------------------------------------
# REDIS_PASSWORD=
# REDIS_PORT=6379
# ---------------------------------------------------------------------------
# Redis
# ---------------------------------------------------------------------------
# Redis password (leave empty for no authentication)
REDIS_PASSWORD=
# -- Authentication ------------------------------------------------------------
# TURNSTONE_AUTH_ENABLED=true
# TURNSTONE_AUTH_TOKEN=your-secret-token
# TURNSTONE_JWT_SECRET=python -c "import secrets; print(secrets.token_hex(32))"
# Host port for Redis
REDIS_PORT=6379
# ---------------------------------------------------------------------------
# Server
# ---------------------------------------------------------------------------
# Host port for the turnstone web UI
SERVER_PORT=8080
# Set to any non-empty value to auto-approve all tool calls
SKIP_PERMISSIONS=
# ---------------------------------------------------------------------------
# Bridge
# ---------------------------------------------------------------------------
# Heartbeat TTL in seconds
HEARTBEAT_TTL=60
# Seconds to wait for external approval responses
APPROVAL_TIMEOUT=300
# ---------------------------------------------------------------------------
# Console (Cluster Dashboard)
# ---------------------------------------------------------------------------
# Host port for the cluster dashboard
CONSOLE_PORT=8090
# Seconds between node polling cycles
CONSOLE_POLL_INTERVAL=10
# ---------------------------------------------------------------------------
# Auth (optional)
# ---------------------------------------------------------------------------
# Set to "1" to require Bearer token authentication
TURNSTONE_AUTH_ENABLED=
# Bearer token for server/bridge/console authentication
TURNSTONE_AUTH_TOKEN=
# ---------------------------------------------------------------------------
# Simulator (used with: docker compose --profile sim up)
# ---------------------------------------------------------------------------
# Number of simulated nodes
SIM_NODES=100
# Scenario: steady, burst, node_failure, directed, lifecycle
SIM_SCENARIO=steady
# Scenario duration in seconds
SIM_DURATION=60
# Messages per second (steady scenario)
SIM_MPS=5.0
# Log level
SIM_LOG_LEVEL=INFO
# Random seed for reproducibility (leave empty for random)
SIM_SEED=
# Path to write JSON metrics report (leave empty to skip)
SIM_METRICS_FILE=
# -- Ports ---------------------------------------------------------------------
# SERVER_PORT=8080
# CONSOLE_PORT=8090
+208
View File
@@ -0,0 +1,208 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:recommended",
"helpers:pinGitHubActionDigests",
":separateMajorReleases"
],
"labels": [
"dependencies"
],
"prConcurrentLimit": 5,
"prHourlyLimit": 2,
"schedule": [
"before 9am on Monday"
],
"timezone": "America/New_York",
"lockFileMaintenance": {
"enabled": true,
"schedule": [
"before 9am on Monday"
]
},
"customManagers": [
{
"customType": "regex",
"description": "Track vendored KaTeX version",
"managerFilePatterns": [
"/pyproject\\.toml$/"
],
"matchStrings": [
"katex-(?<currentValue>[\\d.]+)/"
],
"depNameTemplate": "katex",
"datasourceTemplate": "npm"
},
{
"customType": "regex",
"description": "Track vendored Highlight.js version",
"managerFilePatterns": [
"/pyproject\\.toml$/"
],
"matchStrings": [
"hljs-(?<currentValue>[\\d.]+)/"
],
"depNameTemplate": "highlight.js",
"datasourceTemplate": "npm"
},
{
"customType": "regex",
"description": "Track vendored Mermaid version",
"managerFilePatterns": [
"/pyproject\\.toml$/"
],
"matchStrings": [
"mermaid-(?<currentValue>[\\d.]+)/"
],
"depNameTemplate": "mermaid",
"datasourceTemplate": "npm"
}
],
"packageRules": [
{
"description": "LLM SDKs — always review manually",
"groupName": "LLM SDKs",
"matchPackageNames": [
"openai",
"anthropic",
"mcp"
],
"schedule": [
"before 9am on Monday"
],
"automerge": false
},
{
"description": "Web framework stack",
"groupName": "Web Framework",
"matchPackageNames": [
"starlette",
"uvicorn",
"sse-starlette",
"httpx",
"httpx-sse",
"pydantic"
],
"schedule": [
"before 9am on Wednesday"
],
"automerge": true,
"matchUpdateTypes": [
"patch"
]
},
{
"description": "Database layer",
"groupName": "Database",
"matchPackageNames": [
"sqlalchemy",
"alembic",
"psycopg"
],
"schedule": [
"before 9am on Wednesday"
],
"automerge": true,
"matchUpdateTypes": [
"patch"
]
},
{
"description": "Security-critical — always review manually",
"groupName": "Security",
"matchPackageNames": [
"PyJWT",
"pyjwt",
"bcrypt"
],
"automerge": false
},
{
"description": "Infrastructure dependencies",
"groupName": "Infrastructure",
"matchPackageNames": [
"structlog",
"redis",
"croniter",
"discord.py"
],
"schedule": [
"before 9am on the first day of the month"
],
"automerge": true,
"matchUpdateTypes": [
"patch"
]
},
{
"description": "Vendored JS — requires manual file download after merge",
"groupName": "Vendored JS",
"matchPackageNames": [
"katex",
"highlight.js",
"mermaid"
],
"schedule": [
"before 9am on the first day of the month"
],
"automerge": false,
"prBodyNotes": [
"This PR updates version references only.",
"After merging, run `scripts/update-vendored-js.sh <lib> <version>` to download the actual files."
]
},
{
"description": "Dev/test tooling",
"groupName": "Tooling",
"matchPackageNames": [
"ruff",
"mypy",
"types-redis",
"pytest",
"pytest-cov",
"pre-commit"
],
"schedule": [
"before 9am on the first day of the month"
],
"automerge": true,
"matchUpdateTypes": [
"patch"
]
},
{
"description": "Docker base images",
"groupName": "Docker Images",
"matchManagers": [
"dockerfile",
"docker-compose"
],
"schedule": [
"before 9am on the first day of the month"
],
"automerge": false
},
{
"description": "TypeScript SDK dev dependencies",
"groupName": "TypeScript SDK",
"matchFileNames": [
"sdk/typescript/**"
],
"schedule": [
"before 9am on the first day of the month"
],
"automerge": true,
"matchUpdateTypes": [
"patch"
]
},
{
"description": "GitHub Actions — group all action updates",
"groupName": "GitHub Actions",
"matchManagers": [
"github-actions"
],
"automerge": false
}
]
}
+49 -12
View File
@@ -10,21 +10,21 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.13"
- run: pip install ruff
- run: ruff check turnstone/ tests/
- run: ruff format --check turnstone/ tests/
python-version: "3.14"
- run: pip install pre-commit
# mypy runs separately in typecheck job with full project deps
- run: SKIP=mypy pre-commit run --all-files
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.13"
python-version: "3.14"
- run: pip install mypy types-redis
- run: pip install -e ".[mq]"
- run: mypy turnstone/
@@ -35,14 +35,51 @@ jobs:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
- run: pip install -e ".[test,mq]"
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -q
- uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
if: always()
with:
name: coverage-${{ matrix.python-version }}
path: coverage.xml
lock-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7
with:
uv-version: "0.9.18"
- run: uv lock --check
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7
with:
uv-version: "0.9.18"
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: uv sync --frozen --all-extras
- run: uv pip install pip-audit
- name: Security audit (dependencies)
run: uv export --no-emit-project --frozen | uv run pip-audit --strict --desc -r /dev/stdin
security-ts:
runs-on: ubuntu-latest
defaults:
run:
working-directory: sdk/typescript
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6
with:
node-version: "24"
- run: npm ci
- run: npm audit --audit-level=moderate
+22
View File
@@ -0,0 +1,22 @@
name: Docker Security Scan
on:
push:
branches: [main]
schedule:
- cron: "0 6 * * 1" # Weekly Monday 06:00 UTC
permissions:
contents: read
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- run: docker build -t turnstone:scan .
- uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # 0.35.0
with:
image-ref: "turnstone:scan"
severity: "HIGH,CRITICAL"
exit-code: "1"
+11 -3
View File
@@ -5,6 +5,7 @@ on:
tags: ["v*"]
permissions:
contents: write
id-token: write
jobs:
@@ -12,10 +13,17 @@ jobs:
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.13"
python-version: "3.14"
- run: pip install build
- run: python -m build
- uses: pypa/gh-action-pypi-publish@release/v1
- name: Create GitHub Release
uses: softprops/action-gh-release@153bb8e04406b158c6c84fc1615b65b24149a1fe # v2
with:
generate_release_notes: true
draft: false
prerelease: ${{ contains(github.ref, '-') }}
+4
View File
@@ -17,3 +17,7 @@ venv/
.plan.md
.plan-*.md
.hypothesis/
PROGRESS.md
.coverage
tools/skill_audit_analysis/data/
tools/skill_audit_analysis/output/
+2 -2
View File
@@ -1,13 +1,13 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.9.10
rev: v0.15.6
hooks:
- id: ruff
args: [--fix]
- id: ruff-format
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.14.1
rev: v1.19.1
hooks:
- id: mypy
additional_dependencies: [types-redis>=4.6, redis>=7.2]
+30 -23
View File
@@ -1,46 +1,53 @@
# =============================================================================
# Turnstone — multi-stage Docker build
# Turnstone — Docker build with uv for reproducible, locked installs
# Single image for all services: server, bridge, console, sim, eval
# =============================================================================
# ----------------------------------------------------------------------------
# Stage 1: Builder — build the wheel
# ----------------------------------------------------------------------------
FROM python:3.13-slim AS builder
WORKDIR /build
RUN pip install --no-cache-dir hatchling
COPY pyproject.toml README.md LICENSE ./
COPY turnstone/ turnstone/
RUN pip wheel --no-deps --wheel-dir /build/wheels .
# ----------------------------------------------------------------------------
# Stage 2: Runtime — slim image with the installed package
# ----------------------------------------------------------------------------
FROM python:3.13-slim
FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.10.10 /uv /usr/local/bin/uv
# System dependencies for psycopg (PostgreSQL client library)
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends libpq5 \
&& rm -rf /var/lib/apt/lists/*
# Non-root user
RUN useradd --create-home --shell /bin/bash turnstone
# Install the wheel with all optional extras (redis for mq/console/sim)
COPY --from=builder /build/wheels/*.whl /tmp/wheels/
RUN pip install --no-cache-dir "$(ls /tmp/wheels/*.whl)[mq,console,sim]" \
&& rm -rf /tmp/wheels
WORKDIR /app
# Compile bytecode for faster startup
ENV UV_COMPILE_BYTECODE=1
# Install dependencies first (cached layer — only re-runs when deps change)
COPY pyproject.toml uv.lock README.md LICENSE ./
RUN uv sync --frozen --no-install-project --no-dev \
--extra mq --extra console --extra sim --extra postgres --extra discord --extra anthropic
# Install the project itself
COPY turnstone/ turnstone/
RUN uv sync --frozen --no-dev \
--extra mq --extra console --extra sim --extra postgres --extra discord --extra anthropic
# Add venv to PATH so entry points are found
ENV PATH="/app/.venv/bin:$PATH"
# Health check script (stdlib only, no pip deps needed)
COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
# Entrypoint script — runs migrations before starting
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
# Data directory — SQLite DB is created in CWD
WORKDIR /data
RUN chown turnstone:turnstone /data
USER turnstone
ENTRYPOINT ["entrypoint.sh"]
# Default command (overridden per service in compose.yaml)
CMD ["turnstone-server", "--host", "0.0.0.0", "--port", "8080"]
+92
View File
@@ -0,0 +1,92 @@
# Bootstrap Wizard
Interactive, AI-guided setup for Turnstone deployments. Instead of manually
editing `.env` files and reading deployment docs, the wizard walks you through
every decision conversationally and generates all the config files for you.
## Quick Start
```bash
turnstone-bootstrap
```
That's it — no flags, no arguments. The wizard prompts for everything.
## How It Works
1. **Pick a model** — Choose OpenAI, Anthropic, or a local/vLLM endpoint to
power the wizard. Local endpoints auto-detect available models.
2. **Answer questions** — The AI walks you through deployment mode, LLM
provider, database, authentication, ports, and optional features.
3. **Review generated files** — Each file is previewed before writing. You
confirm or reject every write.
4. **Start the stack** — The wizard prints the exact `docker compose` command
and a `setup.sh` script to create your first admin user, roles, and policies.
## What Gets Generated
| File | Purpose |
|------|---------|
| `.env` | All environment variables for `compose.yaml` |
| `setup.sh` | Post-start script: creates admin user, roles, tool policies, prompt templates via the API |
| `docker-compose.override.yaml` | Only if customizations beyond env vars are needed |
## Requirements
- **Python 3.11+** with turnstone installed (`pip install turnstone`)
- **An LLM API key** — for the wizard itself (OpenAI, Anthropic, or a local
model). This can differ from the LLM your deployment will use.
- **Docker & Docker Compose** — needed to run the stack. The wizard detects
whether Docker is installed and gives platform-specific install instructions
if it's missing. You can still generate config files without Docker.
## Deployment Modes
The wizard supports two deployment modes:
- **Single-node production** (`docker compose --profile production up`) —
1 server + bridge + console + PostgreSQL + Redis. Good for most use cases.
- **Multi-node cluster** (`docker compose --profile cluster up`) —
10-node server/bridge fleet + PostgreSQL + Redis. For high-throughput or
HA deployments.
## Example Session
```
$ turnstone-bootstrap
Turnstone Bootstrap Wizard v0.5.4
────────────────────────────────────────────────
Which provider for this wizard?
[1] OpenAI
[2] Anthropic
[3] OpenAI-compatible (local/vLLM)
> 3
Base URL [http://localhost:8000/v1]:
API key (press Enter for 'none'):
Querying http://localhost:8000/v1 for available models...
Found model: Qwen/Qwen3-32B
Connected to Qwen/Qwen3-32B. Handing off to AI assistant...
> (AI walks you through the rest interactively)
```
## Tips
- **Re-run safely** — running the wizard again detects your existing `.env`
and offers to update it rather than overwriting.
- **Duplicate writes are skipped** — if the LLM tries to write the same file
twice with identical content, it's silently ignored.
- **Type `quit` to exit** at any time during the conversation.
- **Ctrl+C** is handled gracefully — press once to interrupt, twice to exit.
## See Also
- [Docker Deployment](docker.md) — manual compose setup and profiles
- [Security](security.md) — auth architecture and token types
- [Governance](governance.md) — roles, policies, and templates
+122 -83
View File
@@ -16,16 +16,16 @@ Turnstone gives LLMs tools — shell, files, search, web, planning — and orche
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
- **Multi-node clusters** — generic work load-balances across nodes, directed work routes to a specific server
- **Cluster dashboard** — real-time view of all nodes, workstreams, and resource utilization
- **Cluster dashboard** — real-time view of all nodes and workstreams, reverse proxy for server UIs
- **Intent validation** — an LLM judge evaluates every tool call before approval, presenting risk assessments and evidence-based recommendations so users can make informed decisions instead of blindly approving raw tool calls
- **Governance & compliance** — RBAC, OIDC SSO (Okta, Azure AD, Google, Keycloak), tool policies, skills (reusable behavioral profiles with security scanning), usage tracking, and append-only audit logs
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
```
External System → Message Queue → Bridge (per node) → Turnstone Server → LLM + Tools
Pub/Sub → Progress Events → External System
turnstone-console → Cluster Dashboard (browser)
```
Works with any OpenAI-compatible API (vLLM, llama.cpp, NVIDIA NIM) or Anthropic's native Messages API. Supports [MCP](https://modelcontextprotocol.io/) for external tool servers with native deferred tool loading on Anthropic and OpenAI APIs (BM25 fallback for local models).
<p align="center">
<img src="docs/diagrams/architecture-overview.svg" alt="Turnstone system architecture — data flow from clients through gateways, Redis MQ, cluster nodes, to LLM providers" width="960"/>
</p>
## Quickstart
@@ -72,13 +72,20 @@ pip install turnstone[console]
turnstone-console --redis-host localhost --port 8090
```
Then open `http://localhost:8090` for the cluster-wide dashboard.
Then open `http://localhost:8090` for the cluster-wide dashboard. Create workstreams from the console and interact with any node's server UI through the built-in reverse proxy — no direct server port access required.
### Docker
```bash
cp .env.example .env # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose up # starts redis + server + bridge + console
docker compose up # starts redis + server + bridge + console (SQLite)
```
For production with PostgreSQL:
```bash
# Requires POSTGRES_PASSWORD and DB_BACKEND=postgresql in .env (or exported)
docker compose --profile production up # adds PostgreSQL, uses it as database
```
Console dashboard at http://localhost:8090. See [docs/docker.md](docs/docker.md) for configuration, scaling, and profiles.
@@ -100,62 +107,9 @@ turnstone-sim --nodes 100 --scenario steady --duration 60 --mps 10
See [docs/simulator.md](docs/simulator.md) for scenarios, CLI reference, and metrics.
All frontends connect to any OpenAI-compatible API (vLLM, NVIDIA NIM/NGC, llama.cpp, OpenAI, etc.) and auto-detect the model.
## Architecture
```
turnstone/
├── core/ # UI-agnostic engine
│ ├── session.py # ChatSession — multi-turn loop, tool dispatch, agents
│ ├── tools.py # Tool definitions (auto-loaded from JSON)
│ ├── workstream.py # WorkstreamManager — parallel independent sessions
│ ├── mcp_client.py # MCP client manager (external tool servers)
│ ├── model_registry.py # ModelRegistry — named models, fallback routing, per-workstream selection
│ ├── config.py # Unified TOML config (~/.config/turnstone/config.toml)
│ ├── memory.py # SQLite persistence (memories, conversations, FTS5)
│ ├── metrics.py # Prometheus-compatible metrics collector
│ ├── healthcheck.py # Backend health monitor + circuit breaker
│ ├── ratelimit.py # Per-IP token-bucket rate limiter
│ ├── edit.py # File editing (fuzzy match, indentation)
│ ├── safety.py # Path validation, sandbox checks
│ ├── sandbox.py # Command sandboxing
│ └── web.py # Web fetch/search helpers
├── mq/ # Message queue integration
│ ├── protocol.py # Typed message dataclasses (JSON serialization)
│ ├── broker.py # Abstract MessageBroker + RedisBroker
│ ├── bridge.py # Bridge service (queue ↔ HTTP API, multi-node routing)
│ └── client.py # TurnstoneClient — Python API for external systems
├── console/ # Cluster dashboard
│ ├── collector.py # ClusterCollector — aggregates all nodes via Redis + HTTP
│ ├── server.py # Dashboard HTTP server + SSE
│ └── static/ # Cluster dashboard web UI
├── tools/ # Tool schemas (one JSON file per tool)
├── ui/ # Frontend assets and terminal rendering
│ └── static/ # Web UI (HTML, CSS, JS)
├── sim/ # Cluster simulator
│ ├── cluster.py # SimCluster — orchestrates N nodes + dispatchers
│ ├── node.py # SimNode + SimWorkstream — protocol-compatible node
│ ├── engine.py # LLM + tool execution simulation
│ ├── scenario.py # 5 workload scenarios (steady, burst, node_failure, …)
│ ├── metrics.py # Latency, throughput, utilization collection
│ └── cli.py # CLI entry point (turnstone-sim)
├── cli.py # Terminal frontend (+ /cluster commands for console)
├── server.py # Web frontend (HTTP + SSE)
└── eval.py # Evaluation and prompt optimization harness
docs/
├── architecture.md # System architecture and threading model
├── api-reference.md # Web server API and SSE event reference
├── console.md # Cluster dashboard service (turnstone-console)
├── docker.md # Docker Compose deployment and configuration
├── simulator.md # Cluster simulator usage and scenarios
├── tools.md # Tool schemas, execution pipeline, approval flow
├── eval.md # Evaluation harness internals
└── diagrams/ # UML architecture diagrams (PlantUML sources + PNGs)
└── png/ # Pre-rendered diagram images
```
### Architecture Diagrams
### Diagrams
Detailed UML diagrams are available in [`docs/diagrams/`](docs/diagrams/):
@@ -163,8 +117,8 @@ Detailed UML diagrams are available in [`docs/diagrams/`](docs/diagrams/):
|---------|-------------|
| [System Context](docs/diagrams/png/01-system-context.png) | Top-level components and external dependencies |
| [Package Structure](docs/diagrams/png/02-package-structure.png) | Python modules and dependency graph |
| [Core Engine Classes](docs/diagrams/png/03-core-engine-classes.png) | SessionUI protocol, ChatSession, WorkstreamManager |
| [Conversation Turn](docs/diagrams/png/04-conversation-turn.png) | Full message lifecycle through the engine |
| [Core Engine Classes](docs/diagrams/png/03-core-engine-classes.png) | SessionUI protocol, ChatSession, LLMProvider, WorkstreamManager |
| [Conversation Turn](docs/diagrams/png/04-conversation-turn.png) | Full message lifecycle through the engine (provider-agnostic) |
| [Tool Pipeline](docs/diagrams/png/05-tool-pipeline.png) | Three-phase prepare/approve/execute |
| [MQ Protocol](docs/diagrams/png/06-mq-protocol.png) | 9 inbound + 19 outbound message types |
| [Message Routing](docs/diagrams/png/07-message-routing.png) | Multi-node routing scenarios |
@@ -173,6 +127,56 @@ Detailed UML diagrams are available in [`docs/diagrams/`](docs/diagrams/):
| [Simulator](docs/diagrams/png/10-simulator-architecture.png) | SimCluster, dispatchers, scenarios |
| [Console Data Flow](docs/diagrams/png/11-console-data-flow.png) | Dashboard data collection threads |
| [Deployment](docs/diagrams/png/12-deployment.png) | Docker Compose service topology |
| [SDK Architecture](docs/diagrams/png/13-sdk-architecture.png) | Python + TypeScript client libraries |
| [Storage Architecture](docs/diagrams/png/14-storage-architecture.png) | Pluggable database backends (SQLite + PostgreSQL) |
| [Auth Architecture](docs/diagrams/png/15-auth-architecture.png) | JWT, scopes, token types, login flows |
| [Channel Architecture](docs/diagrams/png/16-channel-architecture.png) | Discord/Slack adapter protocol and routing |
| [Notify Flow](docs/diagrams/png/17-notify-flow.png) | Channel notification dispatch |
| [Watch Architecture](docs/diagrams/png/18-watch-architecture.png) | Periodic command polling daemon |
| [Governance Architecture](docs/diagrams/png/19-governance-architecture.png) | RBAC, policies, audit, usage enforcement flow |
| [WS Template Architecture](docs/diagrams/png/21-ws-template-architecture.png) | Workstream template application and lifecycle |
| [Judge Architecture](docs/diagrams/png/22-judge-architecture.png) | Intent validation two-tier evaluation pipeline |
| [OIDC Architecture](docs/diagrams/png/25-oidc-architecture.png) | OIDC SSO authorization code flow with PKCE |
### Governance
Turnstone includes a built-in governance layer for enterprise deployments — manage who can do what, which tools run unattended, and where every token goes.
- **RBAC** — 15 granular permissions, 3 built-in roles (admin / operator / viewer), custom roles, privilege escalation prevention
- **OIDC SSO** — single sign-on via any OpenID Connect provider (Okta, Azure AD, Google, Keycloak); Authorization Code Flow with PKCE, auto-provisioning, claim-based role mapping with demotion propagation; see [docs/oidc.md](docs/oidc.md)
- **Tool policies** — glob-pattern rules (`allow` / `deny` / `ask`) with priority ordering; automate approvals or lock down dangerous tools
- **Skills** — reusable behavioral profiles with system prompts, `{{variable}}` substitution, session config (model, temperature, token budget), install-time security scanning, version history, external discovery (skills.sh / GitHub), and runtime `load_skill` tool for model-driven skill activation
- **Usage tracking** — per-request token and tool metrics, aggregation by day / model / user, automatic 90-day pruning
- **Audit logging** — append-only event trail for all admin mutations, IP-aware, 365-day retention
All governance features are managed through the console admin panel (13 tabs) and the full REST API. Runtime settings (model, tools, rate limiting, health, judge, memory) are configurable via the admin Settings tab — no config file edits or restarts needed for most changes. See [docs/governance.md](docs/governance.md) for setup and [docs/settings.md](docs/settings.md) for the settings reference.
### Intent Validation (LLM Judge)
Every tool call that requires human approval is evaluated by an intent validation judge that provides a structured risk assessment alongside the approval prompt — so instead of "approve this bash command?", users see a verdict with risk level, confidence, recommendation, and reasoning.
The system uses a two-tier evaluation pipeline:
1. **Heuristic tier** (instant, free) — 36 pattern-based rules classify tool calls by severity. Catches destructive commands (`rm -rf /`, `DROP TABLE`), privilege escalation (`sudo`), credential access, supply chain risks, browser data export, cloud infrastructure mutations, and more. Results appear immediately.
2. **LLM judge tier** (async) — A full LLM evaluation runs in the background with access to `read_file` and `list_directory` for evidence gathering. The judge can inspect files that a write would overwrite, check directory contents before a delete, and cite specific evidence in its reasoning. Results update the UI progressively when ready.
The judge defaults to the same model as the session (self-consistency) but can be configured to use a separate model — useful when running a small local model for tasks but wanting a commercial model for safety evaluation.
```toml
[judge]
enabled = true # on by default
model = "" # empty = same as session model
provider = "" # empty = same as session provider
timeout = 60.0 # generous for local models
```
Verdicts are persisted for audit and exposed via Prometheus metrics (`turnstone_judge_verdicts_total`, `turnstone_judge_llm_latency_seconds`).
Skills are also scanned at install time — the scanner evaluates content, supply chain, vulnerability, and declared capability risk across four independent axes. Results populate `scan_status` (tier) and `scan_report` (structured JSON breakdown) on the skill record so administrators can assess risk before enabling a skill.
Tool execution results are evaluated by an output guard before entering the conversation — detecting prompt injection payloads in fetched content, credential leakage in command output, and encoded payloads. Detected credentials are automatically redacted.
See [docs/judge.md](docs/judge.md) for the full guide.
## Multi-node routing
@@ -197,29 +201,32 @@ Bridges BLPOP from their per-node queue (priority) then the shared queue. Direct
## Tools
14 built-in tools, 2 agent tools, plus external tools via MCP:
15 built-in tools, 2 agent tools, plus external tools via MCP:
| Tool | Description | Auto-approved |
|------|-------------|:---:|
| `bash` | Execute shell commands | |
| `read_file` | Read file contents | yes |
| `read_file` | Read file contents (text or images with vision models) | yes |
| `write_file` | Write/create files | |
| `edit_file` | Fuzzy-match file editing | |
| `search` | Search files by name/content | yes |
| `math` | Sandboxed Python evaluation | |
| `man` | Read man pages | yes |
| `web_fetch` | Fetch URL content | |
| `web_search` | Search via Tavily API | |
| `remember` | Save persistent facts | yes |
| `recall` | Search memories and history | yes |
| `forget` | Remove a memory | yes |
| `web_search` | Web search (provider-native or Tavily) | |
| `memory` | Structured persistent memory (save/search/delete/list) | yes |
| `recall` | Search conversation history | yes |
| `notify` | Send notifications to linked channels | yes |
| `watch` | Periodic command polling with conditions | |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
| `mcp__*` | External tools from MCP servers | |
When the total tool count exceeds a configurable threshold (default 20), MCP tools are automatically deferred using native `defer_loading` on Anthropic and OpenAI APIs, or a transparent client-side BM25 search for local models. The LLM discovers deferred tools on demand via a `tool_search` capability — no configuration needed beyond `--tool-search auto` (the default).
### MCP Tool Servers
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions.
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions. Tool lists stay fresh via push notifications (`tools.listChanged`), periodic polling for servers without push, and manual `/mcp refresh`.
Configure via `config.toml` or `--mcp-config`:
@@ -239,30 +246,39 @@ turnstone --mcp-config ~/.config/turnstone/mcp.json
turnstone-server --mcp-config ~/.config/turnstone/mcp.json
```
Use `/mcp` in the REPL to list connected tools. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
Use `/mcp` in the REPL to list connected tools, `/mcp refresh` to re-fetch tool lists from servers. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
### Multi-Model Support
### Multi-Model and Multi-Provider Support
Turnstone supports multiple model backends per server instance. Define named models in `config.toml` and select per-workstream or switch mid-session with `/model <alias>`.
Turnstone supports multiple model backends per server instance, including different LLM providers. `ChatSession` delegates all API communication to pluggable `LLMProvider` adapters — the internal message format stays OpenAI-like, and each provider translates at the API boundary. Define named models in `config.toml` and select per-workstream or switch mid-session with `/model <alias>`.
```toml
[models.local]
base_url = "http://localhost:8000/v1"
model = "qwen3-32b"
# provider defaults to "openai" (works with vLLM, llama.cpp, etc.)
[models.claude]
provider = "anthropic"
api_key = "sk-ant-..."
model = "claude-opus-4-6"
context_window = 200000
[models.openai]
base_url = "https://api.openai.com/v1"
api_key = "sk-..."
model = "gpt-4o"
context_window = 128000
model = "gpt-5"
context_window = 400000
[model]
default = "local" # which model to use by default
fallback = ["openai"] # try these if the primary is unreachable
agent_model = "local" # optional: cheaper model for plan/task sub-agents
fallback = ["claude", "openai"] # try these if the primary is unreachable
agent_model = "claude" # optional: separate model for plan/task sub-agents
```
Use `/model` to show available models, `/model openai` to switch. Workstreams created via the API accept an optional `model` parameter.
Supported providers: `"openai"` (default -- OpenAI, vLLM, llama.cpp, any OpenAI-compatible API) and `"anthropic"` (Anthropic Messages API, requires `pip install turnstone[anthropic]`).
Use `/model` to show available models, `/model claude` to switch. Workstreams created via the API accept an optional `model` parameter.
## Configuration
@@ -272,7 +288,7 @@ All entry points read `~/.config/turnstone/config.toml`. CLI flags override conf
[api]
base_url = "http://localhost:8000/v1"
api_key = ""
tavily_key = ""
tavily_key = "" # only needed for local/vLLM models without native search
[model]
name = "" # empty = auto-detect
@@ -285,6 +301,9 @@ agent_model = "" # model alias for plan/task sub-agents
[tools]
timeout = 30
skip_permissions = false
search = "auto" # "auto" (enable when >threshold tools), "on", "off"
search_threshold = 20 # min tools before tool search activates
search_max_results = 5 # max tools returned per search query
[server]
host = "0.0.0.0"
@@ -317,8 +336,22 @@ enabled = true
requests_per_second = 10.0
burst = 20
[database]
backend = "sqlite" # "sqlite" (default) or "postgresql"
path = ".turnstone.db" # SQLite file path (relative to working directory)
# url = "postgresql+psycopg://user:pass@host:5432/turnstone" # PostgreSQL
# pool_size = 5 # PostgreSQL connection pool size
[judge]
enabled = true # intent validation for tool approvals (--no-judge to disable)
model = "" # empty = same as session model (self-consistency)
provider = "" # empty = same as session provider
timeout = 60.0 # LLM judge timeout in seconds
confidence_threshold = 0.7
[mcp]
config_path = "" # path to MCP JSON config file (alternative to TOML sections)
refresh_interval = 14400 # periodic refresh for servers without push notifications (seconds, 0 to disable)
[mcp.servers.example] # one section per MCP server
command = "npx"
@@ -357,6 +390,9 @@ Idle workstreams are automatically cleaned up after 2 hours (configurable). In m
- `turnstone_backend_up` — LLM backend reachability (0/1)
- `turnstone_circuit_state` — circuit breaker state (0=closed, 1=open, 2=half_open)
- `turnstone_workstreams_evicted_total` — workstreams auto-evicted at capacity
- `turnstone_judge_verdicts_total{tier,risk_level}` — intent validation verdicts by tier and risk
- `turnstone_judge_llm_latency_seconds` — LLM judge evaluation latency histogram
- `turnstone_judge_enabled` — whether the intent validation judge is active (0/1)
Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
@@ -373,8 +409,11 @@ Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
## Requirements
- Python 3.11+
- An OpenAI-compatible API endpoint ([vLLM](https://github.com/vllm-project/vllm), [NVIDIA NIM](https://build.nvidia.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), etc.)
- An OpenAI-compatible API endpoint ([vLLM](https://github.com/vllm-project/vllm), [NVIDIA NIM](https://build.nvidia.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), etc.) or an Anthropic API key
- Redis (for message queue bridge — `pip install turnstone[mq]`)
- Anthropic provider (optional — `pip install turnstone[anthropic]`)
- PostgreSQL (optional, for production — `pip install turnstone[postgres]`)
- [Git LFS](https://git-lfs.com/) (for cloning — diagram PNGs are stored in LFS)
## License
+97
View File
@@ -0,0 +1,97 @@
Turnstone — Third-Party Notices
This file contains the licenses and notices for third-party software bundled
with Turnstone. Each bundled dependency retains its original license; the
Turnstone BUSL-1.1 license does not apply to these components.
================================================================================
KaTeX 0.16.38
https://katex.org/
https://github.com/KaTeX/KaTeX
The MIT License (MIT)
Copyright (c) 2013-2020 Khan Academy and other contributors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================================================
highlight.js 11.11.1
https://highlightjs.org/
https://github.com/highlightjs/highlight.js
BSD 3-Clause License
Copyright (c) 2006, Ivan Sagalaev.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================================================
Mermaid 11.13.0
https://mermaid.js.org/
https://github.com/mermaid-js/mermaid
The MIT License (MIT)
Copyright (c) 2014-2022 Knut Sveidqvist
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+356 -6
View File
@@ -2,10 +2,12 @@
# Turnstone Docker Compose Stack
#
# Usage:
# Full stack: docker compose up
# With simulator: docker compose --profile sim up
# Sim only: docker compose --profile sim up redis console sim
# Scale bridges: docker compose up --scale bridge=3
# Infra only: docker compose up
# Single node: docker compose --profile production up
# Production (PG): DB_BACKEND=postgresql docker compose --profile production up
# 10-node cluster: docker compose --profile cluster up
# Cluster + DDG: docker compose --profile ddgCluster up
# With simulator: docker compose --profile sim up
# =============================================================================
name: turnstone
@@ -17,13 +19,45 @@ networks:
volumes:
redis-data:
turnstone-data:
postgres-data:
services:
# -------------------------------------------------------------------
# PostgreSQL — production database (profile: production)
# -------------------------------------------------------------------
postgres:
image: pgautoupgrade/pgautoupgrade:18-alpine
profiles:
- production
- cluster
- ddgCluster
environment:
POSTGRES_DB: turnstone
POSTGRES_USER: ${POSTGRES_USER:-turnstone}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required for production profile}
PGDATA: /var/lib/postgresql/data
volumes:
- postgres-data:/var/lib/postgresql/data
networks:
- turnstone-net
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-turnstone}"]
interval: 5s
timeout: 3s
retries: 5
start_period: 30s
deploy:
resources:
limits:
memory: 512M
cpus: '1.0'
restart: unless-stopped
# -------------------------------------------------------------------
# Redis — message broker, pub/sub, node registry
# -------------------------------------------------------------------
redis:
image: redis:7.4-alpine
image: redis:8.6-alpine
command:
- sh
- -c
@@ -57,6 +91,8 @@ services:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
command:
- sh
- -c
@@ -66,11 +102,14 @@ services:
--port 8080
--base-url "$${LLM_BASE_URL}"
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
ports:
- "${SERVER_PORT:-8080}:8080"
volumes:
- turnstone-data:/data
- ./docker/mcp-ddg.json:/etc/turnstone/mcp-ddg.json:ro
environment:
- LLM_BASE_URL=${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
- OPENAI_API_KEY=${OPENAI_API_KEY:-dummy}
@@ -78,6 +117,12 @@ services:
- SKIP_PERMISSIONS=${SKIP_PERMISSIONS:-}
- TURNSTONE_AUTH_ENABLED=${TURNSTONE_AUTH_ENABLED:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- MODEL=${MODEL:-}
- MCP_CONFIG=${MCP_CONFIG:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
- TURNSTONE_NODE_ID=${TURNSTONE_NODE_ID:-}
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -85,6 +130,12 @@ services:
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
required: false
ddg-search:
condition: service_healthy
required: false
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
@@ -101,16 +152,19 @@ services:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
command:
- turnstone-bridge
- --server-url=http://server:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-300}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
environment:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
networks:
- turnstone-net
depends_on:
@@ -140,6 +194,9 @@ services:
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_AUTH_ENABLED=${TURNSTONE_AUTH_ENABLED:-}
- TURNSTONE_AUTH_TOKEN=${TURNSTONE_AUTH_TOKEN:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
networks:
- turnstone-net
depends_on:
@@ -153,6 +210,78 @@ services:
start_period: 10s
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-channel — Channel gateway (Discord, Slack, etc.)
# Requires TURNSTONE_DISCORD_TOKEN to enable Discord adapter
# -------------------------------------------------------------------
channel:
build:
context: .
dockerfile: Dockerfile
profiles:
- production
- cluster
- ddgCluster
command:
- sh
- -c
- >-
turnstone-channel
--redis-host=redis
--redis-port=6379
--http-host=0.0.0.0
$${TURNSTONE_DISCORD_GUILD:+--discord-guild $$TURNSTONE_DISCORD_GUILD}
environment:
- TURNSTONE_DISCORD_TOKEN=${TURNSTONE_DISCORD_TOKEN:-}
- TURNSTONE_DISCORD_GUILD=${TURNSTONE_DISCORD_GUILD:-0}
- REDIS_PASSWORD=${REDIS_PASSWORD:-}
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:-}
- TURNSTONE_DB_BACKEND=${DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${DATABASE_URL:-}
- TURNSTONE_CHANNEL_ADVERTISE_URL=http://channel:8091
networks:
- turnstone-net
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
required: false
restart: unless-stopped
# -------------------------------------------------------------------
# ddg-search — DuckDuckGo Search MCP server (HTTP transport)
# Provides web search + content fetch tools to turnstone via MCP.
# No API key required.
#
# Start with: MCP_CONFIG=/etc/turnstone/mcp-ddg.json \
# docker compose --profile ddgCluster up
# -------------------------------------------------------------------
ddg-search:
image: python:3.14-slim
profiles:
- ddgCluster
command:
- sh
- -c
- >-
pip install --no-cache-dir duckduckgo-mcp-server &&
python -c "from mcp.server.transport_security import TransportSecuritySettings; import duckduckgo_mcp_server.server as s; s.safe_search=s.SafeSearchMode.OFF; s.mcp.settings.host='0.0.0.0'; s.mcp.settings.port=3000; s.mcp.settings.transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False); s.mcp.run(transport='streamable-http')"
networks:
- turnstone-net
healthcheck:
test: ["CMD-SHELL", "python -c \"import socket; s=socket.create_connection(('0.0.0.0',3000),2); s.close()\""]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
deploy:
resources:
limits:
memory: 256M
cpus: '0.25'
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-sim — Multi-node cluster simulator (no LLM needed)
# Start with: docker compose --profile sim up
@@ -192,3 +321,224 @@ services:
redis:
condition: service_healthy
restart: "no"
# ===================================================================
# 10-node cluster (profile: cluster)
#
# Each node is a server + bridge pair. All share the same PostgreSQL
# and Redis instances. Access via console at :8090.
#
# Start: docker compose --profile cluster up
# ===================================================================
# -- cluster servers ------------------------------------------------
server-1: &cluster-server
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster, ddgCluster]
command: &cluster-server-cmd
- sh
- -c
- >-
turnstone-server
--host 0.0.0.0
--port 8080
--base-url "$${LLM_BASE_URL}"
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
volumes:
- turnstone-data:/data
- ./docker/mcp-ddg.json:/etc/turnstone/mcp-ddg.json:ro
environment: &cluster-server-env
LLM_BASE_URL: ${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
OPENAI_API_KEY: ${OPENAI_API_KEY:-dummy}
TAVILY_API_KEY: ${TAVILY_API_KEY:-}
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_AUTH_ENABLED: ${TURNSTONE_AUTH_ENABLED:-}
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
MODEL: ${MODEL:-}
MCP_CONFIG: ${MCP_CONFIG:-}
TURNSTONE_DB_BACKEND: ${DB_BACKEND:-postgresql}
TURNSTONE_DB_URL: ${DATABASE_URL:-postgresql://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:?}@postgres:5432/turnstone}
TURNSTONE_NODE_ID: node-1
extra_hosts: ["host.docker.internal:host-gateway"]
networks: [turnstone-net]
depends_on:
redis: { condition: service_healthy }
postgres: { condition: service_healthy }
ddg-search: { condition: service_healthy, required: false }
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
deploy:
resources:
limits: { memory: 384M, cpus: '0.5' }
restart: unless-stopped
server-2:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-2 }
server-3:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-3 }
server-4:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-4 }
server-5:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-5 }
server-6:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-6 }
server-7:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-7 }
server-8:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-8 }
server-9:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-9 }
server-10:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-10 }
# -- cluster bridges ------------------------------------------------
bridge-1: &cluster-bridge
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster, ddgCluster]
command:
- turnstone-bridge
- --server-url=http://server-1:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
environment: &cluster-bridge-env
REDIS_PASSWORD: ${REDIS_PASSWORD:-}
TURNSTONE_AUTH_TOKEN: ${TURNSTONE_AUTH_TOKEN:-}
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:-}
networks: [turnstone-net]
depends_on:
server-1: { condition: service_healthy }
redis: { condition: service_healthy }
deploy:
resources:
limits: { memory: 256M, cpus: '0.25' }
restart: unless-stopped
bridge-2:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-2:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-2: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-3:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-3:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-3: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-4:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-4:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-4: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-5:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-5:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-5: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-6:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-6:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-6: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-7:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-7:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-7: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-8:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-8:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-8: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-9:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-9:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-9: { condition: service_healthy }
redis: { condition: service_healthy }
bridge-10:
<<: *cluster-bridge
command:
- turnstone-bridge
- --server-url=http://server-10:8080
- --redis-host=redis
- --redis-port=6379
- --heartbeat-ttl=${HEARTBEAT_TTL:-60}
- --approval-timeout=${APPROVAL_TIMEOUT:-3600}
depends_on:
server-10: { condition: service_healthy }
redis: { condition: service_healthy }
-221
View File
@@ -1,221 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 520" font-family="ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,monospace" font-size="13">
<style>
@keyframes pulse-green { 0%,100% { opacity:0.5 } 50% { opacity:1 } }
@keyframes pulse-yellow { 0%,100% { opacity:0.4 } 50% { opacity:1 } }
@keyframes pulse-blue { 0%,100% { opacity:0.3 } 50% { opacity:1 } }
@keyframes fadein { from { opacity:0 } to { opacity:1 } }
.pg { animation: pulse-green 2s infinite }
.py { animation: pulse-yellow 1.8s infinite }
.pb { animation: pulse-blue 2.2s infinite }
.f1 { animation: fadein 0.4s 0.2s both }
.f2 { animation: fadein 0.4s 0.4s both }
.f3 { animation: fadein 0.4s 0.6s both }
.f4 { animation: fadein 0.4s 0.8s both }
.f5 { animation: fadein 0.4s 1.0s both }
.f6 { animation: fadein 0.4s 1.3s both }
.f7 { animation: fadein 0.4s 1.5s both }
.f8 { animation: fadein 0.4s 1.7s both }
.f9 { animation: fadein 0.4s 1.9s both }
.f10 { animation: fadein 0.4s 2.1s both }
.f11 { animation: fadein 0.4s 2.3s both }
.f12 { animation: fadein 0.4s 2.5s both }
</style>
<!-- Window chrome -->
<rect rx="10" width="860" height="520" fill="#1a1b26"/>
<rect width="860" height="36" rx="10" fill="#16161e"/>
<rect y="26" width="860" height="10" fill="#16161e"/>
<circle cx="20" cy="18" r="6" fill="#f7768e"/>
<circle cx="40" cy="18" r="6" fill="#e0af68"/>
<circle cx="60" cy="18" r="6" fill="#9ece6a"/>
<text x="430" y="22" text-anchor="middle" fill="#565f89" font-size="12">turnstone — console</text>
<!-- Header -->
<rect y="36" width="860" height="30" fill="#24283b"/>
<rect y="66" width="860" height="1" fill="#3b4261"/>
<text x="16" y="56" fill="#7aa2f7" font-size="14" font-weight="bold">turnstone console</text>
<text x="200" y="56" fill="#565f89" font-size="12">6 nodes · 10 workstreams</text>
<!-- ====== State cards ====== -->
<g transform="translate(16, 78)" class="f1" opacity="0">
<!-- RUN card -->
<rect x="0" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="0" y="0" width="156" height="3" rx="6" fill="#9ece6a"/>
<text x="78" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">3</text>
<text x="78" y="50" text-anchor="middle" fill="#565f89" font-size="10">▸ RUN</text>
<!-- THINK card -->
<rect x="168" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="168" y="0" width="156" height="3" rx="6" fill="#7aa2f7"/>
<text x="246" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">2</text>
<text x="246" y="50" text-anchor="middle" fill="#565f89" font-size="10">◌ THINK</text>
<!-- ATTN card -->
<rect x="336" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="336" y="0" width="156" height="3" rx="6" fill="#e0af68"/>
<text x="414" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">1</text>
<text x="414" y="50" text-anchor="middle" fill="#565f89" font-size="10">◆ ATTN</text>
<!-- ERR card -->
<rect x="504" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="504" y="0" width="156" height="3" rx="6" fill="#f7768e"/>
<text x="582" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">0</text>
<text x="582" y="50" text-anchor="middle" fill="#565f89" font-size="10">✖ ERR</text>
<!-- IDLE card -->
<rect x="672" y="0" width="156" height="64" rx="6" fill="#24283b" stroke="#3b4261"/>
<rect x="672" y="0" width="156" height="3" rx="6" fill="#565f89"/>
<text x="750" y="30" text-anchor="middle" fill="#a9b1d6" font-size="22" font-weight="bold">4</text>
<text x="750" y="50" text-anchor="middle" fill="#565f89" font-size="10">· IDLE</text>
</g>
<!-- Aggregate bar -->
<text x="16" y="160" fill="#565f89" font-size="11" class="f2" opacity="0">197k tokens · 42 tool calls</text>
<!-- ====== NODES section ====== -->
<text x="16" y="182" fill="#7aa2f7" font-size="12" font-weight="bold" class="f3" opacity="0">NODES</text>
<!-- Node column headers -->
<g transform="translate(0, 190)" class="f4" opacity="0">
<rect width="860" height="20" fill="#24283b"/>
<rect y="20" width="860" height="1" fill="#3b4261"/>
<text y="14" fill="#565f89" font-size="10" letter-spacing="0.5">
<tspan x="36">NODE</tspan>
<tspan x="560">WS</tspan>
<tspan x="610">RUN</tspan>
<tspan x="660">ATTN</tspan>
<tspan x="710">TOKENS</tspan>
<tspan x="790">LOAD</tspan>
</text>
</g>
<!-- Node rows -->
<g transform="translate(0, 214)">
<!-- Node 1: db-west-04 — 3 ws, 1 running, has-running bar -->
<g class="f5" opacity="0">
<rect y="0" width="860" height="38" fill="#1a1b26"/>
<rect y="0" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="19" r="4" fill="#9ece6a"/>
<text x="36" y="23" fill="#a9b1d6" font-size="12" font-weight="bold">db-west-04</text>
<text x="566" y="23" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="23" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="23" fill="#565f89" font-size="11">0</text>
<text x="710" y="23" fill="#565f89" font-size="11">57.6k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="15" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="15" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="23" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 2: api-east-01 — 3 ws, 1 attention, has-attention bar -->
<g class="f6" opacity="0">
<rect y="40" width="860" height="38" fill="#24283b"/>
<rect y="40" width="3" height="38" fill="#e0af68"/>
<circle cx="22" cy="59" r="4" fill="#9ece6a"/>
<text x="36" y="63" fill="#a9b1d6" font-size="12" font-weight="bold">api-east-01</text>
<text x="566" y="63" fill="#a9b1d6" font-size="11">3</text>
<text x="616" y="63" fill="#565f89" font-size="11">0</text>
<text x="666" y="63" fill="#a9b1d6" font-size="11">1</text>
<text x="710" y="63" fill="#565f89" font-size="11">109k</text>
<!-- Load bar: 3/10 = 30% -->
<rect x="770" y="55" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="55" width="18" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="63" fill="#565f89" font-size="11">30%</text>
</g>
<!-- Node 3: sre-node-03 — 2 ws, 1 running, has-running bar -->
<g class="f7" opacity="0">
<rect y="80" width="860" height="38" fill="#1a1b26"/>
<rect y="80" width="3" height="38" fill="#9ece6a"/>
<circle cx="22" cy="99" r="4" fill="#9ece6a"/>
<text x="36" y="103" fill="#a9b1d6" font-size="12" font-weight="bold">sre-node-03</text>
<text x="566" y="103" fill="#a9b1d6" font-size="11">2</text>
<text x="616" y="103" fill="#a9b1d6" font-size="11">1</text>
<text x="666" y="103" fill="#565f89" font-size="11">0</text>
<text x="710" y="103" fill="#565f89" font-size="11">64.4k</text>
<!-- Load bar: 2/10 = 20% -->
<rect x="770" y="95" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="95" width="12" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="103" fill="#565f89" font-size="11">20%</text>
</g>
<!-- Node 4: analytics-02 — 1 ws, thinking, has-thinking bar -->
<g class="f8" opacity="0">
<rect y="120" width="860" height="38" fill="#24283b"/>
<rect y="120" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="139" r="4" fill="#9ece6a"/>
<text x="36" y="143" fill="#a9b1d6" font-size="12" font-weight="bold">analytics-02</text>
<text x="566" y="143" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="143" fill="#565f89" font-size="11">0</text>
<text x="666" y="143" fill="#565f89" font-size="11">0</text>
<text x="710" y="143" fill="#565f89" font-size="11">18.3k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="135" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="135" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="143" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 5: data-ops-05 — 1 ws, thinking, has-thinking bar -->
<g class="f9" opacity="0">
<rect y="160" width="860" height="38" fill="#1a1b26"/>
<rect y="160" width="3" height="38" fill="#7aa2f7"/>
<circle cx="22" cy="179" r="4" fill="#9ece6a"/>
<text x="36" y="183" fill="#a9b1d6" font-size="12" font-weight="bold">data-ops-05</text>
<text x="566" y="183" fill="#a9b1d6" font-size="11">1</text>
<text x="616" y="183" fill="#565f89" font-size="11">0</text>
<text x="666" y="183" fill="#565f89" font-size="11">0</text>
<text x="710" y="183" fill="#565f89" font-size="11">8.7k</text>
<!-- Load bar: 1/10 = 10% -->
<rect x="770" y="175" width="60" height="6" rx="3" fill="#292e42"/>
<rect x="770" y="175" width="6" height="6" rx="3" fill="#9ece6a"/>
<text x="838" y="183" fill="#565f89" font-size="11">10%</text>
</g>
<!-- Node 6: ml-gpu-07 — 0 ws, empty, no bar -->
<g class="f10" opacity="0">
<rect y="200" width="860" height="38" fill="#24283b"/>
<rect y="200" width="3" height="38" fill="transparent"/>
<circle cx="22" cy="219" r="4" fill="#9ece6a"/>
<text x="36" y="223" fill="#a9b1d6" font-size="12" font-weight="bold">ml-gpu-07</text>
<text x="566" y="223" fill="#565f89" font-size="11">0</text>
<text x="616" y="223" fill="#565f89" font-size="11">0</text>
<text x="666" y="223" fill="#565f89" font-size="11">0</text>
<text x="710" y="223" fill="#565f89" font-size="11">0</text>
<!-- Load bar: 0/10 = 0% (empty track) -->
<rect x="770" y="215" width="60" height="6" rx="3" fill="#292e42"/>
<text x="842" y="223" fill="#565f89" font-size="11">0%</text>
</g>
</g>
<!-- ====== Footer ====== -->
<g transform="translate(0, 468)" class="f12" opacity="0">
<rect width="860" height="1" fill="#3b4261"/>
<rect y="1" width="860" height="24" fill="#16161e"/>
<circle cx="20" cy="14" r="3" fill="#9ece6a"/>
<text x="28" y="18" fill="#565f89" font-size="10">db-west-04</text>
<circle cx="120" cy="14" r="3" fill="#9ece6a"/>
<text x="128" y="18" fill="#565f89" font-size="10">api-east-01</text>
<circle cx="225" cy="14" r="3" fill="#9ece6a"/>
<text x="233" y="18" fill="#565f89" font-size="10">sre-node-03</text>
<circle cx="335" cy="14" r="3" fill="#9ece6a"/>
<text x="343" y="18" fill="#565f89" font-size="10">analytics-02</text>
<circle cx="450" cy="14" r="3" fill="#9ece6a"/>
<text x="458" y="18" fill="#565f89" font-size="10">data-ops-05</text>
<circle cx="560" cy="14" r="3" fill="#9ece6a"/>
<text x="568" y="18" fill="#565f89" font-size="10">ml-gpu-07</text>
<text x="680" y="18" fill="#3b4261" font-size="10">258k tokens · 42 calls · 12m</text>
</g>
<!-- Bottom edge -->
<rect y="493" width="860" height="27" fill="#16161e"/>
<rect y="510" width="860" height="10" rx="10" fill="#16161e"/>
</svg>

Before

Width:  |  Height:  |  Size: 11 KiB

+16
View File
@@ -0,0 +1,16 @@
apiVersion: v2
name: turnstone
description: Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation
type: application
version: 0.1.0
appVersion: "0.3.0"
dependencies:
- name: postgresql
version: ~18.5.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
- name: redis
version: ~25.3.0
repository: https://charts.bitnami.com/bitnami
condition: redis.enabled
+42
View File
@@ -0,0 +1,42 @@
Turnstone {{ .Chart.AppVersion }} has been deployed.
{{- if .Values.ingress.enabled }}
Access the application via your ingress:
{{- range .Values.ingress.hosts }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ .host }}
{{- end }}
{{- else }}
To access the Turnstone server, run:
kubectl port-forward svc/{{ include "turnstone.fullname" . }}-server {{ .Values.server.service.port }}:{{ .Values.server.service.port }}
Then open: http://localhost:{{ .Values.server.service.port }}
To access the Turnstone console (cluster dashboard), run:
kubectl port-forward svc/{{ include "turnstone.fullname" . }}-console {{ .Values.console.service.port }}:{{ .Values.console.service.port }}
Then open: http://localhost:{{ .Values.console.service.port }}
{{- end }}
Components deployed:
- Server: {{ include "turnstone.fullname" . }}-server ({{ .Values.server.replicas }} replica(s))
- Bridge: {{ include "turnstone.fullname" . }}-bridge ({{ .Values.bridge.replicas }} replica(s))
- Console: {{ include "turnstone.fullname" . }}-console ({{ .Values.console.replicas }} replica(s))
{{- if .Values.postgresql.enabled }}
- PostgreSQL (bitnami subchart)
{{- end }}
{{- if .Values.redis.enabled }}
- Redis (bitnami subchart)
{{- end }}
{{- if not .Values.llm.apiKey }}
{{- if not .Values.llm.existingSecret }}
WARNING: No LLM API key configured. Set llm.apiKey or llm.existingSecret in your values.
{{- end }}
{{- end }}
@@ -0,0 +1,163 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "turnstone.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this
(by the DNS naming spec). If release name contains chart name it will be used
as a full name.
*/}}
{{- define "turnstone.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "turnstone.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels.
*/}}
{{- define "turnstone.labels" -}}
helm.sh/chart: {{ include "turnstone.chart" . }}
{{ include "turnstone.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels.
*/}}
{{- define "turnstone.selectorLabels" -}}
app.kubernetes.io/name: {{ include "turnstone.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use.
*/}}
{{- define "turnstone.serviceAccountName" -}}
{{- if .Values.serviceAccount }}
{{- if .Values.serviceAccount.name }}
{{- .Values.serviceAccount.name }}
{{- else }}
{{- include "turnstone.fullname" . }}
{{- end }}
{{- else }}
{{- include "turnstone.fullname" . }}
{{- end }}
{{- end }}
{{/*
Determine the PostgreSQL host.
*/}}
{{- define "turnstone.postgresql.host" -}}
{{- if .Values.postgresql.enabled }}
{{- printf "%s-postgresql" .Release.Name }}
{{- else }}
{{- .Values.database.external.host }}
{{- end }}
{{- end }}
{{/*
Determine the PostgreSQL port.
*/}}
{{- define "turnstone.postgresql.port" -}}
{{- if .Values.postgresql.enabled }}
{{- printf "5432" }}
{{- else }}
{{- .Values.database.external.port | toString }}
{{- end }}
{{- end }}
{{/*
Determine the PostgreSQL database name.
*/}}
{{- define "turnstone.postgresql.database" -}}
{{- if .Values.postgresql.enabled }}
{{- .Values.postgresql.auth.database }}
{{- else }}
{{- .Values.database.external.database }}
{{- end }}
{{- end }}
{{/*
Determine the PostgreSQL username.
*/}}
{{- define "turnstone.postgresql.username" -}}
{{- if .Values.postgresql.enabled }}
{{- .Values.postgresql.auth.username }}
{{- else }}
{{- .Values.database.external.username }}
{{- end }}
{{- end }}
{{/*
Determine the Redis host.
*/}}
{{- define "turnstone.redis.host" -}}
{{- if .Values.redis.enabled }}
{{- printf "%s-redis-master" .Release.Name }}
{{- else }}
{{- .Values.redis.external.host }}
{{- end }}
{{- end }}
{{/*
Determine the Redis port.
*/}}
{{- define "turnstone.redis.port" -}}
{{- if .Values.redis.enabled }}
{{- printf "6379" }}
{{- else }}
{{- .Values.redis.external.port | toString }}
{{- end }}
{{- end }}
{{/*
Determine the secret name for LLM API keys.
*/}}
{{- define "turnstone.llm.secretName" -}}
{{- if .Values.llm.existingSecret }}
{{- .Values.llm.existingSecret }}
{{- else }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- end }}
{{- end }}
{{/*
Determine the secret name for auth tokens.
*/}}
{{- define "turnstone.auth.secretName" -}}
{{- if .Values.auth.existingSecret }}
{{- .Values.auth.existingSecret }}
{{- else }}
{{- printf "%s-secrets" (include "turnstone.fullname" .) }}
{{- end }}
{{- end }}
{{/*
Container image reference.
*/}}
{{- define "turnstone.image" -}}
{{- $tag := .Values.image.tag | default .Chart.AppVersion }}
{{- printf "%s:%s" .Values.image.repository $tag }}
{{- end }}
@@ -0,0 +1,23 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "turnstone.fullname" . }}-config
labels:
{{- include "turnstone.labels" . | nindent 4 }}
data:
TURNSTONE_DB_BACKEND: {{ .Values.database.backend | quote }}
TURNSTONE_DB_HOST: {{ include "turnstone.postgresql.host" . | quote }}
TURNSTONE_DB_PORT: {{ include "turnstone.postgresql.port" . | quote }}
TURNSTONE_DB_NAME: {{ include "turnstone.postgresql.database" . | quote }}
TURNSTONE_DB_USER: {{ include "turnstone.postgresql.username" . | quote }}
TURNSTONE_SERVER_HOST: "0.0.0.0"
TURNSTONE_SERVER_PORT: {{ .Values.server.service.port | quote }}
TURNSTONE_CONSOLE_HOST: "0.0.0.0"
TURNSTONE_CONSOLE_PORT: {{ .Values.console.service.port | quote }}
TURNSTONE_REDIS_HOST: {{ include "turnstone.redis.host" . | quote }}
TURNSTONE_REDIS_PORT: {{ include "turnstone.redis.port" . | quote }}
TURNSTONE_POLL_INTERVAL: "5"
{{- if .Values.llm.baseUrl }}
TURNSTONE_LLM_BASE_URL: {{ .Values.llm.baseUrl | quote }}
{{- end }}
TURNSTONE_LLM_PROVIDER: {{ .Values.llm.provider | quote }}
@@ -0,0 +1,45 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "turnstone.fullname" . }}-bridge
labels:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: bridge
spec:
replicas: {{ .Values.bridge.replicas }}
selector:
matchLabels:
{{- include "turnstone.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: bridge
template:
metadata:
labels:
{{- include "turnstone.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: bridge
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
containers:
- name: bridge
image: {{ include "turnstone.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- turnstone-bridge
- --server-url={{ printf "http://%s-server:%s" (include "turnstone.fullname" .) (.Values.server.service.port | toString) }}
- --redis-host={{ include "turnstone.redis.host" . }}
- --redis-port={{ include "turnstone.redis.port" . }}
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
{{- if and .Values.auth.enabled .Values.auth.existingSecret }}
env:
- name: TURNSTONE_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.auth.existingSecret }}
key: TURNSTONE_AUTH_TOKEN
{{- end }}
resources:
{{- toYaml .Values.bridge.resources | nindent 12 }}
@@ -0,0 +1,62 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "turnstone.fullname" . }}-console
labels:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: console
spec:
replicas: {{ .Values.console.replicas }}
selector:
matchLabels:
{{- include "turnstone.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: console
template:
metadata:
labels:
{{- include "turnstone.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: console
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
containers:
- name: console
image: {{ include "turnstone.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- turnstone-console
- --host=0.0.0.0
- --port={{ .Values.console.service.port }}
- --redis-host={{ include "turnstone.redis.host" . }}
- --redis-port={{ include "turnstone.redis.port" . }}
ports:
- name: http
containerPort: {{ .Values.console.service.port }}
protocol: TCP
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
{{- if and .Values.auth.enabled .Values.auth.existingSecret }}
env:
- name: TURNSTONE_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.auth.existingSecret }}
key: TURNSTONE_AUTH_TOKEN
{{- end }}
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
periodSeconds: 20
resources:
{{- toYaml .Values.console.resources | nindent 12 }}
@@ -0,0 +1,64 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "turnstone.fullname" . }}-server
labels:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: server
spec:
replicas: {{ .Values.server.replicas }}
selector:
matchLabels:
{{- include "turnstone.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: server
template:
metadata:
labels:
{{- include "turnstone.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: server
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
containers:
- name: server
image: {{ include "turnstone.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- turnstone-server
- --host
- "0.0.0.0"
- --port
- {{ .Values.server.service.port | quote }}
ports:
- name: http
containerPort: {{ .Values.server.service.port }}
protocol: TCP
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
{{- if and .Values.auth.enabled .Values.auth.existingSecret }}
- name: TURNSTONE_AUTH_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.auth.existingSecret }}
key: TURNSTONE_AUTH_TOKEN
{{- end }}
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
periodSeconds: 20
resources:
{{- toYaml .Values.server.resources | nindent 12 }}
@@ -0,0 +1,47 @@
{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "turnstone.fullname" . }}
labels:
{{- include "turnstone.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
pathType: {{ .pathType | default "Prefix" }}
backend:
service:
{{- if eq (.service | default "server") "console" }}
name: {{ include "turnstone.fullname" $ }}-console
port:
number: {{ $.Values.console.service.port }}
{{- else }}
name: {{ include "turnstone.fullname" $ }}-server
port:
number: {{ $.Values.server.service.port }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
@@ -0,0 +1,38 @@
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "turnstone.fullname" . }}-migrate
labels:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-1"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 3
template:
metadata:
labels:
{{- include "turnstone.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: migrate
spec:
serviceAccountName: {{ include "turnstone.serviceAccountName" . }}
restartPolicy: OnFailure
containers:
- name: migrate
image: {{ include "turnstone.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- python
- -m
- turnstone.core.storage._migrate
envFrom:
- configMapRef:
name: {{ include "turnstone.fullname" . }}-config
- secretRef:
name: {{ include "turnstone.llm.secretName" . }}
optional: true
env:
- name: TURNSTONE_DB_URL
value: "postgresql+psycopg://$(TURNSTONE_DB_USER):$(POSTGRES_PASSWORD)@$(TURNSTONE_DB_HOST):$(TURNSTONE_DB_PORT)/$(TURNSTONE_DB_NAME)"
@@ -0,0 +1,28 @@
{{- if not .Values.llm.existingSecret }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "turnstone.fullname" . }}-secrets
labels:
{{- include "turnstone.labels" . | nindent 4 }}
type: Opaque
data:
{{- if .Values.llm.apiKey }}
OPENAI_API_KEY: {{ .Values.llm.apiKey | b64enc | quote }}
{{- end }}
{{- if and .Values.postgresql.enabled .Values.postgresql.auth.password }}
POSTGRES_PASSWORD: {{ .Values.postgresql.auth.password | b64enc | quote }}
{{- else if and (not .Values.postgresql.enabled) .Values.database.external.password }}
POSTGRES_PASSWORD: {{ .Values.database.external.password | b64enc | quote }}
{{- end }}
{{- if and .Values.auth.enabled .Values.auth.token (not .Values.auth.existingSecret) }}
TURNSTONE_AUTH_TOKEN: {{ .Values.auth.token | b64enc | quote }}
{{- end }}
{{- if and .Values.redis.enabled .Values.redis.auth }}
{{- if .Values.redis.auth.password }}
REDIS_PASSWORD: {{ .Values.redis.auth.password | b64enc | quote }}
{{- end }}
{{- else if and (not .Values.redis.enabled) .Values.redis.external.password }}
REDIS_PASSWORD: {{ .Values.redis.external.password | b64enc | quote }}
{{- end }}
{{- end }}
@@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "turnstone.fullname" . }}-console
labels:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: console
spec:
type: {{ .Values.console.service.type }}
ports:
- port: {{ .Values.console.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "turnstone.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: console
@@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "turnstone.fullname" . }}-server
labels:
{{- include "turnstone.labels" . | nindent 4 }}
app.kubernetes.io/component: server
spec:
type: {{ .Values.server.service.type }}
ports:
- port: {{ .Values.server.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "turnstone.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: server
@@ -0,0 +1,6 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "turnstone.serviceAccountName" . }}
labels:
{{- include "turnstone.labels" . | nindent 4 }}
+103
View File
@@ -0,0 +1,103 @@
# -- Container image settings
image:
repository: ghcr.io/turnstonelabs/turnstone
tag: ""
pullPolicy: IfNotPresent
# -- Database configuration
database:
# Backend type (postgresql)
backend: postgresql
# External database settings (used when postgresql.enabled is false)
external:
host: ""
port: 5432
database: turnstone
username: turnstone
existingSecret: ""
sslmode: prefer
# -- Bitnami PostgreSQL subchart
postgresql:
enabled: true
auth:
database: turnstone
username: turnstone
# -- Redis configuration
redis:
enabled: true
architecture: standalone
# External Redis settings (used when redis.enabled is false)
external:
host: ""
port: 6379
existingSecret: ""
# -- Turnstone server (main API + web UI)
server:
replicas: 1
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "2"
memory: 1Gi
service:
type: ClusterIP
port: 8080
# -- Turnstone bridge (Redis MQ connector)
bridge:
replicas: 1
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
# -- Turnstone console (cluster dashboard)
console:
replicas: 1
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
service:
type: ClusterIP
port: 8090
# -- LLM provider configuration
llm:
baseUrl: ""
provider: openai
apiKey: ""
existingSecret: ""
# -- Authentication
auth:
enabled: false
token: ""
existingSecret: ""
# -- Ingress configuration
ingress:
enabled: false
className: ""
annotations: {}
hosts: []
# - host: turnstone.example.com
# paths:
# - path: /
# pathType: Prefix
# service: server
tls: []
# - secretName: turnstone-tls
# hosts:
# - turnstone.example.com
@@ -0,0 +1,36 @@
terraform {
required_version = ">= 1.5"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
}
}
provider "aws" {
region = var.aws_region
}
module "turnstone" {
source = "../../modules/aws-ecs"
vpc_id = var.vpc_id
private_subnet_ids = var.private_subnet_ids
public_subnet_ids = var.public_subnet_ids
image_repository = var.image_repository
image_tag = var.image_tag
llm_base_url = var.llm_base_url
openai_api_key = var.openai_api_key
environment = var.environment
name_prefix = var.name_prefix
auth_token = var.auth_token
tags = {
Example = "aws-ecs-basic"
}
}
@@ -0,0 +1,29 @@
output "alb_dns_name" {
description = "DNS name of the Application Load Balancer."
value = module.turnstone.alb_dns_name
}
output "server_url" {
description = "HTTP URL for the Turnstone server."
value = module.turnstone.server_url
}
output "console_url" {
description = "HTTP URL for the Turnstone console."
value = module.turnstone.console_url
}
output "cluster_arn" {
description = "ARN of the ECS cluster."
value = module.turnstone.cluster_arn
}
output "rds_endpoint" {
description = "RDS PostgreSQL endpoint."
value = module.turnstone.rds_endpoint
}
output "redis_endpoint" {
description = "ElastiCache Redis endpoint."
value = module.turnstone.redis_endpoint
}
@@ -0,0 +1,22 @@
# --- Required ---
# VPC and subnet IDs from your existing AWS infrastructure.
# The VPC must have DNS support and DNS hostnames enabled.
vpc_id = "vpc-0123456789abcdef0"
private_subnet_ids = ["subnet-aaa111", "subnet-bbb222"]
public_subnet_ids = ["subnet-ccc333", "subnet-ddd444"]
# LLM provider configuration.
# For OpenAI: https://api.openai.com/v1
# For a self-hosted vLLM instance: http://your-vllm-host:8000/v1
llm_base_url = "https://api.openai.com/v1"
openai_api_key = "sk-..."
# --- Optional ---
# aws_region = "us-east-1"
# image_repository = "ghcr.io/turnstonelabs/turnstone"
# image_tag = "0.3.0"
# environment = "production"
# name_prefix = "turnstone"
# auth_token = "my-secret-token"
@@ -0,0 +1,62 @@
variable "aws_region" {
description = "AWS region to deploy into."
type = string
default = "us-east-1"
}
variable "vpc_id" {
description = "ID of the VPC where all resources will be created."
type = string
}
variable "private_subnet_ids" {
description = "List of private subnet IDs for ECS tasks, RDS, and ElastiCache."
type = list(string)
}
variable "public_subnet_ids" {
description = "List of public subnet IDs for the Application Load Balancer."
type = list(string)
}
variable "image_repository" {
description = "Container image repository."
type = string
default = "ghcr.io/turnstonelabs/turnstone"
}
variable "image_tag" {
description = "Container image tag."
type = string
default = "latest"
}
variable "llm_base_url" {
description = "Base URL for the LLM provider API."
type = string
}
variable "openai_api_key" {
description = "API key for the LLM provider."
type = string
sensitive = true
}
variable "environment" {
description = "Deployment environment name."
type = string
default = "production"
}
variable "name_prefix" {
description = "Prefix for all resource names."
type = string
default = "turnstone"
}
variable "auth_token" {
description = "Optional authentication token for the Turnstone API."
type = string
sensitive = true
default = ""
}
+150
View File
@@ -0,0 +1,150 @@
# ---------- Application Load Balancer ----------
#
# HTTP listeners are provided as a starter baseline. For production, set
# var.certificate_arn to an ACM certificate ARN to enable HTTPS listeners
# that redirect HTTP traffic to TLS.
resource "aws_lb" "this" {
name = "${var.name_prefix}-${var.environment}"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = var.public_subnet_ids
tags = local.common_tags
}
# ---------- Server Target Group + Listeners ----------
resource "aws_lb_target_group" "server" {
name = "${var.name_prefix}-server-${var.environment}"
port = 8080
protocol = "HTTP"
vpc_id = var.vpc_id
target_type = "ip"
tags = local.common_tags
health_check {
path = "/health"
port = "traffic-port"
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 5
interval = 30
matcher = "200"
}
}
# HTTP listener: forwards directly when no certificate, redirects to HTTPS otherwise.
resource "aws_lb_listener" "server" {
count = var.certificate_arn == "" ? 1 : 0
load_balancer_arn = aws_lb.this.arn
port = 80
protocol = "HTTP"
tags = local.common_tags
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.server.arn
}
}
resource "aws_lb_listener" "server_http_redirect" {
count = var.certificate_arn != "" ? 1 : 0
load_balancer_arn = aws_lb.this.arn
port = 80
protocol = "HTTP"
tags = local.common_tags
default_action {
type = "redirect"
redirect {
port = "443"
protocol = "HTTPS"
status_code = "HTTP_301"
}
}
}
resource "aws_lb_listener" "server_https" {
count = var.certificate_arn != "" ? 1 : 0
load_balancer_arn = aws_lb.this.arn
port = 443
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"
certificate_arn = var.certificate_arn
tags = local.common_tags
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.server.arn
}
}
# ---------- Console Target Group + Listeners ----------
resource "aws_lb_target_group" "console" {
name = "${var.name_prefix}-console-${var.environment}"
port = 8090
protocol = "HTTP"
vpc_id = var.vpc_id
target_type = "ip"
tags = local.common_tags
health_check {
path = "/health"
port = "traffic-port"
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 5
interval = 30
matcher = "200"
}
}
# HTTP listener: forwards directly when no certificate, redirects to HTTPS otherwise.
resource "aws_lb_listener" "console" {
count = var.certificate_arn == "" ? 1 : 0
load_balancer_arn = aws_lb.this.arn
port = 8090
protocol = "HTTP"
tags = local.common_tags
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.console.arn
}
}
resource "aws_lb_listener" "console_http_redirect" {
count = var.certificate_arn != "" ? 1 : 0
load_balancer_arn = aws_lb.this.arn
port = 8090
protocol = "HTTP"
tags = local.common_tags
default_action {
type = "redirect"
redirect {
port = "8443"
protocol = "HTTPS"
status_code = "HTTP_301"
}
}
}
resource "aws_lb_listener" "console_https" {
count = var.certificate_arn != "" ? 1 : 0
load_balancer_arn = aws_lb.this.arn
port = 8443
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"
certificate_arn = var.certificate_arn
tags = local.common_tags
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.console.arn
}
}
@@ -0,0 +1,30 @@
# ---------- ElastiCache Subnet Group ----------
resource "aws_elasticache_subnet_group" "this" {
name = "${var.name_prefix}-${var.environment}"
subnet_ids = var.private_subnet_ids
tags = local.common_tags
}
# ---------- ElastiCache Redis Replication Group ----------
resource "aws_elasticache_replication_group" "this" {
replication_group_id = "${var.name_prefix}-${var.environment}"
description = "Turnstone Redis for MQ and session state"
engine = "redis"
engine_version = "7.1"
node_type = var.redis_node_type
num_cache_clusters = 1
port = 6379
subnet_group_name = aws_elasticache_subnet_group.this.name
security_group_ids = [aws_security_group.redis.id]
at_rest_encryption_enabled = true
transit_encryption_enabled = true
automatic_failover_enabled = false
tags = local.common_tags
}
+70
View File
@@ -0,0 +1,70 @@
# ---------- ECS Task Execution Role ----------
# Used by the ECS agent to pull images and retrieve secrets.
resource "aws_iam_role" "ecs_execution" {
name = "${var.name_prefix}-ecs-execution-${var.environment}"
tags = local.common_tags
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Service = "ecs-tasks.amazonaws.com"
}
Action = "sts:AssumeRole"
},
]
})
}
resource "aws_iam_role_policy_attachment" "ecs_execution_base" {
role = aws_iam_role.ecs_execution.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
resource "aws_iam_role_policy" "ecs_execution_secrets" {
name = "${var.name_prefix}-secrets-read-${var.environment}"
role = aws_iam_role.ecs_execution.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"secretsmanager:GetSecretValue",
]
Resource = concat(
[
aws_secretsmanager_secret.openai_api_key.arn,
aws_secretsmanager_secret.db_password.arn,
],
var.auth_token != "" ? [aws_secretsmanager_secret.auth_token[0].arn] : [],
)
},
]
})
}
# ---------- ECS Task Role ----------
# Assumed by the running container. Minimal permissions; extend as needed.
resource "aws_iam_role" "ecs_task" {
name = "${var.name_prefix}-ecs-task-${var.environment}"
tags = local.common_tags
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Service = "ecs-tasks.amazonaws.com"
}
Action = "sts:AssumeRole"
},
]
})
}
+313
View File
@@ -0,0 +1,313 @@
terraform {
required_version = ">= 1.5"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
random = {
source = "hashicorp/random"
version = ">= 3.5"
}
}
}
locals {
full_image = "${var.image_repository}:${var.image_tag}"
common_tags = merge(var.tags, {
Project = "turnstone"
Environment = var.environment
ManagedBy = "terraform"
})
# Shared environment variables injected into every container.
common_env = [
{ name = "TURNSTONE_ENV", value = var.environment },
{ name = "TURNSTONE_DB_BACKEND", value = "postgresql" },
{ name = "TURNSTONE_LLM_BASE_URL", value = var.llm_base_url },
{ name = "TURNSTONE_REDIS_URL", value = "redis://${aws_elasticache_replication_group.this.primary_endpoint_address}:6379/0" },
]
# Secrets pulled from Secrets Manager at container start.
common_secrets = [
{
name = "OPENAI_API_KEY"
valueFrom = aws_secretsmanager_secret_version.openai_api_key.arn
},
{
name = "TURNSTONE_DB_URL"
valueFrom = aws_secretsmanager_secret_version.db_url.arn
},
]
auth_env = var.auth_token != "" ? [
{ name = "TURNSTONE_AUTH_ENABLED", value = "true" },
] : []
auth_secrets = var.auth_token != "" ? [
{
name = "TURNSTONE_AUTH_TOKEN"
valueFrom = aws_secretsmanager_secret_version.auth_token[0].arn
},
] : []
}
# ---------- Secrets Manager ----------
resource "aws_secretsmanager_secret" "openai_api_key" {
name = "${var.name_prefix}-${var.environment}-openai-api-key"
tags = local.common_tags
}
resource "aws_secretsmanager_secret_version" "openai_api_key" {
secret_id = aws_secretsmanager_secret.openai_api_key.id
secret_string = var.openai_api_key
}
resource "aws_secretsmanager_secret" "auth_token" {
count = var.auth_token != "" ? 1 : 0
name = "${var.name_prefix}-${var.environment}-auth-token"
tags = local.common_tags
}
resource "aws_secretsmanager_secret_version" "auth_token" {
count = var.auth_token != "" ? 1 : 0
secret_id = aws_secretsmanager_secret.auth_token[0].id
secret_string = var.auth_token
}
resource "aws_secretsmanager_secret" "db_password" {
name = "${var.name_prefix}-${var.environment}-db-password"
tags = local.common_tags
}
resource "aws_secretsmanager_secret_version" "db_password" {
secret_id = aws_secretsmanager_secret.db_password.id
secret_string = random_password.db.result
}
resource "aws_secretsmanager_secret" "db_url" {
name = "${var.name_prefix}-${var.environment}-db-url"
tags = local.common_tags
}
resource "aws_secretsmanager_secret_version" "db_url" {
secret_id = aws_secretsmanager_secret.db_url.id
secret_string = "postgresql+psycopg://${aws_db_instance.this.username}:${random_password.db.result}@${aws_db_instance.this.endpoint}/turnstone"
}
# ---------- ECS Cluster ----------
resource "aws_ecs_cluster" "this" {
name = "${var.name_prefix}-${var.environment}"
tags = local.common_tags
setting {
name = "containerInsights"
value = "enabled"
}
}
# ---------- CloudWatch Log Group ----------
resource "aws_cloudwatch_log_group" "this" {
name = "/ecs/${var.name_prefix}-${var.environment}"
retention_in_days = 30
tags = local.common_tags
}
# ---------- Server Task Definition + Service ----------
resource "aws_ecs_task_definition" "server" {
family = "${var.name_prefix}-server"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = var.server_cpu
memory = var.server_memory
execution_role_arn = aws_iam_role.ecs_execution.arn
task_role_arn = aws_iam_role.ecs_task.arn
tags = local.common_tags
container_definitions = jsonencode([
{
name = "server"
image = local.full_image
essential = true
command = ["turnstone-server", "--host", "0.0.0.0", "--port", "8080"]
portMappings = [
{ containerPort = 8080, protocol = "tcp" },
]
environment = concat(local.common_env, local.auth_env)
secrets = concat(local.common_secrets, local.auth_secrets)
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.this.name
"awslogs-region" = data.aws_region.current.name
"awslogs-stream-prefix" = "server"
}
}
healthCheck = {
command = ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
interval = 30
timeout = 5
retries = 3
startPeriod = 10
}
},
])
}
resource "aws_ecs_service" "server" {
name = "${var.name_prefix}-server"
cluster = aws_ecs_cluster.this.id
task_definition = aws_ecs_task_definition.server.arn
desired_count = 1
launch_type = "FARGATE"
tags = local.common_tags
network_configuration {
subnets = var.private_subnet_ids
security_groups = [aws_security_group.ecs_tasks.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.server.arn
container_name = "server"
container_port = 8080
}
depends_on = [aws_lb_target_group.server]
}
# ---------- Bridge Task Definition + Service ----------
resource "aws_ecs_task_definition" "bridge" {
family = "${var.name_prefix}-bridge"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = var.bridge_cpu
memory = var.bridge_memory
execution_role_arn = aws_iam_role.ecs_execution.arn
task_role_arn = aws_iam_role.ecs_task.arn
tags = local.common_tags
container_definitions = jsonencode([
{
name = "bridge"
image = local.full_image
essential = true
command = ["turnstone-bridge"]
environment = concat(local.common_env, local.auth_env)
secrets = concat(local.common_secrets, local.auth_secrets)
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.this.name
"awslogs-region" = data.aws_region.current.name
"awslogs-stream-prefix" = "bridge"
}
}
},
])
}
resource "aws_ecs_service" "bridge" {
name = "${var.name_prefix}-bridge"
cluster = aws_ecs_cluster.this.id
task_definition = aws_ecs_task_definition.bridge.arn
desired_count = 1
launch_type = "FARGATE"
tags = local.common_tags
network_configuration {
subnets = var.private_subnet_ids
security_groups = [aws_security_group.ecs_tasks.id]
assign_public_ip = false
}
depends_on = [aws_ecs_service.server]
}
# ---------- Console Task Definition + Service ----------
resource "aws_ecs_task_definition" "console" {
family = "${var.name_prefix}-console"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = var.console_cpu
memory = var.console_memory
execution_role_arn = aws_iam_role.ecs_execution.arn
task_role_arn = aws_iam_role.ecs_task.arn
tags = local.common_tags
container_definitions = jsonencode([
{
name = "console"
image = local.full_image
essential = true
command = ["turnstone-console", "--host", "0.0.0.0", "--port", "8090"]
portMappings = [
{ containerPort = 8090, protocol = "tcp" },
]
environment = concat(local.common_env, local.auth_env)
secrets = concat(local.common_secrets, local.auth_secrets)
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.this.name
"awslogs-region" = data.aws_region.current.name
"awslogs-stream-prefix" = "console"
}
}
healthCheck = {
command = ["CMD-SHELL", "curl -f http://localhost:8090/health || exit 1"]
interval = 30
timeout = 5
retries = 3
startPeriod = 10
}
},
])
}
resource "aws_ecs_service" "console" {
name = "${var.name_prefix}-console"
cluster = aws_ecs_cluster.this.id
task_definition = aws_ecs_task_definition.console.arn
desired_count = 1
launch_type = "FARGATE"
tags = local.common_tags
network_configuration {
subnets = var.private_subnet_ids
security_groups = [aws_security_group.ecs_tasks.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.console.arn
container_name = "console"
container_port = 8090
}
depends_on = [aws_lb_target_group.console]
}
# ---------- Data Sources ----------
data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
@@ -0,0 +1,29 @@
output "alb_dns_name" {
description = "DNS name of the Application Load Balancer."
value = aws_lb.this.dns_name
}
output "server_url" {
description = "HTTP URL for the Turnstone server API and web UI."
value = "http://${aws_lb.this.dns_name}"
}
output "console_url" {
description = "HTTP URL for the Turnstone console dashboard."
value = "http://${aws_lb.this.dns_name}:8090"
}
output "cluster_arn" {
description = "ARN of the ECS cluster."
value = aws_ecs_cluster.this.arn
}
output "rds_endpoint" {
description = "Endpoint of the RDS PostgreSQL instance (host:port)."
value = aws_db_instance.this.endpoint
}
output "redis_endpoint" {
description = "Primary endpoint of the ElastiCache Redis replication group."
value = aws_elasticache_replication_group.this.primary_endpoint_address
}
+42
View File
@@ -0,0 +1,42 @@
# ---------- Random Password ----------
resource "random_password" "db" {
length = 32
special = false
}
# ---------- DB Subnet Group ----------
resource "aws_db_subnet_group" "this" {
name = "${var.name_prefix}-${var.environment}"
subnet_ids = var.private_subnet_ids
tags = local.common_tags
}
# ---------- RDS PostgreSQL ----------
resource "aws_db_instance" "this" {
identifier = "${var.name_prefix}-${var.environment}"
engine = "postgres"
engine_version = "17"
instance_class = var.db_instance_class
allocated_storage = 20
storage_type = "gp3"
storage_encrypted = true
deletion_protection = true
skip_final_snapshot = false
final_snapshot_identifier = "${var.name_prefix}-${var.environment}-final"
db_name = "turnstone"
username = "turnstone"
password = random_password.db.result
db_subnet_group_name = aws_db_subnet_group.this.name
vpc_security_group_ids = [aws_security_group.rds.id]
backup_retention_period = 7
multi_az = false
tags = local.common_tags
}
@@ -0,0 +1,133 @@
# ---------- ALB Security Group ----------
resource "aws_security_group" "alb" {
name = "${var.name_prefix}-alb-${var.environment}"
description = "Allow inbound HTTP to ALB for server and console"
vpc_id = var.vpc_id
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "alb_http" {
security_group_id = aws_security_group.alb.id
description = "HTTP traffic to server"
from_port = 80
to_port = 80
ip_protocol = "tcp"
cidr_ipv4 = "0.0.0.0/0"
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "alb_https" {
count = var.certificate_arn != "" ? 1 : 0
security_group_id = aws_security_group.alb.id
description = "HTTPS traffic to server"
from_port = 443
to_port = 443
ip_protocol = "tcp"
cidr_ipv4 = "0.0.0.0/0"
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "alb_console" {
security_group_id = aws_security_group.alb.id
description = "HTTP traffic to console"
from_port = 8090
to_port = 8090
ip_protocol = "tcp"
cidr_ipv4 = "0.0.0.0/0"
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "alb_console_https" {
count = var.certificate_arn != "" ? 1 : 0
security_group_id = aws_security_group.alb.id
description = "HTTPS traffic to console"
from_port = 8443
to_port = 8443
ip_protocol = "tcp"
cidr_ipv4 = "0.0.0.0/0"
tags = local.common_tags
}
resource "aws_vpc_security_group_egress_rule" "alb_all" {
security_group_id = aws_security_group.alb.id
description = "Allow all outbound"
ip_protocol = "-1"
cidr_ipv4 = "0.0.0.0/0"
tags = local.common_tags
}
# ---------- ECS Tasks Security Group ----------
resource "aws_security_group" "ecs_tasks" {
name = "${var.name_prefix}-ecs-tasks-${var.environment}"
description = "Allow traffic from ALB to ECS tasks and outbound internet"
vpc_id = var.vpc_id
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "ecs_from_alb_server" {
security_group_id = aws_security_group.ecs_tasks.id
description = "Server port from ALB"
from_port = 8080
to_port = 8080
ip_protocol = "tcp"
referenced_security_group_id = aws_security_group.alb.id
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "ecs_from_alb_console" {
security_group_id = aws_security_group.ecs_tasks.id
description = "Console port from ALB"
from_port = 8090
to_port = 8090
ip_protocol = "tcp"
referenced_security_group_id = aws_security_group.alb.id
tags = local.common_tags
}
resource "aws_vpc_security_group_egress_rule" "ecs_all" {
security_group_id = aws_security_group.ecs_tasks.id
description = "Allow all outbound (LLM APIs, ECR, Secrets Manager, etc.)"
ip_protocol = "-1"
cidr_ipv4 = "0.0.0.0/0"
tags = local.common_tags
}
# ---------- RDS Security Group ----------
resource "aws_security_group" "rds" {
name = "${var.name_prefix}-rds-${var.environment}"
description = "Allow PostgreSQL access from ECS tasks"
vpc_id = var.vpc_id
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "rds_from_ecs" {
security_group_id = aws_security_group.rds.id
description = "PostgreSQL from ECS tasks"
from_port = 5432
to_port = 5432
ip_protocol = "tcp"
referenced_security_group_id = aws_security_group.ecs_tasks.id
tags = local.common_tags
}
# ---------- Redis Security Group ----------
resource "aws_security_group" "redis" {
name = "${var.name_prefix}-redis-${var.environment}"
description = "Allow Redis access from ECS tasks"
vpc_id = var.vpc_id
tags = local.common_tags
}
resource "aws_vpc_security_group_ingress_rule" "redis_from_ecs" {
security_group_id = aws_security_group.redis.id
description = "Redis from ECS tasks"
from_port = 6379
to_port = 6379
ip_protocol = "tcp"
referenced_security_group_id = aws_security_group.ecs_tasks.id
tags = local.common_tags
}
@@ -0,0 +1,130 @@
# --- Networking ---
variable "vpc_id" {
description = "ID of the VPC where all resources will be created."
type = string
}
variable "private_subnet_ids" {
description = "List of private subnet IDs for ECS tasks, RDS, and ElastiCache."
type = list(string)
}
variable "public_subnet_ids" {
description = "List of public subnet IDs for the Application Load Balancer."
type = list(string)
}
# --- Container Image ---
variable "image_repository" {
description = "Container image repository."
type = string
default = "ghcr.io/turnstonelabs/turnstone"
}
variable "image_tag" {
description = "Container image tag."
type = string
default = "latest"
}
# --- LLM Provider ---
variable "llm_base_url" {
description = "Base URL for the LLM provider API (e.g. https://api.openai.com/v1)."
type = string
}
variable "openai_api_key" {
description = "API key for the LLM provider. Stored in AWS Secrets Manager."
type = string
sensitive = true
}
# --- RDS ---
variable "db_instance_class" {
description = "RDS instance class for PostgreSQL."
type = string
default = "db.t4g.micro"
}
# --- ElastiCache ---
variable "redis_node_type" {
description = "ElastiCache node type for Redis."
type = string
default = "cache.t4g.micro"
}
# --- ECS Task Sizing ---
variable "server_cpu" {
description = "CPU units for the server task (1 vCPU = 1024)."
type = number
default = 512
}
variable "server_memory" {
description = "Memory (MiB) for the server task."
type = number
default = 1024
}
variable "bridge_cpu" {
description = "CPU units for the bridge task."
type = number
default = 256
}
variable "bridge_memory" {
description = "Memory (MiB) for the bridge task."
type = number
default = 512
}
variable "console_cpu" {
description = "CPU units for the console task."
type = number
default = 256
}
variable "console_memory" {
description = "Memory (MiB) for the console task."
type = number
default = 512
}
# --- General ---
variable "environment" {
description = "Deployment environment name (e.g. production, staging)."
type = string
default = "production"
}
variable "name_prefix" {
description = "Prefix for all resource names."
type = string
default = "turnstone"
}
variable "auth_token" {
description = "Optional authentication token for the Turnstone API. Empty string disables auth."
type = string
sensitive = true
default = ""
}
variable "certificate_arn" {
description = "ACM certificate ARN for HTTPS listeners. Leave empty for HTTP-only (not recommended for production)."
type = string
default = ""
}
variable "tags" {
description = "Additional tags to apply to all resources."
type = map(string)
default = {}
}
+5
View File
@@ -0,0 +1,5 @@
#!/bin/sh
# Run database migrations before starting the service
python -m turnstone.core.storage._migrate || true
# Execute the actual command
exec "$@"
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"ddg": {
"url": "http://ddg-search:3000/mcp"
}
}
}
+1148 -57
View File
File diff suppressed because it is too large Load Diff
+693 -137
View File
File diff suppressed because it is too large Load Diff
+382
View File
@@ -0,0 +1,382 @@
# Channel Integrations
The `turnstone-channel` gateway connects external messaging platforms to
turnstone workstreams via Redis MQ. Each platform adapter translates
platform-native events (messages, button clicks, slash commands) into
turnstone MQ messages, and renders workstream output back into the
platform's UI.
Discord ships as the first adapter. The adapter protocol is designed for
future Slack and Teams integrations.
---
## Architecture
```
Discord Gateway
|
v
turnstone-channel (Discord adapter)
|
v
Redis MQ
|
v
turnstone-bridge ──> turnstone-server
```
Key components:
- **ChannelAdapter protocol** (`turnstone/channels/_protocol.py`) — generic
interface for any messaging platform. Defines `start()`, `stop()`,
`send()`, `send_notification()`, `edit_message()`,
`send_approval_request()`, `send_plan_review()`, and `create_thread()`.
- **ChannelRouter** (`turnstone/channels/_routing.py`) — maps
channel/thread IDs to turnstone workstream IDs. Handles workstream
creation via MQ, stale route detection, and user identity resolution.
- **AsyncRedisBroker** (`turnstone/mq/async_broker.py`) — async Redis
client compatible with discord.py's event loop. Used by the router for
pub/sub and queue operations.
- **channel_users table** — maps `(channel_type, channel_user_id)` to a
turnstone `user_id`. Messages from unlinked users are silently dropped.
- **channel_routes table** — persistent channel-to-workstream mappings.
Survives bot restarts. Stale routes (evicted workstreams) are detected
and refreshed on the next message.
---
## Discord Setup
### 1. Create a Discord Application
1. Go to https://discord.com/developers/applications
2. Click **New Application** and give it a name
3. Navigate to the **Bot** tab and click **Reset Token** to generate a
bot token. Copy it immediately — it is shown only once.
4. On the same **Bot** tab, scroll down to **Privileged Gateway Intents**
and enable **MESSAGE CONTENT INTENT**
5. Navigate to **OAuth2 > URL Generator**
6. Under **Scopes**, check `bot` and `applications.commands`
7. Under **Bot Permissions**, check:
- View Channels
- Send Messages
- Send Messages in Threads
- Create Public Threads
- Read Message History
- Add Reactions
- Embed Links
8. Copy the generated URL, open it in a browser, and add the bot to your
Discord server
### 2. Configure Turnstone
**Environment variables** (recommended for Docker):
```bash
TURNSTONE_DISCORD_TOKEN=your-bot-token-here
TURNSTONE_DISCORD_GUILD=123456789 # optional, restrict to one guild
```
**CLI flags** (bare-metal):
```bash
turnstone-channel \
--discord-token "your-bot-token" \
--discord-guild 123456789 \
--redis-host localhost \
--redis-port 6379
```
**Docker Compose** (production profile):
```bash
# In .env file:
TURNSTONE_DISCORD_TOKEN=your-bot-token
TURNSTONE_DISCORD_GUILD=123456789
```
Then start the stack:
```bash
docker compose --profile production up
```
The `channel` service starts automatically when
`TURNSTONE_DISCORD_TOKEN` is set.
### 3. Link User Accounts
Discord users must link their account to a turnstone user before they can
interact with the bot. Unlinked users' messages are silently ignored.
1. The user must have a turnstone API token — created via the admin panel
or `turnstone-admin create-token`
2. In Discord, the user runs `/link`. A modal appears prompting for the
API token (the token is never visible in Discord audit logs because it
is submitted via modal, not as a slash command argument).
3. The token is validated against the database. If valid, a
`channel_users` mapping is created.
4. The user can now @mention the bot or use slash commands.
An admin can also force-link or unlink users via the console admin panel
(Admin > Channels tab).
---
## Usage
### Conversations
- **@mention** the bot in any allowed channel to start a new conversation.
The bot creates a Discord thread from the message and a turnstone
workstream behind it.
- All subsequent messages in the thread are routed to the same workstream.
- The bot streams responses via message edits, updated approximately every
1.5 seconds.
- If the workstream is evicted for capacity, the next message in the
thread auto-creates a new workstream and atomically resumes the
previous workstream via the `resume_ws` field on
`CreateWorkstreamMessage`. The server resumes the workstream during
creation (same HTTP request), and the bridge emits a
`WorkstreamResumedEvent` back to the channel. The thread receives a
*"Resumed: {name} ({count} messages restored)"* confirmation.
### Slash Commands
| Command | Description |
|---------|-------------|
| `/link` | Link Discord account to turnstone (opens modal for API token) |
| `/unlink` | Unlink Discord account |
| `/ask <message>` | Create a new thread and workstream with an initial message |
| `/status` | Show workstream info for the current thread (ephemeral) |
| `/close` | Close the workstream, delete the route, and archive the thread |
### Tool Approvals
When manual approval is enabled (the default), tool calls are displayed as
an orange embed with:
- Tool name and argument preview
- **Approve** (green), **Reject** (red), **Always Approve** (gray) buttons
- Only linked users can interact with approval buttons
- The approval decision is forwarded through MQ to the bridge, which
relays it to the server
Buttons use static `custom_id` values so they survive bot restarts.
Correlation data (`ws_id`, `correlation_id`) is stored in the embed footer.
**Auto-approval:** When `auto_approve` is true (via `--auto-approve`), or when
all tools in the request match the `auto_approve_tools` list in the adapter
config, the bot auto-responds with approval and posts a
"*Tool auto-approved.*" notice to the thread instead of showing buttons. The
`auto_approve_tools` list is set via the `ChannelConfig.auto_approve_tools`
field (useful for allowing specific tools like `bash` or `read_file` while
still requiring manual approval for others).
### Plan Reviews
Plan review requests are displayed as a blue embed with:
- **Approve Plan** (green) button — approves the plan with empty feedback
- **Request Changes** (gray) button — opens a modal for feedback text
(up to 2000 characters)
- Feedback is forwarded through MQ as a `PlanFeedbackMessage`
---
## Configuration Reference
| CLI Flag | Env Var | Default | Description |
|----------|---------|---------|-------------|
| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Bot token (required to enable Discord) |
| `--discord-guild` | — | `0` (all guilds) | Restrict to a single Discord guild |
| `--discord-channels` | — | empty (all) | Comma-separated channel IDs to allow |
| `--redis-host` | `REDIS_HOST` | `localhost` | Redis host |
| `--redis-port` | — | `6379` | Redis port |
| `--redis-password` | `REDIS_PASSWORD` | — | Redis password |
| `--redis-db` | — | `0` | Redis DB number |
| `--model` | — | server default | Default model for new workstreams |
| `--auto-approve` | — | `false` | Auto-approve ALL tool calls (skips approval buttons entirely) |
| `--http-host` | — | `127.0.0.1` | HTTP server bind address for notify endpoint |
| `--http-port` | `TURNSTONE_CHANNEL_PORT` | `8091` | HTTP server port |
| `--auth-token` | `TURNSTONE_CHANNEL_AUTH_TOKEN` | — | Static auth token for `/v1/api/notify` (alternative to JWT) |
| `--log-level` | `TURNSTONE_LOG_LEVEL` | `INFO` | Log level |
| `--log-format` | `TURNSTONE_LOG_FORMAT` | `auto` | Log format (`auto`/`json`/`text`) |
---
## User Identity
- The `channel_users` table maps `(channel_type, channel_user_id)` to a
turnstone `user_id`
- Self-service linking via the `/link` slash command (modal input, not
visible in Discord audit logs)
- Admin can force-link or unlink via the console admin panel (Admin >
Channels tab). Unlinking uses a styled confirmation modal.
- Unlinked users' messages are silently dropped
- A user can be linked across multiple platforms (e.g. Discord + Slack)
See [Security: Database Schema](security.md#database-schema) for the
`channel_users` table definition.
---
## Workstream Lifecycle
1. **Creation**@mention or `/ask` creates a Discord thread and a
turnstone workstream. The `ChannelRouter` persists the mapping in the
`channel_routes` table.
2. **Active** — messages are routed bidirectionally. The bot streams
responses via message edits (updated every ~1.5 seconds).
3. **Eviction** — the server evicts an idle workstream for capacity. The
route is preserved and the thread stays open.
4. **Reactivation** — the next message in the thread detects the stale
route (no MQ owner) and creates a new workstream with the old `ws_id`
as `resume_ws` on the `CreateWorkstreamMessage`. The server resumes
the workstream during creation (no separate command or reverse lookup
needed). The bridge emits a `WorkstreamResumedEvent` to the channel, and
the thread displays *"Resumed: {name} ({count} messages restored)"*.
If the old workstream was pruned, a fresh one starts with no error.
5. **Close**`/close` command closes the workstream via MQ, deletes the
route, unsubscribes from events, and archives the Discord thread.
---
## Notifications
> See also: [Notification Flow diagram](diagrams/png/17-notify-flow.png)
The `notify` tool allows the LLM to proactively send notifications to
users or channels on external platforms. This is useful for alerting
people about task completion, errors, or important updates without
waiting for them to check in.
### Targeting
Two modes:
- **Username** — provide a turnstone `username`. The gateway resolves
it via the `channel_users` table and sends to all linked channels
(e.g. Discord + future Slack).
- **Direct** — provide `channel_type` + `channel_id` to target a
specific platform channel or user DM.
### Delivery Flow
Notifications bypass MQ for lower latency. The server calls the channel
gateway directly over HTTP:
1. The LLM calls the `notify` tool with a message and target
2. `_exec_notify()` queries the `services` table for healthy channel
gateways (heartbeat within the last 120 seconds)
3. The server mints a service JWT (`aud: turnstone-channel`) via
`ServiceTokenManager` and POSTs to the first healthy gateway. The
payload includes the originating `ws_id` for reply routing.
4. The gateway validates the JWT, resolves the target, and calls
`adapter.send_notification()` which sends the message and tracks
the outgoing message ID for reply routing
5. On failure, the server tries the next gateway. If all fail, it
retries up to 2 more times (delays: 1s, 3s), re-querying the
service registry on each attempt
### Bidirectional Replies
Notifications support multi-turn DM conversations. When a user replies
to a notification DM:
1. The bot looks up the originating `ws_id` from the tracked message ID
(`_notify_ws_map`)
2. Verifies the replying user matches the original notification
recipient (defence in depth — Discord DMs are already private)
3. Routes the reply to the workstream via `router.send_message()`
4. Registers the DM channel for response forwarding
(`_notify_reply_channels`)
5. When the workstream responds (`TurnCompleteEvent`), the response is
forwarded to the DM
6. The response message is itself tracked, so the user can reply again
for another turn
This enables scenarios like an oncall engineer responding to a CI/CD
failure notification from their phone before opening a laptop.
**Limits:**
- Tracking map capped at 100 entries (FIFO eviction of oldest)
- Entries cleaned up on workstream close/unsubscribe
- Replying to an expired notification sends
*"This notification is no longer active."*
- DM reply content capped at 4096 characters
### Service Registry
The channel gateway registers itself in the `services` database table
on startup and sends a heartbeat every 30 seconds. On shutdown it
deregisters. Services are considered stale after 120 seconds (4 missed
heartbeats) and are excluded from `list_services()` queries.
The `services` table schema:
| Column | Description |
|--------|-------------|
| `service_type` | Service category (e.g. `"channel"`) |
| `service_id` | Unique instance ID (`channel-<hostname>-<random>`) |
| `url` | HTTP base URL for the service |
| `last_heartbeat` | ISO 8601 timestamp of last heartbeat |
| `created` | ISO 8601 timestamp of initial registration |
### Security
- **Authentication** — the gateway's `POST /v1/api/notify` endpoint
requires authentication. Configure either `TURNSTONE_JWT_SECRET`
(the server mints JWTs with `aud: turnstone-channel` automatically)
or a static token via `--auth-token`. If neither is set, the
gateway fails closed and rejects all requests with 401. Server JWTs
(`aud: turnstone-server`) are rejected.
- **Rate limit** — maximum 5 notifications per turn. The counter only
increments on successful delivery, so failures don't consume the
budget.
- **SSRF protection** — only `http://` and `https://` service URLs
are allowed. Other schemes are silently skipped.
- **Mention sanitization** — `discord.utils.escape_mentions()` is
applied before sending, preventing `@everyone` / `@here` abuse.
- **Error redaction** — generic error messages are returned to the
LLM. Internal details (service IDs, URLs, exception messages) are
logged server-side only.
---
## Adding New Adapters
The `ChannelAdapter` protocol defines the interface any platform adapter
must implement:
```python
class ChannelAdapter(Protocol):
channel_type: str
async def start(self) -> None: ...
async def stop(self) -> None: ...
async def send(self, channel_id: str, content: str) -> str: ...
async def send_notification(self, channel_id: str, content: str, ws_id: str) -> str: ...
async def edit_message(self, channel_id: str, message_id: str, content: str) -> None: ...
async def send_approval_request(self, channel_id: str, ws_id: str, correlation_id: str, items: list[dict]) -> None: ...
async def send_plan_review(self, channel_id: str, ws_id: str, correlation_id: str, content: str) -> None: ...
async def create_thread(self, parent_channel_id: str, name: str, message_id: str = "") -> str: ...
```
`send_notification()` is like `send()` but associates the outgoing
message with a `ws_id` so that user replies can be routed back to the
originating workstream. Adapters must track the mapping from outgoing
message ID to `(ws_id, target_user_id)` and handle DM replies.
To add a new platform:
1. Create `turnstone/channels/<platform>/` package
2. Implement the `ChannelAdapter` protocol
3. Add a `--<platform>-token` flag and detection logic in
`turnstone/channels/cli.py`
4. Add the optional dependency in `pyproject.toml` (e.g.
`turnstone[slack]`)
See `turnstone/channels/discord/` as a reference implementation.
+477 -28
View File
@@ -1,28 +1,40 @@
# Cluster Dashboard (turnstone-console)
`turnstone-console` is a standalone monitoring service that provides cluster-wide visibility across all turnstone nodes. It connects to the shared Redis broker, discovers nodes via heartbeat keys, polls each node's HTTP API for workstream data, and subscribes to a cluster event channel for real-time state changes.
`turnstone-console` is a cluster management service that provides cluster-wide visibility and control across all turnstone nodes. It connects to the shared Redis broker, discovers nodes via heartbeat keys, polls each node's HTTP API for workstream data, and subscribes to a cluster event channel for real-time state changes.
The console is read-only — it observes but does not own workstreams or drive LLM sessions.
The console also supports **workstream creation** (dispatched via MQ to target nodes) and a **reverse proxy** that serves each node's server UI through the console port — so users only need network access to the console, not to individual server nodes.
## Architecture
> See also: [Console Data Flow diagram](diagrams/png/11-console-data-flow.png)
```
turnstone-server ── turnstone-bridge ──→ Redis ──→ turnstone-console ──→ Browser
(per node) (per node) (shared) (one instance)
┌── Redis ←── turnstone-bridge ── turnstone-server
│ (MQ) (per node) (per node)
turnstone-console ──────┤
(one instance) │
└── turnstone-server (direct HTTP proxy)
Browser
```
Each bridge publishes state changes to `{prefix}:events:cluster` on Redis pub/sub. The console subscribes once to that channel for real-time updates and periodically polls each node's `GET /api/dashboard` for full workstream snapshots.
Data flows in two directions:
- **Inbound (monitoring):** Bridges publish state changes to `{prefix}:events:cluster` on Redis pub/sub. The console subscribes for real-time updates and periodically polls each node's `GET /v1/api/dashboard` for full workstream snapshots.
- **Outbound (control):** The console pushes `CreateWorkstreamMessage` to Redis inbound queues targeting specific nodes. Bridges pick up these messages and create workstreams on their local servers.
- **Proxy (pass-through):** The console reverse-proxies each node's server UI at `/node/{node_id}/`, forwarding HTTP and SSE traffic so the browser never contacts server nodes directly.
### Data Sources
| Source | Method | Frequency | Data |
| Source | Method | Direction | Data |
|--------|--------|-----------|------|
| Redis heartbeats | `SCAN turnstone:node:*` | Every 15s | Node discovery (node_id, server_url, started) |
| Redis pub/sub | `SUBSCRIBE turnstone:events:cluster` | Real-time | State changes, creates, closes, renames |
| Node HTTP API | `GET {server_url}/api/dashboard` | Every 10s | Full workstream list with tokens, context, activity |
| Node HTTP API | `GET {server_url}/health` | Every 10s | Node health status |
| Redis heartbeats | `SCAN turnstone:node:*` | Read | Node discovery (node_id, server_url, started) |
| Redis pub/sub | `SUBSCRIBE turnstone:events:cluster` | Read | State changes, creates, closes, renames |
| Node HTTP API | `GET {server_url}/v1/api/dashboard` | Read | Full workstream list with tokens, context, activity |
| Node HTTP API | `GET {server_url}/health` | Read | Node health status |
| Redis inbound queue | `RPUSH turnstone:inbound:{node_id}` | Write | Workstream creation commands |
| Node HTTP API | `GET/POST {server_url}/*` | Proxy | Server UI, API requests, SSE streams |
### Redis Key: Cluster Event Channel
@@ -47,7 +59,9 @@ The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot
2. **Node discovery** — scans heartbeat keys every 15 seconds via `broker.list_nodes()`. Adds newly discovered nodes, removes expired ones, emits `node_joined` / `node_lost` events to SSE listeners.
3. **Poll loop** — fetches `GET /api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
3. **Poll loop** — fetches `GET /v1/api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
A `get_snapshot()` method builds the full cluster state under a single lock acquisition — overview aggregates and per-node workstream lists in one atomic read. This is served both as a REST endpoint and as the initial SSE event on client connect.
### Thread Safety
@@ -64,7 +78,7 @@ All reads and writes to the node/workstream map are protected by a single `threa
## HTTP API
### `GET /api/cluster/overview`
### `GET /v1/api/cluster/overview`
Cluster-wide state counts and aggregate metrics.
@@ -73,11 +87,15 @@ Cluster-wide state counts and aggregate metrics.
"nodes": 847,
"workstreams": 4219,
"states": {"running": 1847, "thinking": 312, "attention": 89, "idle": 1940, "error": 31},
"aggregate": {"total_tokens": 12400000, "total_tool_calls": 34200}
"aggregate": {"total_tokens": 12400000, "total_tool_calls": 34200},
"version_drift": true,
"versions": ["0.3.0", "0.3.1"]
}
```
### `GET /api/cluster/nodes?sort=activity&limit=100&offset=0`
`version_drift` is `true` when nodes report different versions. `versions` lists all unique version strings sorted alphabetically.
### `GET /v1/api/cluster/nodes?sort=activity&limit=100&offset=0`
Paginated node list. Sort options: `activity` (default, by running+attention count), `tokens`, `name`.
@@ -91,14 +109,15 @@ Paginated node list. Sort options: `activity` (default, by running+attention cou
"total_tokens": 48200,
"started": 1709294400.0,
"reachable": true,
"health": {}
"health": {"status": "ok", "version": "0.3.0"},
"version": "0.3.0"
}
],
"total": 847
}
```
### `GET /api/cluster/workstreams?state=running&node=db-west-04&search=perf&page=1&per_page=50`
### `GET /v1/api/cluster/workstreams?state=running&node=db-west-04&search=perf&page=1&per_page=50`
Filtered, paginated workstream list. All query parameters are optional. `per_page` is capped at 200.
@@ -115,7 +134,7 @@ Filtered, paginated workstream list. All query parameters are optional. `per_pag
}
```
### `GET /api/cluster/node/{node_id}`
### `GET /v1/api/cluster/node/{node_id}`
Single node detail with all its workstreams.
@@ -129,9 +148,75 @@ Single node detail with all its workstreams.
}
```
### `GET /api/cluster/events`
### `GET /v1/api/cluster/snapshot`
Server-Sent Events stream for real-time cluster updates.
Full cluster state in a single response — all nodes with their workstreams plus overview aggregates. Built under a single lock for internal consistency. Used by the browser on initial load and SSE reconnect.
```json
{
"nodes": [
{
"node_id": "db-west-04",
"server_url": "http://10.0.3.4:8080",
"max_ws": 10,
"reachable": true,
"version": "0.3.0",
"health": {"status": "ok", "version": "0.3.0"},
"aggregate": {"total_tokens": 48200, "total_tool_calls": 156},
"workstreams": [
{"id": "a1b2c3d4", "name": "perf-db-west", "state": "running", ...}
]
}
],
"overview": {
"nodes": 847,
"workstreams": 4219,
"states": {"running": 1847, "thinking": 312, "attention": 89, "idle": 1940, "error": 31},
"aggregate": {"total_tokens": 12400000, "total_tool_calls": 34200},
"version_drift": false,
"versions": ["0.3.0"]
},
"timestamp": 1709294400.0
}
```
### `POST /v1/api/cluster/workstreams/new`
Create a new workstream on a target node. Dispatches a `CreateWorkstreamMessage` through the Redis MQ pipeline — the bridge on the target node picks it up and creates the workstream on the server. Requires `write` scope.
Request:
```json
{
"node_id": "db-west-04",
"name": "perf-analysis",
"model": "gpt-5"
}
```
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and pushes to its directed queue.
- **`"pool"`** — pushes to the shared inbound queue; the next available bridge picks it up (true general-pool dispatch).
- **specific node ID** — pushes to that node's directed queue.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
Response:
```json
{
"status": "ok",
"correlation_id": "a1b2c3d4e5f6",
"target_node": "db-west-04"
}
```
Creation is asynchronous — the response confirms the MQ message was dispatched. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
### `GET /v1/api/cluster/events`
Server-Sent Events stream for real-time cluster updates. The first event is always a `snapshot` containing the full cluster state (same shape as `GET /v1/api/cluster/snapshot` with an added `type: "snapshot"` field), followed by incremental events:
```
data: {"type":"cluster_state","ws_id":"a1b2","node_id":"db-west-04","state":"running"}
@@ -146,32 +231,395 @@ Keepalive comments (`: keepalive\n\n`) are sent every 5 seconds. Clients should
### `GET /health`
```json
{"status": "ok", "service": "turnstone-console", "nodes": 847, "workstreams": 4219}
{
"status": "ok",
"service": "turnstone-console",
"nodes": 847,
"workstreams": 4219,
"version_drift": false,
"versions": ["0.3.0"]
}
```
### Admin API
User and token management endpoints. All admin endpoints require `approve` scope, except for the setup endpoint which is public.
#### `POST /v1/api/auth/setup`
Creates the first admin user when no users exist. Public endpoint (no auth required). Returns a JWT and sets a session cookie. Returns `409` if users already exist. See [Security: First-time setup](security.md#first-time-setup) for full details.
#### `POST /v1/api/admin/users`
Create a new user.
```json
{
"username": "alice",
"password": "s3cret",
"scopes": ["read", "write"]
}
```
#### `GET /v1/api/admin/users`
List all users.
```json
{
"users": [
{"user_id": "u_abc123", "username": "alice", "scopes": ["read", "write"], "created": "2026-03-01T12:00:00Z"}
]
}
```
#### `DELETE /v1/api/admin/users/{user_id}`
Delete a user and revoke all their tokens.
#### `POST /v1/api/admin/users/{user_id}/tokens`
Create an API token for the given user. Returns a `ts_`-prefixed token string that can be used for Bearer auth or passed to `client.login(token="ts_xxx")`.
```json
{
"name": "CI pipeline",
"scopes": ["read", "write"]
}
```
#### `GET /v1/api/admin/users/{user_id}/tokens`
List active tokens for a user (token strings are not returned, only metadata).
#### `DELETE /v1/api/admin/tokens/{token_id}`
Revoke a specific API token.
### Channel links
| Method | Path | Description |
|--------|------|-------------|
| GET | `/v1/api/admin/users/{user_id}/channels` | List channel links for a user |
| POST | `/v1/api/admin/users/{user_id}/channels` | Link a channel account (channel_type, channel_user_id) |
| DELETE | `/v1/api/admin/channels/{channel_type}/{channel_user_id}` | Unlink a channel account |
These endpoints manage the `channel_users` table mappings that connect external platform identities (e.g. Discord user IDs) to turnstone users. See [Channel Integrations](channels.md) for details on the linking flow.
#### `GET /v1/api/auth/status`
Public endpoint for login UI state detection. Returns auth configuration, not
current-user identity.
```json
{
"auth_enabled": true,
"has_users": true,
"setup_required": false
}
```
### Auth Scopes
The auth system uses three scopes instead of the earlier read/full role model:
| Scope | Grants |
|-------|--------|
| `read` | Read-only access: dashboards, workstream lists, SSE streams, health |
| `write` | Send messages, create/close workstreams, approve tool calls |
| `approve` | Admin operations: manage users and API tokens |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
---
## Reverse Proxy
The console reverse-proxies each node's server UI at `/node/{node_id}/`. This allows users to interact with any node's workstreams through the console port alone — individual server ports do not need to be exposed to the office network.
### Proxy Routes
| Route | Behavior |
|-------|----------|
| `GET /node/{node_id}/` | Fetches the server's `index.html`, rewrites static and shared asset paths, injects a console-return banner and an inline JS proxy shim |
| `GET /node/{node_id}/static/{path}` | Proxies page-specific static files |
| `GET /node/{node_id}/shared/{path}` | Proxies shared static files (`base.css`, `auth.js`, etc.) |
| `GET /node/{node_id}/v1/api/{path}` | Proxies GET API requests; detects SSE endpoints and streams them |
| `POST /node/{node_id}/v1/api/{path}` | Proxies POST API requests with body forwarding |
| `GET /node/{node_id}/{path}` | Proxies non-API endpoints (health, metrics) |
### URL Rewriting
The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/shared/base.css`, etc.). Since `<base>` tags cannot rewrite root-relative URLs, the console uses a JS shim approach:
1. **HTML rewriting** — when serving `index.html`, replaces `href=` and `src=` references to both `/static/` and `/shared/` with the proxy prefix (`/node/{node_id}/static/` and `/node/{node_id}/shared/` respectively).
2. **Inline JS shim** — injects an inline `<script>` block into the proxied HTML (after the console-return banner, before any external scripts) that overrides `window.fetch()` and `window.EventSource()` to prepend the proxy prefix to any root-relative URL. Running the shim inline ensures it executes before any external scripts load, so all API calls and SSE connections are intercepted transparently.
3. **Console-return banner** — injects a thin inline-styled `<div>` after `<body>` with a "← Console" link and the node ID, providing navigation back to the dashboard.
### SSE Proxy
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
### Authentication
The proxy forwards the user's JWT to upstream server nodes — it extracts the token from the incoming request's cookie (or `Authorization` header) and adds it as a `Bearer` header on the proxied request. Since all services share the same `TURNSTONE_JWT_SECRET`, the user's JWT is valid on every node without re-authentication. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require `write` scope, preventing read-only tokens from escalating via proxy. The static `--auth-token` / `proxy_auth_token` is used as a fallback when no user JWT is present.
---
## Browser Dashboard
The web UI has three views, toggled client-side:
The web UI has five views, toggled client-side:
### 1. Cluster Overview (landing)
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, HEALTH. Sorted by activity. Clickable rows drill down to node detail.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
### 2. Node Drill-down
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's own dashboard (`http://{server_url}/`).
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
**Deep linking:** Clicking a workstream row opens the node's server UI in a new tab with `?ws_id=<id>`, which auto-selects that workstream. A `↗` indicator appears on hover to signal the external navigation. Rows without a `server_url` are non-interactive.
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
### 3. Filtered Workstreams
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows are deep-linkable when `server_url` is available (injected by the collector from the parent node).
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
All three views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" pushes to the shared queue for any bridge to pick up, or a specific node from the list (showing capacity).
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
### 5. Admin Panel
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with 13 tabs (see also
[Governance](governance.md) for
the Roles, Policies, Skills, Usage, and Audit tabs, and
[Settings](settings.md) for the database-backed configuration editor):
**Users tab:**
- Grid table listing all users (username, display name, role, creation date)
- "Create User" button opens a modal with fields for username, display name,
and password (validated: username 1-64 ASCII, password min 8 characters)
- Delete button on each row opens a styled confirmation modal before
removing the user and cascading to revoke all their tokens
**Tokens tab:**
- User selector dropdown to pick which user's tokens to manage
- Grid table listing tokens for the selected user (name, prefix, scopes,
creation date)
- Scope badges rendered as colored pills for visual clarity
- "Create Token" button opens a modal with fields for token name and scope
checkboxes
- On creation, a "Token Created" modal displays the raw `ts_`-prefixed
token with a copy button. The token is shown once and cannot be retrieved
again.
- Revoke button on each row opens a styled confirmation modal before
deleting the token
**Channels tab:**
- User selector dropdown to pick which user's channel links to manage
- Grid table listing linked channel accounts for the selected user
(channel type, channel user ID, creation date)
- "Link Channel" button opens a modal with fields for channel type
(e.g. `discord`) and the platform user ID
- Unlink button on each row opens a styled confirmation modal before
removing the channel mapping
- Admins can force-link users who have not self-linked via `/link` in
Discord
**MCP Servers tab:**
The tab has two views toggled via a pill control: **Servers** and
**Registry**.
- **Servers view** -- lists all installed MCP servers with source badges
(CONFIG, MANUAL, REGISTRY), transport badges, tool/resource/prompt
counts, per-node connection status, and CRUD actions for DB-managed
servers
- **Registry view** -- search the official MCP Registry to discover and
install servers. Results show server name, description, version, source
type badges (remote/npm/pypi), and Install/Installed/Update buttons.
Remote servers without required configuration are installed with one
click; servers needing env vars, headers, or URL variables open an
install modal for configuration
**Accessibility:**
- Full keyboard navigation: focus traps in modals, Escape to close, arrow
keys for tab switching
- Responsive layout with column hiding at 700px breakpoint
**First-time setup:**
The console also exposes `POST /v1/api/auth/setup` for first-time
bootstrap. When no users exist, the setup wizard calls this public endpoint
to create the initial admin user and receive a JWT in one step. See
[Security: First-time setup](security.md#first-time-setup) for details.
---
## Scheduled Tasks
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via the MQ broker. It supports cron-based recurring schedules and one-shot `at` schedules.
### Architecture
The scheduler runs as a daemon thread inside the console process. Every `check_interval` seconds (default 15) it:
1. Acquires a distributed lock via Redis `SET NX EX` (prevents duplicate dispatch in multi-console deployments)
2. Queries the storage backend for tasks whose `next_run <= now` and `enabled = true`
3. Dispatches each due task as one or more `CreateWorkstreamMessage` via MQ
4. Updates `last_run` and computes the next `next_run` (or disables one-shot `at` tasks)
5. Releases the lock via Lua script (safe conditional delete)
Run history is automatically pruned (runs older than 90 days) approximately once per hour.
### Schedule Types
| Type | Field | Behavior |
|------|-------|----------|
| `cron` | `cron_expr` | Recurring schedule using standard 5-field cron syntax. Requires `croniter`. |
| `at` | `at_time` | One-shot: fires once at the given ISO 8601 timestamp (must include timezone), then auto-disables. |
### Target Modes
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Pushes to the shared inbound queue (any bridge picks it up) |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
### Configuration
| Parameter | Default | Description |
|-----------|---------|-------------|
| `check_interval` | `15.0` | Seconds between scheduler ticks |
| `lock_ttl` | `60` | Distributed lock TTL in seconds |
| `max_fan_out` | `20` | Maximum nodes for `all` target mode |
Dependency: `croniter` (installed with turnstone).
### Schedule API
All schedule endpoints require `approve` scope. Maximum 200 schedules.
#### `GET /v1/api/admin/schedules`
List all scheduled tasks.
```json
{
"schedules": [
{
"task_id": "a1b2c3d4",
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"auto_approve_tools": [],
"enabled": true,
"created_by": "u_admin",
"last_run": "2026-03-05T02:00:00Z",
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-01T12:00:00Z",
"updated": "2026-03-05T02:00:01Z"
}
]
}
```
#### `POST /v1/api/admin/schedules`
Create a scheduled task.
Request:
```json
{
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 * * *",
"target_mode": "auto",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"enabled": true
}
```
Required fields: `name`, `schedule_type`, `initial_message`. For `cron` schedules provide `cron_expr`; for `at` schedules provide `at_time` (ISO 8601 with timezone, must be in the future).
Response: `ScheduleInfo` (same shape as list items above). Returns `400` for invalid cron syntax, naive timestamps, or past `at_time`. Returns `409` if the 200-schedule cap is reached.
#### `GET /v1/api/admin/schedules/{task_id}`
Get a single scheduled task. Returns `ScheduleInfo` or `404`.
#### `PUT /v1/api/admin/schedules/{task_id}`
Partial update — only include fields to change. If `schedule_type`, `cron_expr`, or `at_time` change, `next_run` is recomputed automatically.
```json
{
"enabled": false
}
```
Response: updated `ScheduleInfo`. Returns `400` for validation errors, `404` if not found.
#### `DELETE /v1/api/admin/schedules/{task_id}`
Delete a scheduled task and all its run history. Returns `{"status": "ok"}` or `404`.
#### `GET /v1/api/admin/schedules/{task_id}/runs?limit=50`
List execution history for a task (most recent first). `limit` defaults to 50, max 200.
```json
{
"runs": [
{
"run_id": "r_abc123",
"task_id": "a1b2c3d4",
"node_id": "db-west-04",
"ws_id": "ws_xyz",
"correlation_id": "corr_789",
"started": "2026-03-05T02:00:00Z",
"status": "dispatched",
"error": ""
}
]
}
```
Status is `dispatched` on success or `failed` with an `error` message (e.g. no reachable nodes). Failed runs do not advance `next_run`.
---
@@ -201,6 +649,7 @@ CLI flags for `turnstone-console`:
| `--redis-password` | `$REDIS_PASSWORD` | Redis password |
| `--redis-db` | `0` | Redis DB |
| `--poll-interval` | `10` | Node polling interval (seconds) |
| `--auth-token` | `$TURNSTONE_AUTH_TOKEN` | Bearer token for server node communication and proxy |
| `--log-level` | `INFO` | Log level |
Config file (`~/.config/turnstone/config.toml`):
@@ -233,7 +682,7 @@ turnstone-server --port 8080
turnstone-bridge --server-url http://localhost:8080 --node-id node-a
# Start cluster console (one instance)
turnstone-console --redis-host localhost --port 8090
turnstone-console --redis-host localhost --port 8090 --auth-token "$TURNSTONE_AUTH_TOKEN"
```
Open `http://localhost:8090` for the cluster dashboard.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
+11 -8
View File
@@ -9,7 +9,10 @@ actor "External Client\n(Python / CI)" as ext_client
actor "Eval Harness" as eval_user
' External Systems
cloud "LLM Provider\n(OpenAI-compatible API)" as llm
cloud "LLM Providers" as llm {
component [OpenAI-compatible API\n(OpenAI, vLLM, llama.cpp)] as llm_openai
component [Anthropic Messages API] as llm_anthropic
}
database "Redis" as redis
database "SQLite\n(.turnstone.db)" as sqlite
@@ -31,21 +34,21 @@ ext_client --> redis : Redis LIST\n(push commands)
eval_user --> eval : Python API
' Internal connections
cli --> llm : OpenAI Streaming API\n(HTTPS)
cli --> llm : LLM Provider API\n(via provider adapters)
cli --> sqlite : SQLite
server --> llm : OpenAI Streaming API\n(HTTPS)
server --> llm : LLM Provider API\n(via provider adapters)
server --> sqlite : SQLite
eval --> llm : OpenAI API\n(non-streaming)
eval --> llm : LLM Provider API\n(non-streaming)
eval --> sqlite : SQLite
bridge --> server : HTTP REST\n(POST /api/send, etc.)
bridge <-- server : SSE\n(GET /api/events)
bridge --> server : HTTP REST\n(POST /v1/api/send, etc.)
bridge <-- server : SSE\n(GET /v1/api/events)
bridge --> redis : Redis LIST + PUBSUB\n+ STRING (routing, heartbeats)
console --> redis : Redis PUBSUB + STRING\n(cluster channel, heartbeats)
console --> server : HTTP polling\n(GET /api/dashboard)
console --> redis : Redis PUBSUB + STRING + LIST\n(cluster events, heartbeats,\nworkstream creation commands)
console --> server : HTTP polling + reverse proxy\n(GET /v1/api/dashboard,\nproxy /node/{id}/* traffic)
sim --> redis : Redis LIST + PUBSUB\n+ STRING (heartbeats)
+48 -3
View File
@@ -11,6 +11,8 @@ skinparam component {
BackgroundColor<<console>> #B2EBF2
BackgroundColor<<ui>> #F0F4C3
BackgroundColor<<artifact>> #ECEFF1
BackgroundColor<<sdk>> #FFCDD2
BackgroundColor<<api>> #D1C4E9
}
' Entry points
@@ -24,9 +26,11 @@ package "Entry Points" <<Rectangle>> {
' Core engine
package "turnstone/core/" <<Rectangle>> {
component [session.py\nChatSession, SessionUI] as session <<core>>
component [providers/\nLLMProvider, OpenAI, Anthropic] as providers <<core>>
component [workstream.py\nWorkstreamManager] as workstream <<core>>
component [tools.py\nTool loader] as tools <<core>>
component [memory.py\nSQLite + FTS5] as memory <<core>>
component [memory.py\nPersistence facade] as memory <<core>>
component [storage/\nStorageBackend protocol\nSQLite + PostgreSQL] as storage <<core>>
component [metrics.py\nPrometheus metrics] as metrics <<core>>
component [config.py\nTOML config] as config <<core>>
component [safety.py\nPath validation] as safety <<core>>
@@ -36,7 +40,8 @@ package "turnstone/core/" <<Rectangle>> {
component [auth.py\nAuthentication] as auth <<core>>
component [healthcheck.py\nBackendHealthMonitor] as healthcheck <<core>>
component [ratelimit.py\nRateLimiter] as ratelimit <<core>>
component [mcp_client.py\nMCPClientManager] as mcp <<core>>
component [mcp_client.py\nMCPClientManager\n(push + periodic refresh)] as mcp <<core>>
component [tool_search.py\nToolSearchManager, BM25] as toolsearch <<core>>
component [model_registry.py\nModelRegistry] as registry <<core>>
}
@@ -70,11 +75,29 @@ package "turnstone/ui/" <<Rectangle>> {
component [colors.py\nANSI colors] as colors <<ui>>
component [markdown.py\nMD rendering] as markdown <<ui>>
component [spinner.py\nTerminal spinner] as spinner <<ui>>
component [renderer.js\nBrowser MD + LaTeX] as renderer <<ui>>
}
' API schemas
package "turnstone/api/" <<Rectangle>> {
component [schemas.py\nShared Pydantic models] as apischemas <<api>>
component [server_spec.py\nServer OpenAPI spec] as serverspec <<api>>
component [console_spec.py\nConsole OpenAPI spec] as consolespec <<api>>
component [openapi.py\nSpec builder] as openapi <<api>>
component [docs.py\nSwagger UI handler] as apidocs <<api>>
}
' SDK
package "turnstone/sdk/" <<Rectangle>> {
component [server.py\nTurnstoneServer (sync+async)] as sdkserver <<sdk>>
component [console.py\nTurnstoneConsole (sync+async)] as sdkconsole <<sdk>>
component [events.py\n27 SSE event types] as sdkevents <<sdk>>
component [_base.py\nhttpx client base] as sdkbase <<sdk>>
}
' Tool schemas
package "turnstone/tools/" <<Rectangle>> {
component [*.json\n14 tool schemas] as schemas <<artifact>>
component [*.json\n18 tool schemas] as schemas <<artifact>>
}
' Entry point dependencies
@@ -105,15 +128,19 @@ eval --> tools
chat --> session
' Core internal deps
session --> providers
session --> tools
session --> memory
memory --> storage
session --> safety
session --> sandbox
session --> edit
session --> web
session --> healthcheck
session --> mcp : optional
session --> toolsearch : optional
session --> registry : optional
registry --> providers
healthcheck --> metrics
mcp --> config
registry --> config
@@ -149,4 +176,22 @@ consoleserver --> config
consoleserver --> auth
collector --> broker
' API dependencies
serverspec --> openapi
consolespec --> openapi
serverspec --> apischemas
consolespec --> apischemas
server --> apidocs
server --> serverspec
consoleserver --> apidocs
consoleserver --> consolespec
' SDK dependencies
sdkserver --> sdkbase
sdkconsole --> sdkbase
sdkserver --> sdkevents
sdkconsole --> sdkevents
sdkserver --> apischemas : returns models
sdkconsole --> apischemas : returns models
@enduml
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b3f76042c8046560502fa3132351821d56e8526b13d38d7be8b839fcf3d5f648
size 373463
+117 -8
View File
@@ -41,7 +41,7 @@ class "WorkstreamTerminalUI" as WsTermUI {
}
class "WebUI" as WebUI {
- _event_queue: Queue
- _listeners: list[Queue]
- _approval_event: Event
- _plan_event: Event
- _ws_prompt_tokens: int
@@ -52,6 +52,8 @@ class "WebUI" as WebUI {
Enqueues JSON events for SSE.
Blocks on threading.Event for
approval/plan review.
SSE handlers bridge Queue to
async via run_in_executor().
--
server.py
}
@@ -63,15 +65,68 @@ class "NullUI" as NullUI {
eval.py
}
' LLMProvider Protocol
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ...) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ...) → CompletionResult
+ convert_tools(tools) → list[dict]
+ retryable_error_names: frozenset[str] {property}
--
core/providers/_protocol.py
}
class "OpenAIProvider" as OpenAIProv {
Model capability lookup table
(GPT-5.x, O-series, search)
Passthrough: messages already
in OpenAI format.
Search models: web_search_options
+ url_citation annotations.
Extended cache: 24h retention
for GPT-5.x (free).
--
core/providers/_openai.py
}
class "AnthropicProvider" as AnthropicProv {
Converts OpenAI messages to
Anthropic content blocks.
Adaptive + manual thinking.
Native web search via
web_search_20250305 server tool.
Auto prompt caching via
cache_control: ephemeral.
Lazy anthropic SDK import.
--
core/providers/_anthropic.py
}
' ModelCapabilities
class "ModelCapabilities" as ModelCaps <<frozen>> {
+ context_window: int
+ max_output_tokens: int
+ supports_temperature: bool
+ token_param: str
+ thinking_mode: str
+ supports_effort: bool
+ supports_web_search: bool
+ supports_tool_search: bool
+ supports_vision: bool
}
' ChatSession
class "ChatSession" as ChatSession {
- client: OpenAI
- client: Any
- provider: LLMProvider
- model: str
- ui: SessionUI
- messages: list[dict]
- _msg_tokens: list[int]
- _session_id: str
- _ws_id: str
- _mcp_client: MCPClientManager | None
- _tool_search: ToolSearchManager | None
- _registry: ModelRegistry | None
+ model_alias: str | None {property}
- _tools: list[dict]
@@ -82,7 +137,7 @@ class "ChatSession" as ChatSession {
--
+ send(user_input: str)
+ handle_command(command: str)
+ resume_session(session_id: str)
+ resume(ws_id: str)
- _save_config()
- _stream_response(stream) → dict
- _create_stream_with_retry(msgs) → Stream (+ fallback)
@@ -91,6 +146,12 @@ class "ChatSession" as ChatSession {
- _prepare_tool(tc) → item dict
- _prepare_mcp_tool(call_id, name, args) → item dict
- _exec_mcp_tool(item) → (call_id, output)
- _get_active_tools() → list[dict]
- _prepare_tool_search() → None
- _exec_tool_search(item) → (call_id, output)
- _on_mcp_tools_changed()
- _rebuild_tool_search()
+ close()
- _run_agent(messages, tools, ...) → str
- _compact_messages(auto: bool)
- _full_messages() → list[dict]
@@ -153,39 +214,77 @@ enum "WorkstreamState" as WsState {
' MCPClientManager
class "MCPClientManager" as MCPMgr {
- _sessions: dict[str, ClientSession]
- _per_server_tools: dict[str, list[dict]]
- _per_server_resources: dict[str, list[dict]]
- _per_server_prompts: dict[str, list[dict]]
- _tools: list[dict]
- _tool_map: dict[str, tuple]
- _resource_map: dict[str, tuple]
- _prompt_map: dict[str, tuple]
- _supports_list_changed: dict[str, bool]
- _listeners: list[Callable]
--
+ start()
+ get_tools() → list[dict]
+ get_resources() → list[dict]
+ get_prompts() → list[dict]
+ is_mcp_tool(name) → bool
+ call_tool_sync(name, args) → str
+ read_resource_sync(uri) → str
+ get_prompt_sync(name, args?) → list[dict]
+ refresh_sync(server?) → dict
+ add_listener(callback)
+ remove_listener(callback)
+ server_names: list[str] {property}
+ shutdown()
--
Background asyncio event loop
bridges async MCP SDK to
sync ChatSession dispatch.
Push + periodic + manual refresh.
Resources + prompts discovered
alongside tools at startup.
--
core/mcp_client.py
}
' ToolSearchManager
class "ToolSearchManager" as ToolSearchMgr {
- _all_tools: list[dict]
- _always_on: list[dict]
- _deferred: list[dict]
- _expanded: dict[str, None]
- _index: BM25Index
--
+ should_activate() → bool
+ get_visible_tools() → list[dict]
+ get_deferred_tools() → list[dict]
+ get_expanded_names() → list[str]
+ search(query, k) → list[dict]
+ expand_visible(names) → list[dict]
+ get_search_tool_definition() → dict
+ format_search_results(tools) → str
}
' ModelRegistry
class "ModelRegistry" as ModelReg {
- _models: dict[str, ModelConfig]
- _clients: dict[str, OpenAI]
- _clients: dict[str, Any]
- _providers: dict[str, LLMProvider]
- _client_lock: Lock
+ default: str
+ fallback: list[str]
+ agent_model: str | None
--
+ resolve(alias) → (client, model, config)
+ get_client(alias) → OpenAI
+ get_client(alias) → Any
+ get_provider(alias) → LLMProvider
+ has_alias(alias) → bool
+ list_aliases() → list[str]
+ shutdown()
--
Thread-safe lazy client creation.
Loaded by load_model_registry()
Thread-safe lazy client + provider
creation. Loaded by load_model_registry()
from CLI args + [models.*] config.
--
core/model_registry.py
@@ -193,6 +292,7 @@ class "ModelRegistry" as ModelReg {
class "ModelConfig" as ModelCfg <<frozen>> {
+ alias: str
+ provider: str
+ base_url: str
+ model: str
+ context_window: int
@@ -260,8 +360,13 @@ TerminalUI <|-- WsTermUI
SessionUI <|.. WebUI
SessionUI <|.. NullUI
LLMProvider <|.. OpenAIProv
LLMProvider <|.. AnthropicProv
ChatSession --> SessionUI : uses
ChatSession --> LLMProvider : delegates LLM calls
ChatSession --> MCPMgr : optional
ChatSession --o ToolSearchMgr : _tool_search
ChatSession --> ModelReg : optional
ChatSession <|-- HeadlessSession
@@ -273,6 +378,8 @@ Ws --> "1" WsState : has
WsMgr ..> ChatSession : creates via\nsession_factory(ui, model_alias)
ModelReg --> "*" ModelCfg : holds
ModelReg --> "*" LLMProvider : caches
LLMProvider --> ModelCaps : returns
ChatSession --> HealthMon : checks circuit
HealthMon --> "1" CircuitState : has
@@ -282,6 +389,8 @@ note bottom of ChatSession
Central engine: multi-turn LLM loop
with tool dispatch, agent sub-sessions,
context compaction, and memory persistence.
Provider-agnostic — delegates all LLM
communication to LLMProvider adapters.
core/session.py (~2700 lines)
end note
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fa1c94a0a7489cb9e6e6cad17f7e3577c458ec89d2a47fb2669fe935768837cf
size 276863
+25 -9
View File
@@ -8,7 +8,7 @@ skinparam sequenceLifeLineBackgroundColor #F5F5F5
participant "User /\nHTTP Client" as User
participant "ChatSession" as CS
participant "SessionUI" as UI
participant "OpenAI API\n(LLM)" as LLM
participant "LLMProvider\n(OpenAI / Anthropic)" as LLM
participant "Tool Executor\n(ThreadPool)" as TP
database "SQLite" as DB
@@ -18,7 +18,7 @@ User -> CS : send(user_input)
activate CS
CS -> CS : messages.append({role: "user", content: input})
CS -> DB : save_message(session_id, "user", input)
CS -> DB : save_message(ws_id, "user", input)
== LLM Call Loop ==
@@ -27,7 +27,7 @@ group loop [while tool_calls present]
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
CS -> LLM : client.chat.completions.create(\n model, messages, tools,\n stream=True, stream_options={include_usage})
CS -> LLM : provider.create_streaming(\n client, model, messages, tools, ...)\n (normalized StreamChunk iterator)
activate LLM
note right of CS
@@ -52,9 +52,19 @@ group loop [while tool_calls present]
CS -> UI : on_content_token(text)
else tool_call delta
CS -> CS : accumulate in tool_calls_acc
else info_delta present
CS -> UI : on_info(text)\n(e.g. server-side web search status)
end
end
note right of CS
**Cancellation checkpoint:**
_check_cancelled() runs per chunk.
If cancel_event is set, raises
GenerationCancelled — preserves
partial content, emits idle state.
end note
LLM --> CS : stream complete (usage stats)
deactivate LLM
@@ -63,8 +73,8 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> DB : save_message(session_id, "assistant", content)
CS -> DB : save_message(session_id, "tool_call", ...) ×N
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
== Tool Dispatch (if tool_calls) ==
@@ -110,14 +120,14 @@ group loop [while tool_calls present]
note right of TP
Parallel execution:
bash → Popen + line-by-line streaming
read_file → open().read()
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task/plan → _run_agent() sub-loop
math → sandboxed subprocess
web_fetch → httpx + LLM summarize
web_search → Tavily API
remember/recall/forget → SQLite
web_search → provider-native or Tavily fallback
memory/recall → SQLite
end note
note right of TP
@@ -134,7 +144,7 @@ group loop [while tool_calls present]
loop for each result
CS -> CS : messages.append({role: "tool", ...})
CS -> DB : save_message(session_id, "tool_result", ...)
CS -> DB : save_message(ws_id, "tool_result", ...)
end
opt user_feedback from approval
@@ -142,6 +152,12 @@ group loop [while tool_calls present]
end
note right of CS : Loop back for next LLM call
else GenerationCancelled
CS -> CS : Preserve partial content\nor roll back incomplete tools
CS -> UI : on_info("[Generation cancelled]")
CS -> UI : on_state_change("idle")
CS --> User : return (no re-raise)
end
end
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06fb9faa29c11395c6fc54ebddc79994b000dee78d56e0c13cb689fd6a82e37a
size 237255
+36 -28
View File
@@ -24,27 +24,30 @@ partition "Phase 1: Prepare" #E8F5E9 {
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (14 tools):**
┌─────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├─────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✓ Yes
│ man │ ✗ Auto-approve │
│ web_fetch │ ✓ Yes
│ web_search │ ✓ Yes
│ task │ ✓ Yes
plan │ ✓ Yes │
remember │ ✗ Auto-approve
recall │ ✗ Auto-approve │
forget │ ✗ Auto-approve │
├─────────────┼──────────────────┤
mcp__* │ ✓ Yes (external)
└─────────────┴──────────────────┘
**Dispatch table (17 tools):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
│ bash │ ✓ Yes │
│ read_file │ ✗ Auto-approve │
│ write_file │ ✓ Yes │
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ math │ ✗ Auto-approve
│ man │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve
│ web_search │ ✗ Auto-approve
│ tool_search │ ✗ Auto-approve
task │ ✓ Yes │
plan │ ✓ Yes
memory │ ✗ Auto-approve │
recall │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
read_resource │ ✓ Yes
│ use_prompt │ ✓ Yes │
├───────────────┼──────────────────┤
│ mcp__* │ ✓ Yes (external) │
└───────────────┴──────────────────┘
end note
:Build item dict:
@@ -66,8 +69,8 @@ partition "Phase 2: Approve" #FFF3E0 {
**TerminalUI**: Print headers/previews,
prompt [y/n/a, optional message]
If user chose "always":
Set ui.auto_approve = True
(auto-approve all future tools in this session)
Add pending tool names to auto_approve_tools
(auto-approve these tool types going forward)
**WebUI**: Enqueue approve_request,
block on _approval_event.wait()
**NullUI**: Return (True, None)
@@ -86,6 +89,8 @@ partition "Phase 2: Approve" #FFF3E0 {
}
partition "Phase 3: Execute" #E3F2FD {
:_check_cancelled();
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
if (single tool call?) then (yes)
:Execute sequentially:\nrun_one(items[0]);
else (multiple)
@@ -98,19 +103,22 @@ partition "Phase 3: Execute" #E3F2FD {
if item.denied → return denial message
else → item["execute"](item)
├─ _exec_bash: subprocess.run(["bash", script.sh])
├─ _exec_read_file: open().readlines()
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
├─ _exec_math: sandboxed subprocess
├─ _exec_man: man/info subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: Tavily API POST
├─ _exec_web_search: Tavily API POST (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_plan: _run_agent(AGENT_TOOLS, read-only)
├─ _exec_remember: SQLite INSERT OR REPLACE
├─ _exec_recall: SQLite FTS5/LIKE search
├─ _exec_forget: SQLite DELETE
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_memory: structured memory save/search/delete/list
├─ _exec_recall: conversation history FTS5 search
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
end note
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:13b2da77312e5abb44c81aabb7f0addccab31d9bc7e8ef2f1c3563ef985ed503
size 186941
+17 -2
View File
@@ -3,8 +3,9 @@
title Turnstone — Message Queue Protocol Types
skinparam classAttributeIconSize 0
skinparam packageStyle rectangle
skinparam packageBorderThickness 2
package "Inbound Messages (Client → Bridge)" #FFF3E0 {
package "Inbound Messages (Client → Bridge)" as InPkg #FFF3E0 {
abstract class "InboundMessage" as IM {
+ type: str
@@ -58,6 +59,8 @@ package "Inbound Messages (Client → Bridge)" #FFF3E0 {
+ auto_approve: bool = False
+ auto_approve_tools: list[str] = []
+ target_node: str = ""
+ initial_message: str = ""
+ skill: str = ""
}
class CloseWorkstreamMessage {
@@ -78,6 +81,12 @@ package "Inbound Messages (Client → Bridge)" #FFF3E0 {
type = "list_nodes"
}
class CancelMessage {
type = "cancel"
--
+ ws_id: str
}
IM <|-- SendMessage
IM <|-- ApproveMessage
IM <|-- PlanFeedbackMessage
@@ -87,9 +96,10 @@ package "Inbound Messages (Client → Bridge)" #FFF3E0 {
IM <|-- ListWorkstreamsMessage
IM <|-- HealthMessage
IM <|-- ListNodesMessage
IM <|-- CancelMessage
}
package "Outbound Events (Bridge → Client)" #E3F2FD {
package "Outbound Events (Bridge → Client)" as OutPkg #E3F2FD {
abstract class "OutboundEvent" as OE {
+ type: str
@@ -157,6 +167,8 @@ package "Outbound Events (Bridge → Client)" #E3F2FD {
+ context_window: int
+ pct: float
+ effort: str
+ cache_creation_tokens: int
+ cache_read_tokens: int
}
class StateChangeEvent {
type = "state_change"
@@ -164,6 +176,7 @@ package "Outbound Events (Bridge → Client)" #E3F2FD {
}
class TurnCompleteEvent {
type = "turn_complete"
+ content: str
}
}
@@ -235,6 +248,8 @@ package "Outbound Events (Bridge → Client)" #E3F2FD {
OE <|-- ClusterStateEvent
}
SendMessage -[hidden]down- OE
note bottom of IM
**Deserialization**: Strict type-dispatch via _INBOUND_REGISTRY.
Unknown type raises ValueError.
+7 -7
View File
@@ -18,20 +18,20 @@ note right of Redis : Shared queue — any bridge can pick up
BridgeA -> Redis : BLPOP [turnstone:inbound:nodeA,\n turnstone:inbound]
Redis --> BridgeA : SendMessage (from shared queue)
BridgeA -> ServerA : POST /api/workstreams/new\n{name:"", auto_approve:false}
BridgeA -> ServerA : POST /v1/api/workstreams/new\n{name:"", auto_approve:false}
ServerA --> BridgeA : {ws_id:"abc12345", name:"ws-abc1"}
BridgeA -> Redis : SET turnstone:ws:abc12345 "nodeA"
note right : Register workstream ownership
BridgeA -> ServerA : GET /api/events?ws_id=abc12345
BridgeA -> ServerA : GET /v1/api/events?ws_id=abc12345
note right : Start per-WS SSE thread
BridgeA -> Redis : PUBLISH turnstone:events:global\nWorkstreamCreatedEvent
BridgeA -> Redis : PUBLISH turnstone:events:cluster\nClusterStateEvent(ws_id, state:"idle", node_id:"nodeA")
BridgeA -> ServerA : POST /api/send\n{message:"...", ws_id:"abc12345"}
BridgeA -> ServerA : POST /v1/api/send\n{message:"...", ws_id:"abc12345"}
ServerA --> BridgeA : {status:"ok"}
BridgeA -> Redis : PUBLISH turnstone:events:abc12345\nAckEvent(status:"ok")
@@ -39,8 +39,8 @@ BridgeA -> Redis : PUBLISH turnstone:events:abc12345\nAckEvent(status:"ok")
... SSE events flow: content, tool_output_chunk, tool_result, status, state_change ...
BridgeA -> Redis : PUBLISH turnstone:events:abc12345\nContentEvent, ToolResultEvent, ...
BridgeA -> Redis : PUBLISH turnstone:events:global\nStateChangeEvent(state:"idle")
BridgeA -> Redis : PUBLISH turnstone:events:abc12345\nTurnCompleteEvent
BridgeA -> Redis : PUBLISH turnstone:events:global\nStateChangeEvent(state:"idle", content:"...")
BridgeA -> Redis : PUBLISH turnstone:events:abc12345\nTurnCompleteEvent(content:"...")
== Scenario B: Directed Message to Specific Node ==
@@ -79,7 +79,7 @@ note right of BridgeA
1. _ws_auto_approve[ws_id]? → auto
2. All tools in safe set? → auto
(read_file, search, man,
remember, recall, forget)
memory, recall)
3. Otherwise → manual approval
end note
@@ -93,7 +93,7 @@ note right : Response queue — bypasses inbound queue
BridgeA -> Redis : BLPOP turnstone:resp:req_xyz\n(spawned approval thread, timeout 300s)
Redis --> BridgeA : ApproveMessage
BridgeA -> ServerA : POST /api/approve\n{approved:true, ws_id:"abc12345"}
BridgeA -> ServerA : POST /v1/api/approve\n{approved:true, ws_id:"abc12345"}
== Heartbeat (continuous) ==
+6
View File
@@ -40,6 +40,12 @@ running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
thinking --> idle : cancel() called\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval/plan wait\n_emit_state("idle")
note right of thinking
**Emitted via:**
session._emit_state(state)
+137 -16
View File
@@ -1,13 +1,14 @@
@startuml
!theme plain
title Turnstone — Console Dashboard Data Collection
title Turnstone — Console Dashboard Data Flow
skinparam sequenceArrowThickness 1.5
participant "Browser" as Browser
participant "Console\nHTTP Server" as Server
participant "Console\nStarlette App" as Server
participant "ClusterCollector" as CC
collections "Redis" as Redis
participant "Node-A Bridge" as BridgeA
participant "Node-A\n(real server)" as NodeA
participant "Node-B\n(sim node)" as NodeB
@@ -67,17 +68,29 @@ note right of CC
from the cluster event channel.
end note
CC -> NodeA : GET /api/dashboard
CC -> NodeA : GET /v1/api/dashboard
activate NodeA
NodeA --> CC : {workstreams: [...],\naggregate: {total_tokens, ...}}
deactivate NodeA
CC -> NodeA : GET /health
activate NodeA
NodeA --> CC : {status:"ok", model:"...",\nworkstreams:{total, idle, ...}}
NodeA --> CC : {status:"ok", version:"0.3.0",\nmodel:"...", workstreams:{...}}
deactivate NodeA
CC -> CC : Diff old vs new workstream IDs
CC -> CC : Replace NodeSnapshot["nodeA"]\n.workstreams, .health, .aggregate
CC -> CC : _fanout(ws_created) for\nnewly appeared workstreams
CC -> CC : _fanout(ws_closed) for\nremoved workstreams
note right of CC
Poll-diff fanout ensures
browser SSE clients learn
about workstreams that
appeared without a real-time
cluster event (e.g. bridge
startup recovery).
end note
CC -x NodeB : (SKIPPED: sim:// URL)
@@ -85,40 +98,148 @@ deactivate CC
== Browser SSE Stream ==
Browser -> Server : GET /api/cluster/events
Browser -> Server : GET /v1/api/cluster/events
activate Server
Server -> CC : register_listener(queue)
note right : Per-client queue.Queue(maxsize=500)
Server -> CC : get_snapshot()
CC --> Server : ClusterSnapshot\n(full current state)
loop continuous
Server -> CC : register_listener(queue)
note right : Per-client queue.Queue(maxsize=500)\nSSE via EventSourceResponse + run_in_executor()
Server -> Browser : data: {"type":"snapshot",...}\n(full state as first SSE event)
loop continuous (incremental updates)
CC -> Server : event via listener queue\n(from any of the 3 threads)
Server -> Browser : data: {"type":"cluster_state",...}\n\n
end
alt timeout (5s no events)
Server -> Browser : : keepalive\n\n
alt keepalive (sse-starlette ping=5)
Server -> Browser : : ping\n\n
end
Browser -> Server : connection closed
Server -> CC : unregister_listener(queue)
deactivate Server
== Browser REST Requests ==
== Browser REST: Snapshot ==
Browser -> Server : GET /api/cluster/overview
Server -> CC : get_overview()
CC --> Server : {nodes: 10, workstreams: 47,\nstates: {running:5, thinking:3, ...},\naggregate: {total_tokens: 50000}}
Browser -> Server : GET /v1/api/cluster/snapshot
Server -> CC : get_snapshot()
CC --> Server : ClusterSnapshot\n(full current state)
Server --> Browser : JSON response
Browser -> Server : GET /api/cluster/nodes?sort=activity
== Browser REST Requests ==
Browser -> Server : GET /v1/api/cluster/overview
Server -> CC : get_overview()
CC --> Server : {nodes: 10, workstreams: 47,\nstates: {running:5, ...},\naggregate: {total_tokens: 50000},\nversion_drift: false, versions: ["0.3.0"]}
Server --> Browser : JSON response
Browser -> Server : GET /v1/api/cluster/nodes?sort=activity
Server -> CC : get_nodes(sort_by="activity")
CC --> Server : {nodes: [...], total: 10}
Server --> Browser : JSON response
Browser -> Server : GET /api/cluster/workstreams\n?state=running&node=sim-0003
Browser -> Server : GET /v1/api/cluster/workstreams\n?state=running&node=sim-0003
Server -> CC : get_workstreams(state="running",\nnode="sim-0003")
CC --> Server : {workstreams: [...], total: 5,\npage: 1, per_page: 50, pages: 1}
Server --> Browser : JSON response
== Workstream Creation (via MQ) ==
Browser -> Server : POST /v1/api/cluster/workstreams/new\n{node_id:"nodeA", name:"new-task"}
activate Server #FFECB3
Server -> CC : _pick_best_node() or\nget_node_detail(node_id)
CC --> Server : node validated
Server -> Server : Build CreateWorkstreamMessage\n{target_node:"nodeA", name:"new-task"}
Server -> Redis : RPUSH turnstone:inbound:nodeA\n(directed queue)
Server --> Browser : {status:"ok", correlation_id:"abc",\ntarget_node:"nodeA"}
deactivate Server
note right of Redis
Bridge on Node-A picks up the
message from its directed queue,
POSTs to /v1/api/workstreams/new,
registers ownership, publishes
ws_created to cluster channel.
end note
Redis --> BridgeA : BLPOP turnstone:inbound:nodeA
activate BridgeA
BridgeA -> NodeA : POST /v1/api/workstreams/new\n{name:"new-task"}
NodeA --> BridgeA : {ws_id:"ws789", name:"new-task"}
BridgeA -> Redis : SET turnstone:ws:ws789 = nodeA
BridgeA -> Redis : PUBLISH turnstone:events:cluster\n{type:"ws_created", ws_id:"ws789",\nnode_id:"nodeA", name:"new-task"}
deactivate BridgeA
Redis --> CC : ws_created event
CC -> CC : Add workstream to\nNodeSnapshot["nodeA"]
CC -> CC : _fanout(event)
Server -> Browser : SSE: data: {"type":"ws_created",...}
== Reverse Proxy (server UI through console port) ==
Browser -> Server : GET /node/nodeA/
activate Server #FFF9C4
Server -> CC : get_node_detail("nodeA")\n→ server_url = "http://10.0.1.1:8080"
Server -> NodeA : GET http://10.0.1.1:8080/\n(via httpx.AsyncClient)
activate NodeA
NodeA --> Server : index.html
deactivate NodeA
Server -> Server : Rewrite static paths:\nhref="/static/" → "/node/nodeA/static/"\nInject console-return banner\nafter <body>
Server --> Browser : Rewritten HTML
deactivate Server
Browser -> Server : GET /node/nodeA/static/app.js
activate Server #FFF9C4
Server -> NodeA : GET http://10.0.1.1:8080/static/app.js
activate NodeA
NodeA --> Server : app.js
deactivate NodeA
Server -> Server : Prepend JS proxy shim:\nOverride fetch() and EventSource()\nto prepend "/node/nodeA" prefix
Server --> Browser : Shimmed app.js
deactivate Server
note right of Browser
All fetch("/v1/api/send") calls in the
server UI now become fetch("/node/nodeA/v1/api/send"),
routed through the console proxy.
end note
Browser -> Server : GET /node/nodeA/v1/api/events?ws_id=ws789
activate Server #FFF9C4
Server -> NodeA : GET http://10.0.1.1:8080/v1/api/events?ws_id=ws789\n(SSE stream via httpx.AsyncClient timeout=None)
activate NodeA
loop SSE streaming
NodeA --> Server : data: {"type":"content","text":"..."}\n\n
Server --> Browser : data: {"type":"content","text":"..."}\n\n
end
deactivate NodeA
deactivate Server
Browser -> Server : POST /node/nodeA/v1/api/send\n{message:"hello", ws_id:"ws789"}
activate Server #FFF9C4
Server -> NodeA : POST http://10.0.1.1:8080/v1/api/send\n(body forwarded)
activate NodeA
NodeA --> Server : {status:"ok"}
deactivate NodeA
Server --> Browser : {status:"ok"}
deactivate Server
@enduml
+4 -4
View File
@@ -83,12 +83,12 @@ mqclient --> redis : Redis protocol\nport 6379
server --> redis : Redis protocol\n(6379)
server --> llm_api : OpenAI API\n(HTTPS/HTTP)
bridge --> server : HTTP REST\n(POST /api/send, etc.)
bridge <-- server : SSE\n(GET /api/events)
bridge --> server : HTTP REST\n(POST /v1/api/send, etc.)
bridge <-- server : SSE\n(GET /v1/api/events)
bridge --> redis : Redis protocol\n(queues + pubsub)
console --> redis : Redis PUBSUB\n(cluster channel)
console --> server : HTTP polling\n(GET /api/dashboard)
console --> redis : Redis PUBSUB + LIST\n(cluster events,\nws creation commands)
console --> server : HTTP polling + proxy\n(GET /v1/api/dashboard,\nproxy /node/{id}/*)
sim --> redis : Redis protocol\n(queues + pubsub + keys)
+158
View File
@@ -0,0 +1,158 @@
@startuml
!theme plain
title Turnstone — Client SDK Architecture
skinparam class {
BackgroundColor<<async>> #C8E6C9
BackgroundColor<<sync>> #B8D4E3
BackgroundColor<<event>> #FFE0B2
BackgroundColor<<type>> #F0F4C3
BackgroundColor<<ts>> #E1BEE7
}
skinparam packageBorderColor #888888
skinparam ArrowColor #555555
' Python SDK
package "turnstone/sdk/ (Python)" {
abstract class _BaseClient <<async>> {
- _client: httpx.AsyncClient
- _owns_client: bool
+ _request(method, path, ...) → T
+ _stream_sse(path, ...) → AsyncIterator
+ aclose()
}
class AsyncTurnstoneServer <<async>> {
+ list_workstreams()
+ dashboard()
+ create_workstream()
+ close_workstream()
+ send(message, ws_id)
+ approve()
+ plan_feedback()
+ command()
+ cancel(ws_id)
+ stream_events(ws_id)
+ stream_global_events()
+ send_and_wait()
+ list_saved_workstreams()
+ login() / logout()
+ health()
}
class AsyncTurnstoneConsole <<async>> {
+ overview()
+ nodes()
+ workstreams()
+ node_detail()
+ snapshot()
+ create_workstream()
+ stream_cluster_events()
+ login() / logout()
+ health()
}
class TurnstoneServer <<sync>> {
- _async: AsyncTurnstoneServer
- _runner: _SyncRunner
.. delegates all methods ..
+ __enter__ / __exit__
}
class TurnstoneConsole <<sync>> {
- _async: AsyncTurnstoneConsole
- _runner: _SyncRunner
.. delegates all methods ..
+ __enter__ / __exit__
}
class _SyncRunner <<sync>> {
- _loop: EventLoop
- _thread: Thread
+ run(coro) → T
+ run_iter(async_gen) → Iterator
+ close()
}
class TurnResult <<type>> {
+ ws_id: str
+ content_parts: list[str]
+ reasoning_parts: list[str]
+ tool_results: list
+ errors: list[str]
+ timed_out: bool
--
+ content: str
+ reasoning: str
+ ok: bool
}
class ServerEvent <<event>> {
+ type: str
+ ws_id: str
+ from_dict() → ServerEvent
}
class ClusterEvent <<event>> {
+ type: str
+ from_dict() → ClusterEvent
}
_BaseClient <|-- AsyncTurnstoneServer
_BaseClient <|-- AsyncTurnstoneConsole
TurnstoneServer --> AsyncTurnstoneServer : wraps
TurnstoneServer --> _SyncRunner : uses
TurnstoneConsole --> AsyncTurnstoneConsole : wraps
TurnstoneConsole --> _SyncRunner : uses
AsyncTurnstoneServer ..> TurnResult : returns
AsyncTurnstoneServer ..> ServerEvent : yields
AsyncTurnstoneConsole ..> ClusterEvent : yields
}
' TypeScript SDK
package "sdk/typescript/ (TypeScript)" {
class "BaseClient" as TSBase <<ts>> {
# baseUrl: string
# token: string
# fetchFn: fetch
# request<T>()
# streamSSE<T>()
}
class "TurnstoneServer" as TSServer <<ts>> {
+ listWorkstreams()
+ send()
+ streamEvents()
+ sendAndWait()
...
}
class "TurnstoneConsole" as TSConsole <<ts>> {
+ overview()
+ nodes()
+ snapshot()
+ clusterEvents()
...
}
TSBase <|-- TSServer
TSBase <|-- TSConsole
}
' External connections
class "turnstone-server :8080" as Server <<artifact>>
class "turnstone-console :8081" as Console <<artifact>>
AsyncTurnstoneServer --> Server : httpx REST + SSE
AsyncTurnstoneConsole --> Console : httpx REST + SSE
TSServer --> Server : fetch REST + SSE
TSConsole --> Console : fetch REST + SSE
note right of AsyncTurnstoneServer
Returns Pydantic models from
turnstone.api.server_schemas
(no type duplication)
end note
@enduml
+169
View File
@@ -0,0 +1,169 @@
@startuml
!theme plain
title Turnstone — Storage Architecture
skinparam class {
BackgroundColor<<protocol>> #E8EAF6
BackgroundColor<<sqlite>> #C8E6C9
BackgroundColor<<postgres>> #B3E5FC
BackgroundColor<<facade>> #FFF9C4
BackgroundColor<<migration>> #FFE0B2
BackgroundColor<<schema>> #F3E5F5
}
' -- Protocol --
interface "StorageBackend" as SB <<protocol>> {
+save_message(ws_id, role, content, ...)
+load_messages(ws_id) → list[dict]
+register_workstream(ws_id, node_id, name, state)
+update_workstream_state(ws_id, state)
+update_workstream_name(ws_id, name)
+set_workstream_alias(ws_id, alias) → bool
+update_workstream_title(ws_id, title)
+resolve_workstream(alias_or_id) → str | None
+delete_workstream(ws_id) → bool
+prune_workstreams(retention_days) → (int, int)
+list_workstreams(node_id, limit) → list
+save_workstream_config(ws_id, config)
+load_workstream_config(ws_id) → dict
+kv_get(key) → str | None
+kv_set(key, value) → str | None
+kv_delete(key) → bool
+kv_list() → list[(str, str)]
+kv_search(query) → list[(str, str)]
+search_history(query, limit) → list
+search_history_recent(limit) → list
+create_user(user_id, username, display_name, pw_hash)
+get_user(user_id) / get_user_by_username(username)
+list_users() / delete_user(user_id)
+create_api_token(...) / get_api_token_by_hash(hash)
+list_api_tokens(user_id) / delete_api_token(id)
+close()
}
' -- Backends --
class "SQLiteBackend" as SQLite <<sqlite>> {
-_engine: sa.Engine
-_fts5_available: bool
+__init__(path: str)
--
FTS5 full-text search
Default pool, check_same_thread=False
}
class "PostgreSQLBackend" as PG <<postgres>> {
-_engine: sa.Engine
+__init__(url: str, pool_size: int)
--
tsvector + ILIKE search
Connection pooling
}
' -- Schema --
class "_schema.py" as Schema <<schema>> {
+metadata: MetaData
+memories: Table
+conversations: Table
+workstreams: Table (node_id, alias, title,\n state, skill_id)
+workstream_config: Table
+users: Table (username, password_hash)
+api_tokens: Table (token_hash, scopes)
+channel_users: Table (channel_type)
+scheduled_tasks: Table (..., skill)
--
SQLAlchemy Core
Single source of truth
}
' -- Migration --
class "_migrate.py" as Migrate <<migration>> {
+run_migrations(storage, backend)
-_bootstrap_existing_sqlite()
--
Programmatic Alembic
Auto-bootstrap existing DBs
}
class "migrations/" as Versions <<migration>> {
001_initial_schema.py
002_user_identity.py
}
' -- Registry --
class "_registry.py" as Registry {
-_storage: StorageBackend | None
+init_storage(backend, path, url) → StorageBackend
+get_storage() → StorageBackend
+reset_storage()
--
Auto-initializes SQLite
if not configured
}
' -- Facade --
class "memory.py" as Facade <<facade>> {
+save_message()
+load_messages()
+register_workstream()
+update_workstream_state()
+save_workstream_config()
+save_memory() / delete_memory()
+search_memories()
+... (all delegated functions)
--
Thin delegation to
get_storage()
Silent failure behavior
}
' -- Consumers --
class "session.py\nChatSession" as Session {
}
class "server.py\nWeb UI" as Server {
}
class "cli.py\nTerminal" as CLI {
}
' -- Relationships --
SQLite ..|> SB
PG ..|> SB
SQLite --> Schema : uses
PG --> Schema : uses
Registry --> SB : creates
Registry --> Migrate : calls
Migrate --> Versions : applies
Migrate --> Schema : references
Facade --> Registry : get_storage()
Session --> Facade : imports
Server --> Facade : imports
CLI --> Facade : imports
' -- Config --
note right of Registry
[database]
backend = "sqlite" | "postgresql"
url = "postgresql+psycopg://..."
path = ".turnstone.db"
pool_size = 5
end note
note bottom of SQLite
Default backend.
Zero-config for
single-node / dev.
end note
note bottom of PG
Production backend.
Multi-node / Docker
default.
end note
@enduml
+179
View File
@@ -0,0 +1,179 @@
@startuml
!theme plain
title Turnstone — Authentication Architecture
skinparam class {
BackgroundColor<<core>> #E8EAF6
BackgroundColor<<jwt>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<endpoint>> #FFE0B2
BackgroundColor<<scope>> #F3E5F5
}
' -- Core Auth --
class "AuthConfig" as AC <<core>> {
+enabled: bool
+tokens: dict[str, str]
+check(token) → role | None
--
Static config-file tokens
hmac.compare_digest
}
class "AuthResult" as AR <<core>> {
+user_id: str
+scopes: frozenset[str]
+token_source: str
+has_scope(scope) → bool
}
class "check_request()" as CR <<core>> {
auth_config, method, path,
auth_header, cookie_header,
jwt_secret, storage
→ (allowed, status, msg, AuthResult)
--
1. Auth disabled → allow
2. Public path → allow
3. Extract Bearer / cookie
4. Detect token type
5. Validate → AuthResult
6. Check scope vs path
}
' -- Token Types --
class "JWT (HS256)" as JWT <<jwt>> {
sub: user_id
scopes: "read,write,approve"
src: "password" | "database"
iat, exp (24h default)
--
Detected by: contains "."
Validated locally
No DB call
}
class "API Token" as AT <<jwt>> {
Format: ts_ + 64 hex
Stored: SHA-256 hash
--
Detected by: starts with "ts_"
Lookup by hash in DB
Expiry check
}
class "Config Token" as CT <<core>> {
Raw value in memory
Role: "read" | "full"
--
Detected by: fallback
hmac.compare_digest
No DB needed
}
' -- Scopes --
class "Scope Hierarchy" as SH <<scope>> {
read: {read}
write: {read, write}
approve: {read, write, approve}
--
GET → read
POST write paths → write
POST /api/approve → approve
/api/admin/* → approve
}
' -- Storage --
class "users" as UT <<storage>> {
user_id (PK)
username (unique)
display_name
password_hash (bcrypt)
created
}
class "api_tokens" as TT <<storage>> {
token_id (PK)
token_hash (SHA-256, unique)
token_prefix
user_id → users
name, scopes
created, expires
}
' -- Endpoints --
class "POST /api/auth/login" as Login <<endpoint>> {
{username, password}
OR {token: "ts_xxx"}
→ {jwt, role, scopes, user_id}
--
Sets HttpOnly cookie
}
class "GET /api/auth/status" as Status <<endpoint>> {
→ {auth_enabled, has_users,
setup_required}
--
Public (no auth)
Drives UI setup wizard
}
class "POST /api/auth/setup" as Setup <<endpoint>> {
{username, display_name, password}
→ {jwt, user_id, scopes}
--
Public (no auth)
Only when zero users exist
Returns 409 if already set up
}
class "Admin API (Console)" as Admin <<endpoint>> {
POST/GET/DELETE users
POST/GET tokens
DELETE tokens/{id}
--
Requires approve scope
}
' -- Relationships --
CR --> AC : config tokens
CR --> JWT : validate
CR --> AT : hash lookup
CR --> CT : hmac check
CR --> AR : returns
CR --> SH : checks
Login --> JWT : issues
Login --> UT : verify password
Login --> TT : verify API token
Setup --> UT : create first user
Setup --> JWT : issues
AT --> TT : lookup by hash
Admin --> UT : CRUD
Admin --> TT : CRUD
AR --> SH : scopes from
JWT ..> AR : produces
AT ..> AR : produces
CT ..> AR : produces
note right of CR
**Middleware Flow**
AuthMiddleware on every request:
1. Extract token from header/cookie
2. Detect type (JWT / ts_ / config)
3. Validate → AuthResult
4. Set ctx_user_id for logging
5. Store auth_result in scope state
end note
note bottom of SH
**Console** owns admin endpoints
**Server** validates JWT + config only
Both share JWT signing secret
end note
@enduml
+263
View File
@@ -0,0 +1,263 @@
@startuml
!theme plain
title Turnstone — Channel Integration Architecture
skinparam class {
BackgroundColor<<platform>> #E1BEE7
BackgroundColor<<service>> #E8EAF6
BackgroundColor<<mq>> #FFCDD2
BackgroundColor<<bridge>> #C8E6C9
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
}
' -- External Platforms --
class "Discord" as Discord <<platform>> {
Gateway WebSocket (v10)
Message events
Interaction callbacks (buttons)
Thread-per-workstream
--
discord.py 2.x
asyncio event loop
}
class "Slack (future)" as Slack <<platform>> {
Socket Mode / Events API
Block Kit messages
--
Planned integration
}
class "Teams (future)" as Teams <<platform>> {
Bot Framework
Adaptive Cards
--
Planned integration
}
' -- Channel Service --
class "turnstone-channel" as ChannelService <<service>> {
entry point: turnstone-channel
--
One process per platform
asyncio event loop
Structured logging (structlog)
--log-level, --log-format
--
POST /v1/api/notify (HTTP)
GET /health
}
class "DiscordBot" as Bot <<service>> {
+on_message(msg)
+on_interaction(interaction)
+send(channel_id, content)
+send_notification(channel_id, content, ws_id)
+run(token)
--
discord.py Client
Receives message events
Sends replies + embeds
Creates threads for workstreams
Renders approval buttons
escape_mentions() on send
--
_notify_ws_map: msg_id → (ws_id, user_id)
_notify_reply_channels: ws_id → (dm, user_id)
}
class "ChannelRouter" as Router <<service>> {
+resolve_route(platform, channel_id)
→ ws_id | None
+register_route(channel_id, ws_id)
+resolve_identity(platform, platform_user_id)
→ user_id | None
--
Maps channels → workstreams
Maps platform users → turnstone users
Caches routes in memory
}
class "AsyncRedisBroker" as Broker <<service>> {
+push_inbound(msg)
+subscribe(ws_id) → AsyncIterator
+subscribe_global() → AsyncIterator
+push_response(correlation_id, msg)
--
redis.asyncio client
Pub/sub + queue operations
}
' -- Redis MQ --
class "Redis MQ" as Redis <<mq>> {
turnstone:inbound (LIST)
turnstone:events:{ws_id} (PUBSUB)
turnstone:events:global (PUBSUB)
turnstone:resp:{corr_id} (LIST)
--
Shared message bus
Same queues as bridge protocol
}
' -- Bridge + Server --
class "turnstone-bridge" as Bridge <<bridge>> {
BLPOP turnstone:inbound
Drive server via HTTP
Relay SSE → Redis pub/sub
--
Owns workstream lifecycle
Auto-approve / manual approve
}
class "turnstone-server" as Server <<server>> {
POST /v1/api/send
POST /v1/api/approve
POST /v1/api/workstreams/new
GET /v1/api/events?ws_id=
--
LLM execution + tool use
SSE event stream
--
notify tool: _exec_notify()
ServiceTokenManager (JWT)
}
' -- Storage --
class "channel_users" as CU <<storage>> {
channel_user_id (PK)
platform: "discord" | "slack"
platform_user_id
user_id → users
linked_at
--
/link command creates row
Resolved on each inbound message
}
class "channel_routes" as CR <<storage>> {
channel_type (PK)
channel_id (PK)
ws_id
node_id
created
--
Maps platform channels
to turnstone workstreams
}
class "services" as SVC <<storage>> {
service_type (PK)
service_id (PK)
url
last_heartbeat
created
--
Heartbeat every 30s
Stale after 120s
ON CONFLICT DO UPDATE
}
' -- Relationships --
Discord --> Bot : gateway\nevents
Bot --> Router : on_message\non_interaction
Router --> Broker : SendMessage\nApproveMessage
Router --> CU : resolve identity
Router --> CR : resolve / register route
Broker --> Redis : RPUSH inbound\nRPUSH resp:{id}
Redis --> Bridge : BLPOP inbound
Bridge --> Server : HTTP API
Server --> Bridge : SSE events
Bridge --> Redis : PUBLISH events:{ws_id}\nPUBLISH events:global
Redis --> Broker : SUBSCRIBE events:{ws_id}
Broker --> Bot : event stream
Bot --> Discord : reply / embed\nbutton callback
Slack .[hidden]. Discord
Teams .[hidden]. Slack
ChannelService --> Bot : creates + runs
ChannelService --> Router : creates
ChannelService --> Broker : creates
ChannelService --> SVC : register / heartbeat /\nderegister
' -- Notification path (direct HTTP, bypasses MQ) --
Server --> ChannelService : POST /v1/api/notify\n(JWT: aud=turnstone-channel)
Server --> SVC : list_services("channel",\nmax_age_seconds=120)
' -- Notes --
note right of Bot
**Inbound Flow**
1. Discord message arrives via gateway
2. Bot.on_message() fires
3. ChannelRouter resolves channel → ws_id
(or creates new workstream)
4. ChannelRouter resolves platform user → user_id
via channel_users table
5. Broker.push_inbound(SendMessage)
6. Bridge pops from Redis, drives server
**Workstream Resume (evicted workstreams)**
1. Stale route detected (no MQ owner)
2. Existing ws_id reused directly from route
3. CreateWorkstreamMessage sent with
resume_ws=<ws_id>
4. Server resumes atomically during creation
5. Bridge emits WorkstreamResumedEvent → thread
end note
note right of Broker
**Outbound Flow**
1. Server emits SSE events
2. Bridge relays to Redis events:{ws_id}
3. Broker.subscribe(ws_id) yields events
4. Bot formats and sends to Discord thread
end note
note bottom of CR
**Approval Flow**
1. ApprovalRequestEvent arrives via events:{ws_id}
2. Bot renders Discord buttons (Approve / Deny)
3. User clicks button → on_interaction()
4. Router builds ApproveMessage
5. Broker.push_response(correlation_id, msg)
6. Bridge pops from resp:{id}, calls POST /api/approve
end note
note bottom of CU
**Identity Linking**
1. User runs /link in Discord
2. Bot opens modal requesting API token
3. User submits ts_... API token
4. Bot validates token against storage
5. On success, inserts channel_users row
6. Subsequent messages carry user_id
7. AuthResult scopes applied by server
end note
note bottom of SVC
**Notification Flow** (direct HTTP, bypasses MQ)
1. LLM calls notify tool → _prepare_notify()
2. _exec_notify() checks rate limit (5/turn)
3. Queries services table for healthy gateways
4. Mints JWT (aud: turnstone-channel) via
ServiceTokenManager
5. POSTs to first healthy gateway (incl. ws_id)
6. Gateway validates JWT, resolves target
7. adapter.send_notification() → Discord API
(tracks msg_id → ws_id for reply routing)
8. On failure: retry up to 3× (1s, 3s backoff)
9. SSRF: only http(s) URLs allowed
**Bidirectional DM Replies**
1. User replies to notification DM
2. Bot looks up ws_id from _notify_ws_map
3. Verifies author == notification recipient
4. Routes reply via router.send_message()
5. Response forwarded to DM on TurnCompleteEvent
6. Response tracked for multi-turn conversation
end note
@enduml
+151
View File
@@ -0,0 +1,151 @@
@startuml
!theme plain
title Turnstone — Notification Delivery Flow
skinparam participant {
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<service>> #E8EAF6
BackgroundColor<<platform>> #E1BEE7
}
participant "ChatSession\n(turnstone-server)" as Session <<server>>
participant "StorageBackend" as Storage <<storage>>
participant "ServiceTokenManager" as STM <<server>>
participant "Channel Gateway\n(_http.py)" as Gateway <<service>>
participant "ChannelAdapter\n(Discord bot)" as Adapter <<service>>
participant "Discord API" as Discord <<platform>>
== Prepare Phase ==
Session -> Session : _prepare_notify(call_id, args)
note right
Validates:
- message (required, ≤2000 chars)
- target: username OR channel_type+channel_id
- no ambiguous targeting (both set)
- partial targeting errors
end note
== Execute Phase ==
Session -> Session : _exec_notify(item)
Session -> Session : check rate limit\n(≥5 per turn?)
alt rate limit exceeded
Session --> Session : "Error: rate limit exceeded"
end
loop up to 3 attempts (retry delays: 1s, 3s)
Session -> Storage : list_services("channel",\nmax_age_seconds=120)
Storage --> Session : services[] (sorted by\nlast_heartbeat DESC)
alt no healthy services
Session -> Session : log.warning("notify.no_services")
Session -> Session : sleep(delay)
else services available
Session -> STM : bearer_header
note right
Lazy-init ServiceTokenManager
aud: turnstone-channel
scope: write
Auto-rotates 1h JWTs
end note
STM --> Session : Authorization: Bearer <jwt>
loop for each gateway (first-healthy)
Session -> Session : SSRF check:\nurl.startswith("http://"|"https://")
Session -> Gateway : POST /v1/api/notify\n+ Authorization header
Gateway -> Gateway : _check_auth()\nvalidate JWT (aud=turnstone-channel)\nor static token
alt auth failed
Gateway --> Session : 401 Unauthorized
else auth ok
alt username target
Gateway -> Storage : get_user_by_username()
Storage --> Gateway : user
Gateway -> Storage : list_channel_users_by_user()
Storage --> Gateway : linked channels
else direct target
Gateway -> Gateway : use channel_type + channel_id
end
Gateway -> Adapter : send(channel_id, content)
note right
escape_mentions() applied
Chunked for 2000-char limit
end note
Adapter -> Discord : POST message
Discord --> Adapter : message_id
Adapter --> Gateway : message_id
Gateway --> Session : 200 {results: [{status: "sent"}]}
Session -> Session : _notify_count += 1
Session --> Session : "Notification sent successfully"
note right : Return — no further\ngateways tried
end
end
alt all gateways failed
Session -> Session : log.warning(\n"notify.all_gateways_failed")
Session -> Session : sleep(delay)
end
end
end
alt all retries exhausted
Session -> Session : log.warning("notify.delivery_failed")
Session --> Session : "Error: notification delivery failed"
end
== Bidirectional Reply (User responds to notification DM) ==
Discord -> Adapter : user replies to\nnotification message
Adapter -> Adapter : lookup message_id\nin _notify_ws_map
note right
Maps message_id →
(ws_id, target_user_id)
Atomic pop prevents TOCTOU
end note
alt message not tracked
Adapter -> Discord : "This notification\nis no longer active."
else tracked
Adapter -> Adapter : verify author ==\ntarget_user_id
Adapter -> Adapter : resolve_user()\n(unlinked → drop)
Adapter -> Adapter : router.send_message(ws_id, content)
note right
Routes reply via MQ to
the originating workstream.
Registers DM channel in
_notify_reply_channels[ws_id]
end note
... workstream processes reply ...
Adapter <- Adapter : TurnCompleteEvent\n(with content)
Adapter -> Discord : forward response to DM
Adapter -> Adapter : track response message\nfor multi-turn replies
note right
Response message_id added
to _notify_ws_map — user can
reply again indefinitely
end note
end
== Service Registry (Background) ==
note over Gateway, Storage
**Heartbeat Lifecycle**
1. Gateway startup: register_service("channel", id, url)
2. Every 30s: heartbeat_service("channel", id)
3. Shutdown: deregister_service("channel", id)
4. Stale after 120s (4 missed heartbeats)
end note
@enduml
+166
View File
@@ -0,0 +1,166 @@
@startuml
!theme plain
title Turnstone — Watch Tool Architecture
skinparam participant {
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<ui>> #E8EAF6
}
participant "ChatSession\n(session.py)" as Session <<session>>
participant "WatchRunner\n(watch.py)" as Runner <<server>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
== Create Phase ==
Session -> Session : _prepare_watch(action="create")
note right
Validates:
- command via is_command_blocked()
- poll_every → parse_duration()
- stop_on → validate_condition()
- max watches limit (5)
- duplicate name check
needs_approval = True
end note
Session -> Storage : create_watch(watch_id, ws_id,\nnode_id, command, interval,\nstop_on, max_polls, next_poll)
Session --> UI : tool_result:\n"Watch 'pr-review' created"
== Poll Phase (WatchRunner daemon, every 15s) ==
Runner -> Storage : list_due_watches(now)
Storage --> Runner : due_watches[]
note right
Filters:
active=1 AND
next_poll <= now AND
node_id matches
end note
loop for each due watch
Runner -> Runner : is_command_blocked()?
alt blocked
Runner -> Storage : update_watch(active=False)
else safe
Runner -> Runner : subprocess.run(command)
note right
timeout = tool_timeout
start_new_session = True
output truncated at 64KB
end note
Runner -> Runner : evaluate_condition(\nstop_on, output,\nexit_code, prev_output)
note right
**Variables:**
output, data, exit_code,
prev_output, changed
**Safe builtins only:**
len, str, int, sorted, ...
No import/open/exec/eval
**stop_on=None:**
fires on change (skip 1st poll)
end note
alt condition fired OR max_polls reached
Runner -> Storage : update_watch(\npoll_count++,\nlast_output, active=False)
Runner -> Runner : format_watch_message()
Runner -> Runner : _dispatch_result(ws_id, msg)
else not fired
Runner -> Storage : update_watch(\npoll_count++,\nlast_output, next_poll)
end
end
end
== Dispatch Phase ==
note over Runner, Session
**Three dispatch paths:**
end note
alt Path A: workstream active + idle
Runner -> Session : dispatch_fn(message)\n→ _watch_pending.put()
Session -> Session : _dispatch_pending_watch()\n→ self.send(message)
Session -> UI : SSE: thinking, content,\ntool calls...
note right
Watch result appears as
synthetic user message.
Model sees it and responds.
Depth guard: max 5 chains.
end note
else Path B: workstream active + busy
Runner -> Session : dispatch_fn(message)\n→ _watch_pending.put()
note right
Queued. Dispatched when
current send() reaches IDLE.
end note
else Path C: workstream evicted
Runner -> Runner : restore_fn(ws_id)
note right
1. mgr.create() — may evict
another idle workstream
2. session.resume(ws_id)
3. set_watch_runner()
4. register new dispatch_fn
end note
Runner -> Session : restored dispatch_fn(message)
end
== Cancel / List ==
Session -> Storage : list_watches_for_ws(ws_id)
note right : action="list" (auto-approve)
Session -> Storage : update_watch(active=False)
note right : action="cancel" (auto-approve)
== Server Lifecycle ==
note over Runner, Storage
**Startup:**
1. WatchRunner created in main() with storage + node_id
2. restore_fn closure captures WorkstreamManager
3. Initial workstream: session.set_watch_runner(runner)
4. _lifespan(): runner.start() — daemon thread begins
**New workstream:**
session.set_watch_runner(runner) in create_workstream()
→ registers dispatch_fn for ws_id
**Eviction / close:**
session.close() → runner.remove_dispatch_fn(ws_id)
Watches remain active in DB — WatchRunner uses restore_fn
**Restart recovery:**
Overdue watches fire ONE immediate poll
next_poll updated to now + interval
Normal cadence resumes
**Shutdown:**
_lifespan(): runner.stop() — joins thread
end note
== REST API ==
note over UI, Storage
**GET /v1/api/watches[?ws_id=X]**
List active watches (for node or workstream)
**POST /v1/api/watches/{watch_id}/cancel**
Cancel a watch (sets active=False)
Both require write scope
end note
@enduml
@@ -0,0 +1,100 @@
@startuml
!theme plain
skinparam backgroundColor #FFFFFF
skinparam defaultFontName "IBM Plex Mono"
skinparam componentStyle rectangle
title Turnstone Governance Architecture
package "Auth Flow" {
[Login/Token Auth] as auth
[_load_user_permissions()] as perms
[_permissions_to_scopes()] as scopes
[create_jwt()] as jwt
}
package "Middleware" {
[AuthMiddleware\n(scope check)] as mw
[require_permission()\n(granular check)] as rp
}
package "Governance Storage" {
database "roles" as roles_db
database "user_roles" as ur_db
database "orgs" as orgs_db
database "tool_policies" as tp_db
database "prompt_templates\n(skills)" as pt_db
database "usage_events" as ue_db
database "audit_events" as ae_db
database "skills" as wt_db
}
package "Runtime Enforcement" {
[evaluate_tool_policies_batch()] as eval
[WebUI.approve_tools()] as approve
[record_usage_event()\n+cache_creation/read_tokens] as usage
[record_audit()] as audit
}
package "Template Runtime" {
[_load_templates()] as tload
[_render_template()\n{{model}}, {{ws_id}}, {{node_id}}] as trender
[_init_system_messages()] as tsys
[set_template() / /template] as tset
}
package "Skill Runtime" {
[resolve_skill()] as wtr
[apply settings\n(model, budget, prompt)] as wta
[budget gate\n(session.send)] as wtb
}
package "Console UI" {
[Admin Panel\n10 tabs] as ui
[governance.js] as govjs
[sessionStorage\npermissions] as ss
}
auth --> perms : user_id
perms --> roles_db : JOIN user_roles + roles
perms --> scopes : permission set
scopes --> jwt : scopes + permissions
jwt --> mw : JWT in cookie/header
mw --> rp : scope OK → check permission
rp --> ui : 403 or allow
eval --> tp_db : list_tool_policies()
approve --> eval : tool names
approve --> ae_db : (via audit)
usage --> ue_db : on_status()
audit --> ae_db : admin handlers
govjs --> roles_db : /v1/api/admin/roles
govjs --> tp_db : /v1/api/admin/policies
govjs --> pt_db : /v1/api/admin/skills
govjs --> ue_db : /v1/api/admin/usage
govjs --> ae_db : /v1/api/admin/audit
tload --> pt_db : list_default_templates()\nor get_by_name()
tload --> trender : template content
trender --> tsys : rendered content
tset --> tload : name or None
note right of pt_db
Read-only listing:
GET /v1/api/skills
(read scope, summary only)
end note
govjs --> wt_db : /v1/api/admin/skills
wtr --> wt_db : get_skill_by_name()
wtr --> wta : skill settings
wta --> pt_db : skill lookup
wtb --> approve : __budget_override__
auth -[hidden]-> mw
mw -[hidden]-> approve
@enduml
+203
View File
@@ -0,0 +1,203 @@
@startuml
!theme plain
title Turnstone — MCP Architecture (Resources, Prompts, Tools)
skinparam participant {
BackgroundColor<<mcp>> #E1BEE7
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<ui>> #E8EAF6
BackgroundColor<<registry>> #F8BBD0
}
participant "MCP Server\n(external)" as MCPSrv <<mcp>>
participant "MCPClientManager\n(mcp_client.py)" as MCPMgr <<mcp>>
participant "ChatSession\n(session.py)" as Session <<session>>
participant "StorageBackend\n(governance)" as Storage <<storage>>
participant "Server / Console\n(health + UI)" as UI <<server>>
participant "Console Admin UI\n(admin panel)" as Admin <<ui>>
participant "Database\n(mcp_servers table)" as DB <<storage>>
participant "MCPRegistryClient\n(mcp_registry.py)" as RegClient <<mcp>>
participant "MCP Registry\n(registry.modelcontextprotocol.io)" as Registry <<registry>>
== Admin-Driven Configuration ==
Admin -> DB : CRUD MCP server definitions\n(POST/PUT/DELETE /v1/api/admin/mcp-servers)
Admin -> UI : POST /v1/api/admin/mcp-servers/reload
UI -> MCPMgr : POST /_internal/mcp-reload\n(forwarded to each node)
MCPMgr -> MCPMgr : reconcile_sync()
note right
Diffs running servers against DB:
- New entries → connect
- Removed entries → disconnect
- Changed entries → reconnect
end note
== Registry Discovery & Install ==
Admin -> UI : GET /v1/api/admin/mcp-registry/search?search=...
UI -> RegClient : search(q, limit, cursor)
RegClient -> Registry : GET /v0.1/servers?search=...&latest=true
Registry --> RegClient : Server entries\n(remotes, packages, meta)
RegClient --> UI : RegistrySearchResult\n(annotated with installed status)
UI --> Admin : Search results\n(Install / Installed badges)
Admin -> UI : POST /v1/api/admin/mcp-registry/install
UI -> DB : create_mcp_server()\n(registry_name, version, meta)
UI -> MCPMgr : POST /_internal/mcp-reload\n(fan-out to nodes)
MCPMgr -> MCPMgr : reconcile_sync()
MCPMgr -> MCPSrv : connect to new server
note over RegClient, Registry
MCPRegistryClient is an async httpx client
targeting registry.modelcontextprotocol.io/v0.1.
resolve_install_config() translates registry
remotes/packages into mcp_servers rows.
end note
== Startup: Connection & Discovery ==
MCPMgr -> DB : load_mcp_config(storage=)\n(merge config file + DB)
MCPMgr -> MCPSrv : initialize (stdio or HTTP)
MCPSrv --> MCPMgr : capabilities\n(tools, resources, prompts)
MCPMgr -> MCPSrv : tools/list
MCPSrv --> MCPMgr : Tool[]
opt resources capability
MCPMgr -> MCPSrv : resources/list
MCPSrv --> MCPMgr : Resource[]
MCPMgr -> MCPSrv : resources/templates/list
MCPSrv --> MCPMgr : ResourceTemplate[]
end
opt prompts capability
MCPMgr -> MCPSrv : prompts/list
MCPSrv --> MCPMgr : Prompt[]
end
note over MCPMgr
Per-server storage:
_per_server_tools, _per_server_resources, _per_server_prompts
Copy-on-write rebuild into _tools, _resources, _prompts
Prefix: mcp__{server}__{name}
end note
MCPMgr -> Session : notify tool listeners
MCPMgr -> Session : notify resource listeners
== Governance Sync (on connect & refresh) ==
MCPMgr -> Storage : sync_prompts_to_storage()
note right
For each MCP prompt:
- Manual template exists? → skip
- MCP template exists? → update
(reset is_default=False)
- New? → create (origin="mcp",
readonly=True, is_default=False)
Removed prompts → delete
Protected by _sync_lock
end note
== set_storage() from entry point ==
UI -> MCPMgr : set_storage(backend)
note right
If servers already connected,
triggers immediate sync
end note
== Runtime: Tool Execution ==
Session -> Session : _prepare_mcp_tool(func_name, args)
note right
approval_label = func_name
(e.g. mcp__github__search)
needs_approval = True
end note
Session -> MCPMgr : call_tool_sync(name, args)
MCPMgr -> MCPSrv : tools/call
MCPSrv --> MCPMgr : ToolResult
MCPMgr --> Session : output (text)
== Runtime: Resource Read ==
Session -> Session : _prepare_read_resource(uri)
note right
approval_label = mcp_resource__{normalized_uri}
URI normalized (.. resolved)
needs_approval = True
end note
Session -> MCPMgr : read_resource_sync(uri)
MCPMgr -> MCPSrv : resources/read
MCPSrv --> MCPMgr : ReadResourceResult
MCPMgr --> Session : content (text/blob)
== Runtime: Prompt Invocation ==
Session -> Session : _prepare_use_prompt(name, arguments)
note right
approval_label = mcp__srv__prompt
Validated via is_mcp_prompt()
needs_approval = True
end note
Session -> MCPMgr : get_prompt_sync(name, args)
MCPMgr -> MCPSrv : prompts/get
MCPSrv --> MCPMgr : GetPromptResult
MCPMgr --> Session : messages [{role, content}]
== Three-Tier Refresh ==
group Push Notifications
MCPSrv -> MCPMgr : ToolListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_tools()
MCPSrv -> MCPMgr : ResourceListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_resources()
MCPSrv -> MCPMgr : PromptListChangedNotification
MCPMgr -> MCPMgr : _refresh_server_prompts()
MCPMgr -> Storage : sync_prompts_to_storage()
end
group Periodic Polling (default 4h)
MCPMgr -> MCPMgr : _periodic_refresh()
note right
Only polls capabilities
without push support.
Staggered per-server.
end note
end
group Manual Refresh
Session -> MCPMgr : refresh_sync()
note right: /mcp refresh [server]
end
== Policy Evaluation ==
note over Session
Tool policies use fnmatch on approval_label:
- mcp__github__* → allow (all GitHub tools/prompts)
- mcp_resource__file:///docs/* → allow
- mcp_resource__* → deny (block all resource reads)
- mcp__untrusted__* → ask
end note
== UI Visibility ==
UI -> MCPMgr : server_count, get_resources(), get_prompts()
note over UI
/health → mcp.servers, mcp.resources, mcp.prompts
Server UI: magenta status badge
Console: cluster status bar + node detail
System message: <mcp-resources> + <mcp-prompts> catalogs
end note
@enduml
+218
View File
@@ -0,0 +1,218 @@
@startuml
!theme plain
title Turnstone — Intent Validation (Judge) Architecture
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<judge>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<ui>> #E8EAF6
BackgroundColor<<fs>> #F5F5F5
}
participant "ChatSession\n(session.py)" as Session <<session>>
participant "IntentJudge\n(judge.py)" as Judge <<judge>>
participant "LLM Provider\n(provider)" as LLM <<judge>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "WebUI / SSE\n(server.py)" as UI <<ui>>
participant "Filesystem" as FS <<fs>>
== Tool Call Requires Approval ==
Session -> Session : _prepare_tool_calls()
note right
Tool calls parsed from
LLM response. Auto-approved
tools dispatched immediately.
Remaining items need approval.
end note
Session -> Session : _evaluate_intent(pending_items)
== Tier 1: Heuristic (synchronous, sub-ms) ==
Session -> Judge : evaluate(items, messages, callback)
Judge -> Judge : evaluate_heuristic()\nfor each item
note right
**36 rules (first match wins):**
Critical (0.90, deny): rm /, mkfs,
dd, pipe-to-shell, chmod 777 /,
write/edit /etc/ .ssh/,
download-then-execute chains
High (0.80, review): sudo, kill -9,
destructive git, DROP TABLE,
secrets, HTTP mutations, ssh/scp,
browser+data-export, transitive
install, control-plane mutation
Medium (0.70, review): content
ingestion, interpreter exec,
cloud CLI mutations, pkg install,
write_file, MCP tools, docker ops
Low (0.85, approve): read_file,
list_directory, search, recall,
tool_search, read_resource,
web_search, read-only bash
Default: medium, 0.50, review
end note
Judge --> Session : heuristic_verdicts[]
Session -> Session : attach _heuristic_verdict\nto each pending item
Session -> UI : SSE: approve_request\n{items: [{verdict: ...}],\n judge_pending: true}
note right
Heuristic verdict displayed
immediately as risk badge.
Spinner shown while LLM
judge evaluates.
end note
Session -> Storage : create_intent_verdict()\nfor each heuristic verdict
== Tier 2: LLM Judge (daemon thread, async) ==
Judge -> Judge : spawn daemon thread\n"intent-judge"
note over Judge, LLM
**Context preparation:**
1. FIFO-truncate conversation history
to max_context_ratio of context window
2. Append tool call details as user message
3. System prompt defines judge role + JSON schema
end note
loop up to 3 turns (timeout budget)
Judge -> LLM : create_completion(\nmodel, judge_messages,\ntools=[read_file, list_directory])
LLM --> Judge : CompletionResult
alt tool_calls present (turn < 3)
Judge -> Judge : _exec_read_only_tool()
note right
**Security hardening:**
Blocked: /etc/, /root/,
/proc/, /sys/, /dev/,
.ssh, .gnupg, .aws,
*.pem, *.key, *.p12
File cap: 32KB
Dir cap: 200 entries
end note
Judge -> FS : read_file / list_directory
FS --> Judge : file contents
Judge -> Judge : append tool result\nto judge_messages
else text response (final verdict)
Judge -> Judge : _parse_verdict()
note right
**4-stage JSON parsing:**
1. Direct JSON.loads
2. Markdown code block
3. Brace-counting
4. Regex field extraction
end note
end
end
== Tier 3: Arbitration ==
Judge -> Judge : compare confidence:\nLLM vs heuristic
note right
Only deliver LLM verdict
if confidence > heuristic.
Otherwise heuristic stands.
end note
alt LLM confidence > heuristic confidence
Judge -> Session : callback(llm_verdict)
Session -> UI : SSE: intent_verdict\n{tier: "llm", ...}
note right
UI replaces heuristic badge
with LLM verdict. Spinner
resolves to final assessment.
end note
Session -> Storage : create_intent_verdict()\nfor LLM verdict
end
== User Decision ==
UI -> Session : resolve_approval(\napproved, feedback)
Session -> Storage : update_intent_verdict(\nverdict_id, user_decision)
note right
All tracked verdicts
(heuristic + LLM) updated
with "approved" or "denied".
Swap-and-clear avoids racing
with daemon judge thread.
end note
== Tool Execution ==
Session -> Session : _execute_tools()
note right
Tools execute with
user approval.
end note
== Output Guard (synchronous, time-budgeted) ==
Session -> Session : _evaluate_output()\nfor each tool result
note right
**Priority-ordered checks (5s budget):**
P1: Prompt injection (role injection,
override phrases, instruction tags)
P2: Credential leakage (API keys,
PEM blocks, connection strings)
P3: Encoded payloads (data URIs,
hex shellcode)
P4: Adversarial URLs (cloud metadata,
credential query params)
P5: System info disclosure (private
IPs, sensitive paths)
Annotates + optionally redacts.
Does NOT gate.
end note
alt output_warning flags detected
Session -> UI : SSE: output_warning\n{call_id, risk_level, flags,\nfunc_name, redacted}
note right
Credential values replaced
with [REDACTED:<type>] before
output enters conversation.
sanitized text excluded from
SSE payload (defense in depth).
end note
UI -> Storage : record_output_assessment()\nfire-and-forget persistence
note right
Stored: flags, risk_level,
annotations, output_length,
redacted (bool). Raw tool
output is never stored.
end note
end
== Lifecycle ==
note over Session, Judge
**Lazy initialization:**
IntentJudge created on first approval if judge_config.enabled.
Re-uses session's provider/client by default (self-consistency).
Cross-model: separate provider/client from [judge] config.
**Sub-agent exemption:**
Plan agent and task agent skip intent validation entirely.
**Output guard:**
Runs when judge_config.output_guard is true (default).
Credential redaction when judge_config.redact_secrets is true.
**Storage:**
intent_verdicts table (migration 012), output_assessments table
(migration 022). Both queryable via admin API endpoints
(requires admin.judge permission). Skills store scan_status,
scan_report, scan_version for install-time risk assessment.
end note
@enduml
+159
View File
@@ -0,0 +1,159 @@
@startuml
!theme plain
title Turnstone — Structured Memory Architecture
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<facade>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<api>> #E8EAF6
BackgroundColor<<sdk>> #F5F5F5
}
participant "ChatSession\n(session.py)" as Session <<session>>
participant "MemoryFacade\n(memory.py)" as Facade <<facade>>
participant "MemoryRelevance\n(memory_relevance.py)" as Relevance <<facade>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "Server API\n(server.py)" as API <<api>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
== Phase 1: Tool Path (session.send) ==
Session -> Session : _prepare_tool_calls()\nparse memory(action=...)
note right
Tool schema: 4 actions
save, search, delete, list
Auto-approved (no approval needed)
end note
Session -> Session : _exec_memory(item)
alt action = save
Session -> Facade : save_structured_memory(\nname, content, description,\nmem_type, scope, scope_id)
Facade -> Facade : normalize_key(name)
Facade -> Storage : create_structured_memory()
alt unique constraint violation
Storage --> Facade : IntegrityError
Facade -> Storage : get_structured_memory_by_name()
Storage --> Facade : existing row
Facade -> Storage : update_structured_memory()
end
Storage --> Facade : memory_id
Facade --> Session : (memory_id, old_content)
Session -> Session : _init_system_messages()\nrefresh BM25 context
end
alt action = search
Session -> Facade : search_structured_memories(\nquery, mem_type, scope,\nscope_id, limit)
Facade -> Storage : search_structured_memories()
Storage --> Session : matched rows
end
alt action = delete
Session -> Facade : delete_structured_memory(\nname, scope, scope_id)
Facade -> Storage : delete_structured_memory()
Storage --> Session : bool (existed)
Session -> Session : _init_system_messages()\nrefresh BM25 context
end
== Phase 2: BM25 Relevance Injection ==
Session -> Session : _init_system_messages()\nevery conversation turn
Session -> Session : _get_visible_memories(\nlimit=fetch_limit)
note right
**Scope resolution:**
1. global scope (always)
2. workstream scope (ws_id)
3. user scope (user_id, if auth)
Combined and deduplicated.
end note
Session -> Facade : list_structured_memories()\nper scope
Facade -> Storage : list_structured_memories()
Storage --> Session : up to fetch_limit rows
Session -> Relevance : extract_recent_context(\nmessages, max_messages=3)
Relevance --> Session : user text context
Session -> Relevance : score_memories(\nmemories, context,\nk=relevance_k)
note right
**BM25 scoring:**
Index over name + description
+ content[:200] for each memory.
Returns top-k by relevance.
Empty query returns most recent k.
end note
Relevance --> Session : top-k memories
Session -> Relevance : build_memory_context(\nrelevant_memories)
note right
Formats as XML block:
<memories>
<memory name="..." type="..."
scope="..." description="...">
content (max 500 chars)
</memory>
</memories>
end note
Relevance --> Session : XML string
Session -> Session : inject into\nsystem message
== Phase 3: Server API Path ==
SDK -> API : GET /v1/api/memories\n?type=project&limit=20
API -> Facade : list_structured_memories()
Facade -> Storage : list_structured_memories()
Storage --> API : rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : POST /v1/api/memories\n{name, content, ...}
API -> API : validate type, scope,\nname length, content length
API -> Facade : save_structured_memory()
Facade -> Storage : create / update
Storage --> API : memory row
API --> SDK : 201 (created) / 200 (updated)
SDK -> API : POST /v1/api/memories/search\n{query, type, ...}
API -> Facade : search_structured_memories()
Facade -> Storage : search_structured_memories()
Storage --> API : matched rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : DELETE /v1/api/memories/{name}\n?scope=global
API -> Facade : delete_structured_memory()
Facade -> Storage : delete row
API --> SDK : {"status": "ok"}
== Phase 4: Console Admin Path ==
SDK -> Admin : GET /v1/api/admin/memories\n?type=&scope=&limit=
Admin -> Admin : require_permission(\n"admin.memories")
Admin -> Storage : list_structured_memories()
Storage --> Admin : rows
Admin --> SDK : {"memories": [...], "total": N}
SDK -> Admin : GET /v1/api/admin/memories/{id}
Admin -> Storage : get_structured_memory(id)
Storage --> Admin : memory row
Admin --> SDK : memory JSON
SDK -> Admin : DELETE /v1/api/admin/memories/{id}
Admin -> Storage : delete_structured_memory_by_id()
Admin -> Admin : record_audit(\n"memory.delete")
Admin --> SDK : {"status": "ok"}
== Configuration ==
note over Session, Relevance
**MemoryConfig** (from [memory] in config.toml):
relevance_k = 5 -- top-k memories per turn
fetch_limit = 50 -- max memories fetched for scoring
max_content = 32768 -- max content length per memory
nudge_cooldown = 300 -- seconds between metacognitive nudges
nudges = true -- enable/disable memory nudges
end note
@enduml
+151
View File
@@ -0,0 +1,151 @@
@startuml
!theme plain
title Turnstone — Settings Architecture
skinparam participant {
BackgroundColor<<session>> #C8E6C9
BackgroundColor<<config>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<api>> #E8EAF6
BackgroundColor<<sdk>> #F5F5F5
}
participant "Server\n(main)" as Server <<session>>
participant "ConfigStore\n(config_store.py)" as Store <<config>>
participant "SettingsRegistry\n(settings_registry.py)" as Registry <<config>>
participant "StorageBackend\n(SQLite)" as Storage <<storage>>
participant "Console Admin\n(console/server.py)" as Admin <<api>>
participant "SDK Client\n(sdk/)" as SDK <<sdk>>
participant "ChatSession\n(session.py)" as Session <<session>>
== Phase 1: Server Startup ==
Server -> Server : parse_args()\nCLI flags override defaults
Server -> Server : init_storage()\nSQLite / PostgreSQL
Server -> Store ** : ConfigStore(storage, node_id)
Store -> Storage : get_system_settings_bulk(node_id)
note right
1. Load global settings (node_id="")
2. Overlay per-node settings
Returns {key: json_value} dict
end note
Storage --> Store : raw settings
Store -> Registry : deserialize_value(key, json)\nper entry
Registry --> Store : typed values
Store -> Store : swap _cache atomically\nincrement _version
Server -> Server : warn_migrated_settings()
note right
Scans config.toml for keys
now managed by ConfigStore.
Logs warning for each overlap.
end note
Server -> Server : session_factory captures\nConfigStore reference
== Phase 2: Settings Read (session creation) ==
Server -> Session : session_factory(ws_id)
Session -> Store : get("model.temperature")
Store -> Store : cache[key] lookup\n(lock-free)
alt key in cache
Store --> Session : stored value
else key not in cache
Store -> Registry : SETTINGS[key].default
Registry --> Store : default value
Store --> Session : default value
end
note right of Session
Settings are captured once
at workstream creation.
Not re-read on every turn.
end note
== Phase 3: Admin API — List / Schema ==
SDK -> Admin : GET /v1/api/admin/settings
Admin -> Admin : require_permission(\n"admin.settings")
Admin -> Store : all_effective()
Store -> Store : merge cache with\nregistry defaults
Store --> Admin : {key: effective_value}
Admin -> Registry : SETTINGS (metadata)
note right
Annotates each setting with:
type, default, description,
is_stored, is_secret, constraints,
changed_by, updated
end note
Admin --> SDK : {"settings": [...], "total": N}
SDK -> Admin : GET /v1/api/admin/settings/schema
Admin -> Admin : require_permission(\n"admin.settings")
Admin -> Registry : SETTINGS catalog
Admin --> SDK : {"settings": [...], "total": N}
== Phase 4: Admin API — Update ==
SDK -> Admin : PUT /v1/api/admin/settings/\nmodel.temperature\n{"value": 0.7}
Admin -> Admin : require_permission(\n"admin.settings")
Admin -> Registry : validate_key("model.temperature")
Registry --> Admin : SettingDef
alt is_secret == true
Admin --> SDK : 403 Forbidden
else
Admin -> Registry : validate_value(key, 0.7)
note right
Type coercion: float(0.7)
Range check: 0.0 <= 0.7 <= 2.0
Choices check: (none for this key)
end note
Registry --> Admin : typed value
Admin -> Store : set(key, 0.7, changed_by="admin")
Store -> Registry : serialize_value(0.7)\n=> "0.7"
Store -> Storage : upsert_system_setting(\nkey, "0.7", node_id, ...)
Storage --> Store : ok
Store -> Store : swap _cache atomically
Admin -> Admin : record_audit(\n"setting.update")
Admin --> SDK : {"key": "...", "value": 0.7,\n"previous": 0.5}
end
== Phase 5: Admin API — Delete (reset to default) ==
SDK -> Admin : DELETE /v1/api/admin/settings/\nmodel.temperature
Admin -> Admin : require_permission(\n"admin.settings")
Admin -> Store : delete("model.temperature")
Store -> Registry : validate_key(key)
Store -> Storage : delete_system_setting(key, node_id)
Storage --> Store : bool (existed)
Store -> Store : remove from cache,\nswap atomically
Admin -> Admin : record_audit(\n"setting.delete")
Admin --> SDK : {"status": "ok",\n"key": "...", "default": 0.5}
== Phase 6: Hot Reload ==
SDK -> Admin : POST /v1/api/_internal/\nconfig-reload
Admin -> Store : reload()
Store -> Storage : get_system_settings_bulk(node_id)
Storage --> Store : all settings
Store -> Store : rebuild cache,\nswap atomically,\nincrement _version
note right
Existing sessions: unchanged
(frozen at creation time).
New sessions: pick up
updated values immediately.
end note
Admin --> SDK : {"status": "ok"}
== Precedence Summary ==
note over Server, Registry
**Server entry point:**
CLI flag > ConfigStore (database) > registry default
**CLI entry point:**
CLI flag > config.toml > argparse default
**Bootstrap settings** (database, Redis, auth, server bind):
Always from config.toml / env vars — never in ConfigStore.
end note
@enduml
+147
View File
@@ -0,0 +1,147 @@
@startuml
!theme plain
title Turnstone — OIDC Authorization Code Flow with PKCE
skinparam participant {
BackgroundColor<<browser>> #E8EAF6
BackgroundColor<<server>> #FFE0B2
BackgroundColor<<storage>> #B3E5FC
BackgroundColor<<idp>> #C8E6C9
}
participant "Browser" as Browser <<browser>>
participant "Turnstone\n(Server / Console)" as Server <<server>>
database "SQLite /\nPostgreSQL" as DB <<storage>>
participant "Identity Provider\n(IdP)" as IdP <<idp>>
== Page Load ==
Browser -> Server : GET /v1/api/auth/status
Server --> Browser : {oidc_enabled: true,\noidc_provider_name: "...",\npassword_enabled: true}
note right of Browser
Login screen renders
"Continue with {provider_name}"
button alongside password form.
If password_enabled=false,
only the SSO button is shown.
end note
== Authorization Request ==
Browser -> Server : GET /v1/api/auth/oidc/authorize
Server -> Server : Generate state (random)\nnonce (random)\nPKCE code_verifier + code_challenge
Server -> DB : create_oidc_pending_state(\nstate, nonce, code_verifier, audience)
note right of DB
Stored with created_at timestamp.
Expires after 5 minutes.
end note
Server --> Browser : 302 Redirect to IdP\nauthorization_endpoint
Browser -> IdP : GET /authorize?\nresponse_type=code&\nclient_id=...&\nredirect_uri=...&\nscope=openid email profile&\nstate=...&nonce=...&\ncode_challenge=...&\ncode_challenge_method=S256
== User Authentication (at IdP) ==
IdP -> Browser : Login page (if no\nexisting IdP session)
Browser -> IdP : User authenticates\n(username/password, MFA, etc.)
IdP --> Browser : 302 Redirect to callback\n?code=AUTH_CODE&state=STATE
== Callback Processing ==
Browser -> Server : GET /v1/api/auth/oidc/callback\n?code=AUTH_CODE&state=STATE
Server -> Server : Rate limit check\n(5 per 5min per IP)
Server -> DB : cleanup_expired_oidc_states(300)
note right of DB
Lazy cleanup of states
older than 5 minutes.
end note
Server -> DB : pop_oidc_pending_state(state)
DB --> Server : {nonce, code_verifier, audience}
note right of Server
Atomic fetch-and-delete.
Returns None if state is
expired or unknown.
end note
== Token Exchange ==
Server -> IdP : POST /token\ngrant_type=authorization_code&\ncode=AUTH_CODE&\nclient_id=...&\nclient_secret=...&\ncode_verifier=...&\nredirect_uri=...
note right of Server
Client secret + PKCE verifier
sent server-side only.
Never exposed to browser.
end note
IdP --> Server : {id_token: "eyJ...",\naccess_token: "..."}
== ID Token Validation ==
Server -> IdP : Fetch JWKS public keys\n(cached at startup, refreshed\non-demand when unknown kid\nencountered — key rotation)
Server -> Server : Validate ID token:\n1. Verify signature (RS256/ES256)\n2. Check iss == configured issuer\n3. Check aud == client_id\n4. Check exp (not expired)\n5. Verify nonce matches
== User Provisioning ==
Server -> DB : get_oidc_identity(issuer, sub)
alt Existing identity found
DB --> Server : {user_id, ...}
Server -> DB : update_oidc_identity_login()\nupdate last_login timestamp
Server -> DB : get_user(user_id)
DB --> Server : user record
else New user (first login)
Server -> Server : Derive username from\npreferred_username / email
Server -> DB : create_user(user_id, username,\ndisplay_name, "!oidc")
note right of DB
Password hash set to sentinel
value "!oidc" — not a valid
bcrypt hash, so password login
is always rejected.
end note
Server -> DB : create_oidc_identity(\nissuer, sub, user_id, email)
end
opt Role mapping configured
Server -> Server : Read role_claim from ID token\nMap values via role_map
Server -> DB : Sync roles: add new,\nrevoke stale OIDC-assigned,\npreserve manually assigned
end
== Issue Turnstone JWT ==
Server -> Server : Load user permissions\nDerive scopes from permissions
Server -> Server : Create JWT (HS256)\nsub: user_id\nscopes: read,write,...\nsrc: "oidc"\naud: turnstone-server\nexp: +24h
Server --> Browser : 302 Redirect to /?oidc_success=1\nSet-Cookie: session=JWT\n(HttpOnly, SameSite=Lax, Secure)
== Browser Success Detection ==
Browser -> Browser : Detect ?oidc_success=1\nStrip param from URL\n(history.replaceState)
Browser -> Browser : Hide login overlay\nCall onLoginSuccess()
note right of Browser
Browser is now authenticated.
JWT cookie sent on all
subsequent requests.
end note
== Error Paths ==
note over Browser, IdP
**Error handling:**
- IdP returns error param → redirect to /?oidc_error=...
- State missing/expired → redirect to /?oidc_error=Login+session+expired
- Token exchange fails → redirect to /?oidc_error=...
- ID token validation fails → redirect to /?oidc_error=...
- No admin user exists → redirect to /?oidc_error=Initial+setup+required
- Rate limit exceeded → redirect to /?oidc_error=Too+many+login+attempts
All errors are shown as toast messages on the login screen.
end note
@enduml
@@ -0,0 +1,99 @@
@startuml
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
LAYOUT_LEFT_RIGHT()
title Skills Discovery & Runtime Architecture
skinparam backgroundColor #1e1e2e
skinparam defaultFontColor #cdd6f4
skinparam defaultFontName "JetBrains Mono"
skinparam arrowColor #89b4fa
skinparam rectangleBorderColor #585b70
skinparam rectangleBackgroundColor #313244
skinparam noteBorderColor #585b70
skinparam noteBackgroundColor #45475a
skinparam packageBorderColor #585b70
package "External Sources" as ext #181825 {
rectangle "skills.sh\nRegistry" as skillssh
rectangle "GitHub\nRepositories" as github
}
package "Console Server" as console #181825 {
rectangle "admin_skill_discover\nGET /v1/api/admin/skills/discover" as discover
rectangle "admin_skill_install\nPOST /v1/api/admin/skills/install" as install
rectangle "_get_discovery_url\nsettings fallback" as settings
}
package "Core Modules" as core #181825 {
rectangle "SkillsShClient\nskill_sources.py" as client
rectangle "fetch_skill_from_github\nskill_sources.py" as fetcher
rectangle "parse_skill_md\nskill_parser.py" as parser
rectangle "scan_skill_content\nstorage/_utils.py" as scanner
}
package "Session Runtime" as runtime #181825 {
rectangle "load_skill tool\nsession.py" as loadtool
rectangle "set_skill()\nsession.py" as setskill
rectangle "_load_skills()\nsession.py" as loadskills
}
package "Storage" as storage #181825 {
rectangle "prompt_templates\n(skills)" as skills_table
rectangle "skill_resources\n(bundled files)" as resources_table
rectangle "system_settings\n(discovery_url)" as settings_table
}
package "Admin UI" as ui #181825 {
rectangle "Skills Tab\nInstalled / Discover pill" as pill
rectangle "Discovery View\nsearch + cards" as discoverui
rectangle "GitHub Import\nmodal" as importui
}
' External discovery flow
discover --> settings : resolve URL
settings --> settings_table : DB -> config -> default
discover --> client : search(query)
client --> skillssh : GET /api/search
install --> client : resolve_github_url()
client --> skillssh : GET /api/skills/{id}
install --> fetcher : fetch SKILL.md + resources
fetcher --> github : raw.githubusercontent.com
fetcher --> github : api.github.com/git/trees
fetcher --> parser : parse frontmatter
install --> scanner : auto-scan on create
install --> skills_table : create_prompt_template
install --> resources_table : create_skill_resource
' Runtime skill loading flow
loadtool --> skills_table : search (BM25 ranking)
loadtool --> setskill : load (name)
setskill --> loadskills : reload + reinit system messages
loadskills --> skills_table : get_skill_by_name
' UI flow
pill --> discoverui : switch view
discoverui --> discover : authFetch()
importui --> install : POST (github source)
' Annotations
note right of parser
YAML frontmatter -> ParsedSkill
Anthropic + Hermes tag formats
Name validation (lowercase+hyphens)
end note
note right of loadtool
search: auto-approved (read-only)
load: requires user approval
Main session only (no sub-agents)
end note
note right of scanner
4 risk axes (content, supply chain,
vulnerability, capability)
Auto-triggers on create/update
end note
@enduml
+286
View File
@@ -0,0 +1,286 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 540" font-family="-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif">
<defs>
<!-- Arrowhead markers -->
<marker id="arrow" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#484f58"/>
</marker>
<marker id="arrow-blue" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#58a6ff"/>
</marker>
<marker id="arrow-green" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#3fb950"/>
</marker>
<marker id="arrow-orange" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#f0883e"/>
</marker>
<marker id="arrow-coral" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#f47067"/>
</marker>
<marker id="arrow-muted" viewBox="0 0 10 7" refX="10" refY="3.5" markerWidth="8" markerHeight="6" orient="auto-start-auto">
<path d="M 0 0 L 10 3.5 L 0 7 z" fill="#8b949e"/>
</marker>
<!-- Card shadow filter -->
<filter id="shadow" x="-4%" y="-4%" width="108%" height="112%">
<feDropShadow dx="0" dy="1" stdDeviation="2" flood-color="#000" flood-opacity="0.4"/>
</filter>
</defs>
<!-- Background -->
<rect width="1200" height="540" rx="8" fill="#0d1117"/>
<!-- Title -->
<text x="600" y="36" text-anchor="middle" fill="#e6edf3" font-size="15" font-weight="700" letter-spacing="3">TURNSTONE</text>
<text x="600" y="54" text-anchor="middle" fill="#8b949e" font-size="11" letter-spacing="1">SYSTEM ARCHITECTURE</text>
<!-- ==================== COLUMN HEADERS ==================== -->
<text x="90" y="86" text-anchor="middle" fill="#58a6ff" font-size="9" font-weight="600" letter-spacing="2">CLIENTS</text>
<text x="276" y="86" text-anchor="middle" fill="#3fb950" font-size="9" font-weight="600" letter-spacing="2">GATEWAYS</text>
<text x="480" y="86" text-anchor="middle" fill="#f0883e" font-size="9" font-weight="600" letter-spacing="2">MESSAGE QUEUE</text>
<text x="700" y="86" text-anchor="middle" fill="#f47067" font-size="9" font-weight="600" letter-spacing="2">CLUSTER NODES</text>
<text x="940" y="86" text-anchor="middle" fill="#f778ba" font-size="9" font-weight="600" letter-spacing="2">LLM PROVIDERS</text>
<!-- ==================== CLIENT BOXES ==================== -->
<!-- CLI -->
<g filter="url(#shadow)">
<rect x="30" y="108" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="108" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="108" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="111" width="120" height="2" fill="#161b22"/>
<text x="90" y="130" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">CLI</text>
<text x="90" y="145" text-anchor="middle" fill="#8b949e" font-size="9">terminal REPL</text>
</g>
<!-- Browser UI -->
<g filter="url(#shadow)">
<rect x="30" y="174" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="174" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="174" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="177" width="120" height="2" fill="#161b22"/>
<text x="90" y="196" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Browser UI</text>
<text x="90" y="211" text-anchor="middle" fill="#8b949e" font-size="9">HTTP + SSE</text>
</g>
<!-- SDK / API -->
<g filter="url(#shadow)">
<rect x="30" y="244" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="244" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="244" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="247" width="120" height="2" fill="#161b22"/>
<text x="90" y="266" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">SDK / API</text>
<text x="90" y="281" text-anchor="middle" fill="#8b949e" font-size="9">programmatic</text>
</g>
<!-- Discord / Slack -->
<g filter="url(#shadow)">
<rect x="30" y="314" width="120" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="30" y="314" width="120" height="5" rx="5" fill="#58a6ff"/>
<rect x="30" y="314" width="120" height="5" fill="#58a6ff"/>
<rect x="30" y="317" width="120" height="2" fill="#161b22"/>
<text x="90" y="336" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Discord / Slack</text>
<text x="90" y="351" text-anchor="middle" fill="#8b949e" font-size="9">chat platforms</text>
</g>
<!-- ==================== GATEWAY BOXES ==================== -->
<!-- Console -->
<g filter="url(#shadow)">
<rect x="216" y="118" width="120" height="58" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="216" y="118" width="120" height="5" rx="5" fill="#3fb950"/>
<rect x="216" y="118" width="120" height="5" fill="#3fb950"/>
<rect x="216" y="121" width="120" height="2" fill="#161b22"/>
<text x="276" y="142" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Console</text>
<text x="276" y="157" text-anchor="middle" fill="#8b949e" font-size="9">dashboard + proxy</text>
<text x="276" y="169" text-anchor="middle" fill="#8b949e" font-size="9">cluster management</text>
</g>
<!-- Channel Gateway -->
<g filter="url(#shadow)">
<rect x="216" y="292" width="120" height="58" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="216" y="292" width="120" height="5" rx="5" fill="#3fb950"/>
<rect x="216" y="292" width="120" height="5" fill="#3fb950"/>
<rect x="216" y="295" width="120" height="2" fill="#161b22"/>
<text x="276" y="316" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Channel Gateway</text>
<text x="276" y="331" text-anchor="middle" fill="#8b949e" font-size="9">platform adapter</text>
<text x="276" y="343" text-anchor="middle" fill="#8b949e" font-size="9">Discord, Slack, ...</text>
</g>
<!-- ==================== REDIS MQ ==================== -->
<g filter="url(#shadow)">
<rect x="420" y="168" width="120" height="132" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="420" y="168" width="120" height="5" rx="5" fill="#f0883e"/>
<rect x="420" y="168" width="120" height="5" fill="#f0883e"/>
<rect x="420" y="171" width="120" height="2" fill="#161b22"/>
<text x="480" y="198" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Redis MQ</text>
<line x1="438" y1="210" x2="522" y2="210" stroke="#30363d" stroke-width="1"/>
<text x="480" y="228" text-anchor="middle" fill="#8b949e" font-size="9">inbound queues</text>
<text x="480" y="243" text-anchor="middle" fill="#8b949e" font-size="9">event pub/sub</text>
<text x="480" y="258" text-anchor="middle" fill="#8b949e" font-size="9">node heartbeats</text>
<text x="480" y="273" text-anchor="middle" fill="#8b949e" font-size="9">workstream routing</text>
<text x="480" y="288" text-anchor="middle" fill="#8b949e" font-size="9">cluster state</text>
</g>
<!-- ==================== CLUSTER NODES ==================== -->
<!-- Cluster outline -->
<rect x="598" y="100" width="204" height="310" rx="8" fill="none" stroke="#30363d" stroke-width="1" stroke-dasharray="4,3"/>
<text x="700" y="422" text-anchor="middle" fill="#30363d" font-size="9" letter-spacing="1">CLUSTER</text>
<!-- Node A -->
<g filter="url(#shadow)">
<rect x="614" y="118" width="170" height="100" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="118" width="170" height="5" rx="5" fill="#f47067"/>
<rect x="614" y="118" width="170" height="5" fill="#f47067"/>
<rect x="614" y="121" width="170" height="2" fill="#161b22"/>
<text x="699" y="142" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Node A</text>
<line x1="632" y1="152" x2="766" y2="152" stroke="#30363d" stroke-width="1"/>
<!-- Bridge -->
<rect x="626" y="162" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="661" y="180" text-anchor="middle" fill="#8b949e" font-size="9">bridge</text>
<!-- Server -->
<rect x="704" y="162" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="739" y="180" text-anchor="middle" fill="#8b949e" font-size="9">server</text>
<!-- Arrow bridge to server -->
<line x1="696" y1="176" x2="702" y2="176" stroke="#484f58" stroke-width="1" marker-end="url(#arrow)"/>
<!-- Tools label -->
<text x="699" y="206" text-anchor="middle" fill="#484f58" font-size="8">14 tools + MCP</text>
</g>
<!-- Node B -->
<g filter="url(#shadow)">
<rect x="614" y="238" width="170" height="100" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="238" width="170" height="5" rx="5" fill="#f47067"/>
<rect x="614" y="238" width="170" height="5" fill="#f47067"/>
<rect x="614" y="241" width="170" height="2" fill="#161b22"/>
<text x="699" y="262" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Node B</text>
<line x1="632" y1="272" x2="766" y2="272" stroke="#30363d" stroke-width="1"/>
<!-- Bridge -->
<rect x="626" y="282" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="661" y="300" text-anchor="middle" fill="#8b949e" font-size="9">bridge</text>
<!-- Server -->
<rect x="704" y="282" width="70" height="28" rx="3" fill="#1c2128" stroke="#30363d" stroke-width="1"/>
<text x="739" y="300" text-anchor="middle" fill="#8b949e" font-size="9">server</text>
<!-- Arrow bridge to server -->
<line x1="696" y1="296" x2="702" y2="296" stroke="#484f58" stroke-width="1" marker-end="url(#arrow)"/>
<!-- Tools label -->
<text x="699" y="326" text-anchor="middle" fill="#484f58" font-size="8">14 tools + MCP</text>
</g>
<!-- ==================== LLM PROVIDERS ==================== -->
<!-- OpenAI -->
<g filter="url(#shadow)">
<rect x="870" y="130" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="130" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="130" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="133" width="140" height="2" fill="#161b22"/>
<text x="940" y="153" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">OpenAI</text>
<text x="940" y="167" text-anchor="middle" fill="#8b949e" font-size="9">GPT-5, o-series</text>
</g>
<!-- Anthropic -->
<g filter="url(#shadow)">
<rect x="870" y="196" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="196" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="196" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="199" width="140" height="2" fill="#161b22"/>
<text x="940" y="219" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Anthropic</text>
<text x="940" y="233" text-anchor="middle" fill="#8b949e" font-size="9">Claude 4.5 / 4.6</text>
</g>
<!-- Local / vLLM -->
<g filter="url(#shadow)">
<rect x="870" y="262" width="140" height="46" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="870" y="262" width="140" height="5" rx="5" fill="#f778ba"/>
<rect x="870" y="262" width="140" height="5" fill="#f778ba"/>
<rect x="870" y="265" width="140" height="2" fill="#161b22"/>
<text x="940" y="285" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">Local / vLLM</text>
<text x="940" y="299" text-anchor="middle" fill="#8b949e" font-size="9">llama.cpp, NIM</text>
</g>
<!-- ==================== STORAGE ==================== -->
<g filter="url(#shadow)">
<rect x="614" y="450" width="170" height="52" rx="5" fill="#161b22" stroke="#30363d" stroke-width="1"/>
<rect x="614" y="450" width="170" height="5" rx="5" fill="#bc8cff"/>
<rect x="614" y="450" width="170" height="5" fill="#bc8cff"/>
<rect x="614" y="453" width="170" height="2" fill="#161b22"/>
<text x="699" y="476" text-anchor="middle" fill="#e6edf3" font-size="11" font-weight="600">PostgreSQL / SQLite</text>
<text x="699" y="492" text-anchor="middle" fill="#8b949e" font-size="9">conversations, memory, auth</text>
</g>
<text x="699" y="444" text-anchor="middle" fill="#bc8cff" font-size="9" font-weight="600" letter-spacing="2">STORAGE</text>
<!-- ==================== CONNECTION LINES ==================== -->
<!-- CLIENT -> GATEWAY connections -->
<!-- Browser -> Console -->
<line x1="150" y1="197" x2="214" y2="155" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- Discord -> Channel -->
<line x1="150" y1="337" x2="214" y2="325" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- CLI -> direct to Node A server (top path, curved) -->
<path d="M 150 131 C 200 131, 200 100, 400 100 L 400 100 C 500 100, 570 140, 612 168" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.5" fill="none" stroke-dasharray="6,3" marker-end="url(#arrow-blue)"/>
<text x="370" y="96" fill="#484f58" font-size="8" text-anchor="middle">direct</text>
<!-- SDK -> Redis (direct push) -->
<line x1="150" y1="267" x2="418" y2="240" stroke="#58a6ff" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-blue)"/>
<!-- GATEWAY -> REDIS connections -->
<!-- Console -> Redis -->
<line x1="336" y1="160" x2="418" y2="200" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-green)"/>
<!-- Channel -> Redis -->
<line x1="336" y1="318" x2="418" y2="272" stroke="#3fb950" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-green)"/>
<!-- REDIS -> NODE connections -->
<!-- Redis -> Node A bridge -->
<line x1="540" y1="210" x2="624" y2="176" stroke="#f0883e" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-orange)"/>
<!-- Redis -> Node B bridge -->
<line x1="540" y1="260" x2="624" y2="296" stroke="#f0883e" stroke-width="1.2" stroke-opacity="0.6" marker-end="url(#arrow-orange)"/>
<!-- Console -> Node (proxy, dashed) -->
<path d="M 336 147 C 380 130, 500 108, 612 145" stroke="#3fb950" stroke-width="1" stroke-opacity="0.4" fill="none" stroke-dasharray="4,3" marker-end="url(#arrow-green)"/>
<text x="468" y="120" fill="#484f58" font-size="8" text-anchor="middle">proxy</text>
<!-- NODE -> LLM connections -->
<!-- Node A -> LLM providers -->
<line x1="784" y1="168" x2="868" y2="155" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.5" marker-end="url(#arrow-coral)"/>
<line x1="784" y1="176" x2="868" y2="219" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.3"/>
<line x1="784" y1="180" x2="868" y2="282" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.2"/>
<!-- Node B -> LLM providers -->
<line x1="784" y1="288" x2="868" y2="163" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.2"/>
<line x1="784" y1="296" x2="868" y2="222" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.3"/>
<line x1="784" y1="300" x2="868" y2="288" stroke="#f47067" stroke-width="1.2" stroke-opacity="0.5" marker-end="url(#arrow-coral)"/>
<!-- NODE -> STORAGE connections -->
<line x1="680" y1="338" x2="680" y2="448" stroke="#bc8cff" stroke-width="1.2" stroke-opacity="0.4" stroke-dasharray="4,3" marker-end="url(#arrow-muted)"/>
<line x1="718" y1="218" x2="718" y2="236" stroke="#484f58" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="2,2"/>
<!-- Extensibility hint -->
<text x="699" y="392" text-anchor="middle" fill="#30363d" font-size="10">...</text>
<!-- Event flow: Bridges -> Redis (dashed, bidirectional feel) -->
<line x1="624" y1="186" x2="542" y2="220" stroke="#f0883e" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="3,3"/>
<line x1="624" y1="286" x2="542" y2="250" stroke="#f0883e" stroke-width="1" stroke-opacity="0.3" stroke-dasharray="3,3"/>
<text x="574" y="242" fill="#484f58" font-size="7" text-anchor="middle">events</text>
<!-- ==================== FLOW LABELS ==================== -->
<!-- Interactive flow label -->
<rect x="30" y="395" width="10" height="10" rx="2" fill="none" stroke="#58a6ff" stroke-width="1.5" stroke-dasharray="3,2"/>
<text x="46" y="404" fill="#8b949e" font-size="9">interactive (direct)</text>
<!-- Queue flow label -->
<rect x="160" y="395" width="10" height="10" rx="2" fill="none" stroke="#f0883e" stroke-width="1.5"/>
<text x="176" y="404" fill="#8b949e" font-size="9">queue-driven</text>
<!-- Proxy/event label -->
<rect x="275" y="395" width="10" height="10" rx="2" fill="none" stroke="#3fb950" stroke-width="1.5" stroke-dasharray="3,2"/>
<text x="291" y="404" fill="#8b949e" font-size="9">proxy / events</text>
<!-- ==================== BOTTOM DETAILS ==================== -->
<line x1="30" y1="430" x2="1170" y2="430" stroke="#21262d" stroke-width="1"/>
<!-- Routing rules at bottom, left-aligned -->
<text x="44" y="456" fill="#30363d" font-size="9" font-weight="600" letter-spacing="1">ROUTING</text>
<circle cx="44" cy="474" r="3" fill="#f47067" opacity="0.6"/>
<text x="54" y="477" fill="#484f58" font-size="9">target_node set &#x2192; route to specific node queue</text>
<circle cx="44" cy="494" r="3" fill="#f0883e" opacity="0.6"/>
<text x="54" y="497" fill="#484f58" font-size="9">ws_id set &#x2192; route to owning node</text>
<circle cx="44" cy="514" r="3" fill="#58a6ff" opacity="0.6"/>
<text x="54" y="517" fill="#484f58" font-size="9">neither &#x2192; shared queue, any node picks up</text></svg>

After

Width:  |  Height:  |  Size: 18 KiB

+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:341a8ab1483b1e0146878bd384a11d56bc78d29262de8262d06ef924317e2762
size 139969
oid sha256:d8ce6d2a43a991655c3f64a20b6e810fdb2f78eb767acc3d3d1b8d2c9f443181
size 165011
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:29534422fc31eee613f70a479aa14de5278b98c49bb75fce7a63b72e248f1149
size 323269
oid sha256:efcc7cbe8161a54b5ec24bdfd47e8a142f70029e6e66c707e811b99369f85ebf
size 310079
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f85f24081d32318e079d26a4855ebb6e66df349ca7c7c703db50788af528426b
size 376282
oid sha256:0e605963c649574c7bf987b2d338257c78bc1fc68b524ac8276bb25365035e06
size 594096
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a90dec1546dd0f8343e3e27cf6f235d95f3ffc375928d34d0df5d8f25d63e5ad
size 269901
oid sha256:da9d32000e3d92d92ce621661ced60f276f9b5be652f5ed6123b400505415f4a
size 319702
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cb36b4924394cf54d6aaef454cced317b72e336e580cc6ac25cd4b9d0917bec5
size 243422
oid sha256:43844b07d36beb04db871f6795a3f3be17852a6a484fdc0ea207403bd7f512a6
size 274286
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e660f453a3708d1a7d827f07cd967f122500eb1c4845f5a75cc1309091bce6af
size 185796
oid sha256:2636e2d4d2f84f26f93de6e984b780f6ad5bf8d3618a0202ea0a2ee80859c5b5
size 312409
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8cc7c94d5ac4862c3c09450346f0af923818e02ea0550cc8023f039fdc701179
size 221528
oid sha256:09535722ba975e47cf0557a40b6c481f125ff2022c396f79715c3bba9f715871
size 222032
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d33b9b3affcdb07086b5aebca8a3b9c2b009cdfc6f360950a0e72e65fbcb8f17
size 201602
oid sha256:ed457b10b534b5fc2a5e190b281d7ded4dd1615da2229d67a373cf5dddccd059
size 201601
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:adac93a0bb062d7199b819a600a0983ff011a75d16928fb80322cbb41f9284ea
size 158866
oid sha256:7896c6e041b6dbb89d034468fa980c8fe645df5eb969d45ef966ccc6399edac2
size 200083
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:69f201cff948cb0a19810b7c4ad26d346f869ee2dd3141eba4f353332efa2e21
size 373649
oid sha256:35cf3a6942f62dabcbbe012ac2f9e6f155332c894692981b076de5a25c1f3330
size 374055
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:793d7c2b28a751c6f467f2de788fcd462d3b8fd9cd5cb7adb5b32d78fb185394
size 236004
oid sha256:a74b4b8b5dbfb1a51a01100b731477968942b01218bad9451a3d5a9cb3003294
size 411665
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0e6c1dfaef840d5228645aaad3637c973b2f71c372595814f3b743a991f5c6fc
size 239128
oid sha256:84524f4bc900708ac8adf081591d336f862830188eb8505e71a0f071b339d923
size 252599
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e7c3e40c10425d721f833390ae3531c09af501157fd3142531ba4eba86ff719d
size 197112
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c94556889abb382cd5b818639fc0a4706beef3d9c7a0b4cbedc763943d657dd0
size 244998
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:af5ab3126bf685afe68e24bc4b0ed97371d0ebdb77bf4d76c0331ab120580cc0
size 248809
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6fc99bb8d84d6e9f3dac9d5c12ac7f569a041b29431c57612c24b50f332982ed
size 462992
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc4c511c34a2e5d286fd128c3509405a5b240ca02a4bafb395d2e94d002a5b8b
size 293203
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96176a09e65e90dadc32d5e9ed778423842be89204d2cf382225f53a90cfaf01
size 258547
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:98ba80fa1dab4d37299e61be079a6fbc8740fc3ab92196f828a765f74caf4556
size 200720
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a6b7769aa7e732ffbeb1eb7f5b65273a135fb3a78d9802ec36d3b92801c34f6b
size 427745
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:79a690c466a5d6f6d4292d78a27b9474e9e9c1373fa80e17dfe37700238c8af8
size 382508
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c89628ed917dfd576c1af75c68fe5fed9beadaaee9dcea7aa7a1643867c4f1b9
size 344323
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:83c0e6aad3eb19f6bc475a30a77215e801da3da5930f0462417fe7eb6eda6be2
size 347144
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1c21910e3916be789b0377c8a0dcc8f47d66a967861a543d5bdd0c26da185259
size 309584
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ba14003062fec7eb9eaca7a3de945767e40bdd821468e19e7d1edcfa7ce1eb41
size 193581
+49 -7
View File
@@ -27,6 +27,9 @@ Console dashboard: http://localhost:8090
| `server` | 8080 | default | Web UI + chat workstreams + LLM |
| `bridge` | — | default | Redis-to-HTTP bridge (multi-node routing) |
| `console` | 8090 | default | Cluster dashboard |
| `channel` | — | production | Channel gateway (Discord, Slack, etc.) |
| `server-1``server-10` | — | cluster | 10-node server fleet (PostgreSQL required) |
| `bridge-1``bridge-10` | — | cluster | Matching bridge fleet |
| `sim` | — | sim | Multi-node cluster simulator |
## Profiles
@@ -37,6 +40,18 @@ Console dashboard: http://localhost:8090
docker compose up
```
**Production** — adds PostgreSQL and the channel gateway. Requires `POSTGRES_PASSWORD` and (for Discord) `TURNSTONE_DISCORD_TOKEN`:
```bash
docker compose --profile production up
```
**Cluster** — 10-node server/bridge fleet sharing PostgreSQL and Redis. Access all nodes via the console at `:8090`. Requires `POSTGRES_PASSWORD`:
```bash
docker compose --profile cluster up
```
**Sim** — adds the simulator. Can run alongside the full stack or standalone with just Redis and the console:
```bash
@@ -57,7 +72,7 @@ All configuration is via environment variables in `.env` (copy from `.env.exampl
|----------|---------|-------------|
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | OpenAI-compatible API URL |
| `OPENAI_API_KEY` | `dummy` | API key (`dummy` for local servers) |
| `TAVILY_API_KEY` | — | Web search API key (optional) |
| `TAVILY_API_KEY` | — | Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search) |
### Redis
@@ -84,8 +99,35 @@ All configuration is via environment variables in `.env` (copy from `.env.exampl
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_AUTH_ENABLED` | — | Set to `1` to require Bearer token auth |
| `TURNSTONE_AUTH_TOKEN` | — | Shared auth token for server/bridge/console |
| `TURNSTONE_AUTH_ENABLED` | — | Set to `1` to require authentication |
| `TURNSTONE_AUTH_TOKEN` | — | Config-file token for server/bridge/console (backward compat, works alongside JWT) |
| `TURNSTONE_JWT_SECRET` | — | Secret key for signing JWTs (required when using user identity / JWT auth) |
### Database
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql://user:pass@db:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.
> **First-time setup:** After deploying with auth enabled, create an initial admin user by running `turnstone-admin create-user` inside the container:
>
> ```bash
> docker compose exec server turnstone-admin create-user --username admin --name "Admin"
> ```
>
> You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass `--token --scopes read,write,approve` to also generate an initial API token.
### Channel Gateway
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord adapter) |
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages through Redis MQ to the bridge and server. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking.
### Simulator
@@ -101,13 +143,13 @@ All configuration is via environment variables in `.env` (copy from `.env.exampl
## Scaling
Scale to multiple server/bridge pairs:
For multi-node testing, use the `cluster` profile which provides 10 dedicated server+bridge pairs with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
docker compose up --scale server=3 --scale bridge=3
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
Each bridge auto-generates a unique node ID from its container hostname. When scaling `server`, remove the host port mapping (or use a reverse proxy) to avoid port conflicts.
The default `server` and `bridge` also run alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
## Volumes
@@ -128,7 +170,7 @@ docker compose build
docker compose build --no-cache
```
All five entry points are installed in a single image: `turnstone-server`, `turnstone-bridge`, `turnstone-console`, `turnstone-sim`, `turnstone-eval`.
All entry points are installed in a single image: `turnstone-server`, `turnstone-bridge`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`, `turnstone-sim`, `turnstone-eval`.
## Cleanup

Some files were not shown because too many files have changed in this diff Show More