mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
e60c19befd5e31376bb606cd380c3564ac4e27df
19 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
7053439e84 |
refactor(eval): split measurement core from prompt optimizer
turnstone-eval was misnamed: it was a prompt optimizer, not a measurement harness. Split the 3252-line turnstone/eval.py into a strictly one-way dependency (optimizer -> eval-core; core never imports the optimizer): - turnstone/eval/core.py measurement substrate — everything up to and including _run_iteration: provider detection, NullUI, HeadlessSession, the test runner, score_run, aggregation, and neutral reporting. - turnstone/eval/cli.py new measure-only `turnstone-eval` — the old --no-optimize path promoted to the whole job (one _run_iteration call, then print the summary table). - turnstone/optimizer.py the UCB self-modify loop and its multi-agent pipeline (analyst/optimizer/observer/diversifier/tool optimizer), now `turnstone-optimizer`; imports from eval.core only. - turnstone/eval/__init__.py re-exports the core public API for back-compat (score_run, _match_action, _run_iteration, HeadlessSession). _apply_tool_overrides lives in core (HeadlessSession needs it) rather than alongside the other tree helpers, so the dependency stays one-way. Breaking change: `turnstone-eval` now measures; use `turnstone-optimizer` to optimize. Both code paths are behaviour-preserving — the moved function bodies are byte-identical. |
||
|
Patrick Buckley
|
c0be383f99 |
refactor(doctor): replace turnstone-bootstrap with turnstone-doctor (#718)
* refactor(doctor): replace turnstone-bootstrap with turnstone-doctor turnstone-bootstrap was an LLM setup wizard for Day-0; run.sh now owns install. Repurpose its LLM/conversation plumbing into turnstone-doctor — a diagnose-only tool for a running cluster. - Preflight detects the install kind (docker-compose/systemd/pip/source) from config.toml + TURNSTONE_* env, with secret redaction. - Self-configuring brain resolves the cluster's own model from config/env/storage read-only (no migrations, no create_all), falling back to interactive selection; the attempt itself is the LLM-backend health check. - Deterministic version check: installed version, cluster drift via the console's authoritative /health, and latest upstream stable/experimental (offline-safe). - Read-only diagnostic tools (read_file, compose/systemd/journal, http_health, check_llm_backend, node_health, finish) behind one secret-scrubbing chokepoint; no generic shell, so read-only is structural. - node_health reaches a node the right way for the detected install kind (exec-into-container for compose, direct HTTP otherwise), overridable per node for mixed clusters. - mTLS-aware: forwards [database] SSL params and reports node-mesh mTLS instead of mislabelling healthy nodes "unreachable". init_storage gains a backward-compatible create_tables override for read-only opens. Entry point turnstone-bootstrap -> turnstone-doctor; README/QUICKSTART/ architecture/docker docs, the bundled compose header, run.sh, and the CI smoke updated. CHANGELOG deferred. * fix(doctor): address Copilot + CodeQL review findings on #718 Validated all seven review findings (none false positives) and fixed: - check_llm_backend now applies the same scheme / metadata-host guard as http_health (extracted to _assert_safe_http_url), so a model-supplied base_url can't be steered at the cloud metadata endpoint or a file:// URL. - node_health no longer double-appends the default port when the operator passes host:port (regression: 10.0.0.5:8081 -> http://10.0.0.5:8081:8080). - node_health install_type enum uses "git-source" to match the label the rest of the module and the prompt/report show the model (a schema-strict provider would otherwise reject the value the model is told to use). - _read_api_creds takes base_url + api_key as a unit from the first config source that defines either field, then env-fills, instead of splicing the two across different config files into a pair that exists in no real config. - _mask_secrets masks assignment-shaped content inside comment lines, so a commented-out real secret can't leak through read_file / the report; prose comments (no KEY=value shape) still pass through untouched. - drop the mixed import styles CodeQL flagged in doctor.py and test_doctor.py. Adds 5 tests; ruff + mypy clean; full doctor suite passes (129). |
||
|
Patrick Buckley
|
04b3a3abe4 |
feat(deploy): systemd units for a bare-metal turnstone-server node
Hardened service + slice + node-identity drop-in template + a README for running a turnstone-server outside Docker that joins the compose cluster — the production-shaped counterpart to the one-liner in docs/docker.md. Secrets stay in config.toml; per-host identity + cluster URLs go in the drop-in. The README notes the cross-host mTLS caveat (turnstonelabs/lacme#22). |
||
|
Patrick Buckley
|
1f61350545 |
feat(compose): let bare-metal turnstone-servers join the cluster (incl. mTLS)
A turnstone-server running outside the compose network ("bare-metal", e.g. a
local-GPU box) couldn't fully join: it can't resolve the in-cluster console
(console:8090) to enroll its mTLS cert, and SearxNG was unreachable for
web_search. Only Postgres was published.
Publish the console's plain-HTTP ACME endpoint (:8090) and SearxNG (:8081)
alongside Postgres, all bound via one knob TURNSTONE_HOST_IP (default 127.0.0.1
-- nothing new on the LAN; set it to the host's LAN IP for a node on another
machine). Postgres keeps honoring the legacy POSTGRES_BIND as a fallback, so
existing .env files don't break.
The node's TLS client now honors TURNSTONE_CONSOLE_URL so a bare-metal node can
point at the published ACME endpoint instead of the unreachable in-cluster name
(empty = in-cluster service discovery, unchanged).
Docs (docker.md, tls.md), the run.sh-generated .env, and the bootstrap wizard
updated to match. The advertised host is the cert's primary SAN and the console
collector dials it back, so mTLS hostname verification holds both ways.
|
||
|
Patrick Buckley
|
1728a4c0af |
feat(web-search): replace Tavily/DuckDuckGo backends with self-hosted SearxNG
Drop the Tavily and DuckDuckGo (ddgs) web_search backends for a single self-hosted SearxNG service bundled into the docker-compose stacks. Core: - New SearXNGClient + _format_searxng; rewrite resolve_web_search_client to (backend, searxng_url, searxng_engines, ...). MCP backend + oauth_user guard unchanged. _resolve_search_client follows storage -> toml -> env -> default precedence (explicit "" disables, via ConfigStore.stored_keys()). - Drop the Tavily-era topic=finance (no SearxNG category); topic is now general/news. Settings/config: - Remove tools.tavily_api_key, get_tavily_key, $TAVILY_API_KEY, [api].tavily_key. - Add tools.searxng_url (default http://searxng:8080) + tools.searxng_engines, with get_searxng_url/get_searxng_engines. Compose + bundled config: - Internal-only searxng service (no published API port, :ro config, /healthz healthcheck, persistent searxng-cache volume) in both stacks; bundle turnstone/deploy/searxng/settings.yml (JSON output on, limiter off). - Caddy serves the SearxNG web UI on :8444 (dev: localhost-only; prod: opt-in). - bootstrap extractor + wheel packaging updated. Deps: drop the ddg extra + ddgs mypy override (regenerates uv.lock, removing the lxml/h2/brotli transitives). Docs: tools/docker/architecture/openshell + diagrams + config example + CHANGELOG; docs/docker.md carries the AGPL-3.0 §13 operator note. BREAKING: tools.web_search_backend no longer accepts "tavily"/"ddg"; tools.tavily_api_key and the ddg extra are removed. Run the bundled SearxNG (ships in the compose stacks) or set TURNSTONE_SEARXNG_URL to an external instance. Closes #545 |
||
|
Patrick Buckley
|
631f1b0021 |
feat(compose): cluster-by-default Caddy-fronted stack with bare-metal join
`docker compose up` from a clone builds one image and brings up the whole stack — PostgreSQL, console, Caddy, channel, and 10 server nodes — sharing one Postgres so the console discovers every node. The dashboard is reachable only through Caddy (HTTP/2 avoids the browser's 6-connection cap on the dashboard's SSE streams); the console's plain-HTTP port is no longer published. Postgres binds 127.0.0.1 so a bare-metal turnstone-server can join the cluster — the bare-metal overlay is folded in and removed. Insecure dev defaults keep it zero-config; the bundled production stack mirrors the shape but pulls ghcr images and requires real secrets. Move the Caddyfile under turnstone/deploy so it ships in the wheel; update docs, QUICKSTART, and .env.example to match. |
||
|
Patrick Buckley
|
752fea0fdd |
feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher (#505)
* feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher Add a console-side `NotifyDispatcher` that holds a dedicated PostgreSQL `LISTEN` connection and fans wake-ups out to per-channel handlers on a separate dispatch thread. Cluster collector subscribes to a new `services` channel and runs node discovery reactively — new-node / graceful-deregister visibility drops from up-to-60 s to ~500 ms on Postgres, with the 60 s discovery loop retained as the backstop for crash-shaped node loss (NOTIFY only fires on real writes). Storage layer gains a uniform `notify` / `listen` API: - PostgreSQL: real `pg_notify` / `LISTEN` on a dedicated session-mode connection that bypasses pgbouncer (mandatory: pgbouncer is required in transaction-pool mode per docs, which is incompatible with LISTEN). - SQLite: in-process fan-out + synthetic-sweep fallback so consumer code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or `[database] listen_url` in config.toml) points the dispatcher's connection direct-to-Postgres. Defaults to the main DB URL when unset. Migration 053 installs the `services_notify` trigger; it filters heartbeat-only UPDATEs in-trigger so the 30 s × N-nodes heartbeat tick stays quiet, while INSERT, DELETE, and url/metadata-changing UPDATE still fire. Dispatcher detail: - Two threads: listener (drains stream → bounded queue) and dispatch (invokes handlers under exception suppression). Same-channel notifies coalesce per dispatch batch so an N-node deploy burst is one `_discover_nodes` per channel. - Reconnect uses exponential backoff (1 s → 30 s cap). After any successful reopen — whether the prior failure was a stream-poll error or a connect / initial-LISTEN error — one synthetic Notify with payload="reconcile" is enqueued per channel so handlers re-read on the same code path they use for real events. Future consumers (ConfigStore live reload, scheduler immediate dispatch, audit live-tail) plug in by adding their channel to the dispatcher's construction list. Tests: 22 dispatcher tests (incl. reconnect + coalescing under stub storage), 7 SQLite notify-stream tests, 4 PG-gated trigger-filter tests, 4 collector wire-in tests. All pass; ruff + mypy clean. * fix(notify): address Copilot review on #505 - _sqlite.py: SQLiteBackend.listen() now de-dupes channel names via dict.fromkeys before constructing the stream — duplicates would otherwise register the queue twice and double-deliver each notify. - _sqlite.py: SQLiteBackend.listen() gains a keyword-only sweep_interval parameter (defaults to _SQLITE_NOTIFY_SWEEP_INTERVAL) — matches what the comment at the constant already promised, and lets future consumers without their own polling timer pick a tighter cadence without reaching into private stream attributes. - _sqlite.py: documented the `except queue.Empty: pass` end-of-drain termination so it's not mistaken for swallowing an unexpected error. - _postgresql.py: docstring referenced :func:`_pg_listen_url` which was renamed to _resolve_pg_listen_url during PR development. - notify_dispatcher.py: module docstring referenced a non-existent _bootstrap_console_subsystem; wire-in is at console/server.py::main. Refuted (no change, false positives from github-code-quality bot): - 4× "Statement has no effect" on Protocol-method `...` ellipsis bodies (idiomatic Python Protocol declaration, not dead code). - 2× "Mixed import style" in tests — `import ... as nd_mod` is intentional to allow attribute assignment for monkey-patching the module's `_RECONNECT_BACKOFF_INITIAL` constant inside try/finally. |
||
|
Patrick Buckley
|
471d1a3311 |
docs: audit documentation for 1.4 / 1.5 state
Systematic pass over every doc under docs/, the root-level README /
QUICKSTART / CONTRIBUTING, and the PlantUML diagrams. Memory and docs
had drifted against the code since 1.2 — this catches them up to the
1.4.0 release and the 1.5.0a1 experimental line.
User-facing fixes
- README: fix broken docs/mcp.md link (→ mcp-registry.md); channel
gateway entry reflects shipped Discord + Slack adapters instead of
"Slack/Teams planned"; diagrams table mentions both.
- QUICKSTART: docs/*.md relative links were wrong from the repo root;
wizard version bumped from 0.5.4.
- CONTRIBUTING: add dev extra plus the ruff / mypy / pytest commands
we actually expect before push.
Reference docs
- architecture.md: 19 tool schemas (was 15), 18 admin tabs (was 14),
turnstone-bootstrap added to entry-points table, OpenAI provider
file split (chat/responses/common) documented, 38 SDK event
dataclasses (was 27 and referenced deleted mq/protocol.py), Slack
adapter + multi-adapter gateway, plan_agent/task_agent naming,
governance admin-panel rewrite.
- api-reference.md: full attachment endpoints (POST/GET/content/
DELETE on /v1/api/workstreams/{ws_id}/attachments) plus the
multipart mode on POST /v1/api/workstreams/new.
- channels.md: Slack Setup section (Socket Mode app creation, OAuth
scopes, tokens), Slack CLI/env reference in config table, combined-
adapter architecture diagram.
- console.md: 18-tab listing (was 13) with Channels/Models/Nodes/TLS
descriptions and ConfigStore live-edit note.
- docker.md: Slack env vars block; image entry-point list now
includes turnstone / turnstone-bootstrap.
- sdk.md: attachments methods on the server client, attachments
example (upload-then-send and at-creation), event count fixed.
- releasing.md: four-track table (stable/1.0, 1.3, 1.4 + main 1.5);
promotion workflow uses 1.5 / 1.6 numbering.
- settings.md: plan_model / task_model / plan_effort / task_effort
overrides section.
- governance.md: skill naming (/skill, `skill` field — not /template),
Prompts/Judge tabs called out.
- security.md: two-token-types wording; src claim values match the
AuthResult source strings actually emitted.
- mcp-registry.md: SDK package name is @turnstone/sdk.
- tools.md: plan / task renamed to plan_agent / task_agent in the
section headings and summary table; primary-key table matched.
- design/consistent-hash-ring.md: dead direct-http-transport.md
pointer redirected to architecture.md.
Diagrams
- 02-package-structure: drop phantom chat.py entry point, add admin
and bootstrap, add slack/bot.py, rename channels/gateway.py →
channels/cli.py.
- 16-channel-architecture: Slack is no longer "(future)", add a
SlackBot class and the slack-bolt Socket Mode edges; wire the new
bot into ChannelService. PNGs regenerated from both puml sources.
|
||
|
Patrick Buckley
|
2a05ba5915 |
fix: standardize database env vars on TURNSTONE_DB_* naming (#348)
* fix: standardize database env vars on TURNSTONE_DB_* naming compose.yaml used DB_BACKEND/DATABASE_URL in .env which got mapped to TURNSTONE_DB_BACKEND/TURNSTONE_DB_URL inside containers. Running bare- metal required the TURNSTONE_ prefix, but docs didn't explain this. Eliminate the indirection — use TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL everywhere (compose, .env, bare-metal, docs, bootstrap wizard). * fix: update .env.example to use TURNSTONE_DB_* naming |
||
|
Patrick Buckley
|
62d2a0fe6a |
fix: remove non-auth support from bootstrap wizard (#274)
* fix: remove non-auth support from bootstrap wizard Auth is now mandatory for all deployments. Remove the TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN required in the wizard's system prompt. * fix: remove auth disable support from runtime and infra Remove AuthConfig.enabled field — auth is always on. Drop TURNSTONE_AUTH_ENABLED env var, config toggle, and the check_request bypass. Update compose.yaml, Helm chart, Terraform, docs, and tests to match. * feat: deprecate config tokens, require JWT secret, prefer JWT auth Phase 1 of config-token removal: - load_jwt_secret() now exits with error if no secret is configured (was: silently auto-generated ephemeral secret) - _authenticate_token() logs deprecation warning on config token use - CLI /cluster commands use ServiceTokenManager when JWT secret is set - turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set - Update bootstrap wizard, docker.md, security.md to mark TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required - Console test fixtures use auth token + headers (auth always enforced) * feat: add service scope for inter-service JWT auth Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens bypass require_permission() RBAC checks, replacing the old empty-user-id bypass that config tokens relied on. All ServiceTokenManager instances that need admin access now include "service" in their scopes (console proxy, channel gateway, CLI, admin CLI). Read-only services (collector, notification) unchanged. * feat: phase 2 config token deprecation - SDK doc examples now show API tokens (ts_) instead of config tokens - Remove _get_config_token() from admin CLI (dead code) - Block config token exchange in handle_auth_login — only password and API token login allowed - Update login tests to use password-based auth instead of config token exchange * feat: phase 3 — remove config tokens entirely Complete removal of config-file token authentication: - Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch branch, and config token loading from load_auth_config() - Remove auth_config parameter from _authenticate_token() and check_request() — callers updated throughout - Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts, Terraform, turnstone.example.toml - Remove --auth-token CLI flags from turnstone, turnstone-admin, and turnstone-console - Simplify console main() — always use ServiceTokenManager (no fallback to static tokens) - Delete config-token-specific tests, rewrite check_request and integration tests to use JWT auth with proper audience claims - Remove all config token references from docs (security.md, docker.md, sdk.md, console.md, architecture.md, bootstrap prompt) * fix: address code review findings - Fix 33 broken tests: add JWT auth to test_api_versioning, test_console_routing_proxy, test_tls_admin, test_tls_manager, test_server_live (jwt_secret + audience-scoped auth headers) - Add TestRequirePermissionServiceScope: 4 tests covering the service scope RBAC bypass path - Remove stale comments referencing config tokens in auth.py and console/server.py - Remove dead proxy_auth_token parameter from console create_app() and static token fallback in _proxy_auth_headers() - Remove TURNSTONE_AUTH_TOKEN from env.py scrub list * fix: address Copilot review — JWT audience, compose require secret - CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager (console validates audience, JWTs without it were rejected) - Admin CLI tls-list: same audience fix - compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset - SDK console: fix default port from 8081 to 8090 * test: add auth enforcement tests for TLS admin endpoints 5 new tests: unauthenticated requests return 401 (list, renew, delete), read-only-scoped requests return 403 (renew, delete). Closes the TLS auth enforcement test gap noted in PROGRESS.md. * fix: address remaining Copilot review feedback - Fix token_source="config" → "test" in TLS test fixtures - Fix AuthResult.token_source docstring to include service origins - Require TURNSTONE_JWT_SECRET in cluster compose profile (:?) - Helm: add auth.jwtSecret + auth.existingSecret values, wire TURNSTONE_JWT_SECRET into secret.yaml and both deployments - Terraform: replace auth_token with jwt_secret variable + secret, remove orphaned auth_token resources and IAM reference - Remove [[auth.tokens]] from security.md config example * fix: address full code review — 10 findings Critical: - Terraform: replace concat(common_env, auth_env) with common_env (auth_env local was removed but still referenced) - Channel gateway: remove hmac static token auth from _check_auth(), use JWT-only validation. Remove --auth-token CLI arg from channel - Rebalancer: add token_manager support so migration requests carry JWT auth (was sending unauthenticated POST to /internal/migrate) Major: - Guard _permissions_to_scopes() against "service" privilege escalation from DB role permissions - Remove dead AuthConfig class, load_auth_config(), and all auth_config parameters from create_app() signatures - Helm: inject JWT secret for both inline and existingSecret paths Minor: - Remove dead auth_token param from ClusterCollector - Remove empty TestLoadAuthConfig class - Short JWT secret now exits instead of warning - Compose: add generation command comment above JWT_SECRET - Clean stale config token references from 6 doc files - Clean stale AUTH_TOKEN reference from bootstrap wizard prompt * fix: remove remaining stale config token references from docs - channels.md: remove --auth-token from options table - oidc.md: remove "config-file tokens still work" claim - security.md: remove config token section, fix JWT secret docs (now required/exits, no ephemeral fallback), remove hmac from ASCII diagram, remove --auth-token reference |
||
|
Patrick Buckley
|
9a518657a3 |
feat: replace console HTTP polling with persistent SSE streams (#266)
* feat: replace console HTTP polling with persistent SSE streams Console collector now subscribes to each server node's /v1/api/events/global SSE stream for real-time state updates instead of polling /v1/api/dashboard and /health every 15 seconds. Server changes: - Emit ws_created/ws_closed events on global queue from create/close handlers - Add node_snapshot on SSE connect (workstreams, health, aggregate) - Add ?expected_node_id= identity verification (409 on mismatch) - Add health_changed callback to BackendHealthMonitor circuit breaker - Add periodic aggregate emitter thread (10s) Console collector changes: - Single asyncio event loop on one thread multiplexes all SSE connections (scales to 1000+ nodes vs thread-per-node) - Discovery loop spawns/cancels async SSE tasks per node - Snapshot reconciliation on connect, delta application for live events - Fix ws_state→cluster_state event type mismatch - Remove polling code (poll_interval, max_poll_workers, --poll-interval CLI) SDK changes: - Add NodeSnapshotEvent, HealthChangedEvent, AggregateEvent dataclasses - Add stream_node_events() method (async + sync) * fix: address review feedback on node event streams - Fix stop() to let SSE manager exit naturally instead of force-stopping the event loop (ensures finally cleanup runs) - Guard against empty/invalid SSE data from ping frames - Treat missing node_id as identity mismatch (409) when expected_node_id is provided - Fix stale docstring on _update_metrics |
||
|
Patrick Buckley
|
2bb55590bf |
feat: replace Redis MQ with direct HTTP transport (Phase 1)
Delete the entire turnstone/mq/ package (broker, bridge, protocol, client) and turnstone/sim/ package. Remove Redis as a dependency. Channel gateway and console now communicate with server nodes via direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues. Single-node deployments work with zero infrastructure beyond the database. Key changes: - Channel adapters use httpx POST for create/send/approve/close and httpx-sse for per-workstream event streaming - Console collector discovers nodes via services table instead of Redis SCAN - Console scheduler dispatches tasks via HTTP POST with DB-based leader election - Server registers in services table with 30s heartbeat - Server accepts optional ws_id in create request (for Phase 2 console-generated routing) - SDK events gain IntentVerdictEvent and OutputWarningEvent types - All docs, examples, bootstrap wizard updated 63 files changed, -5968 net lines (Redis transport fully removed) |
||
|
Patrick Buckley
|
414eb52d67 |
feat: raise scaling limits for 1000-node clusters (#129)
* feat: raise scaling limits for 1000-node clusters Raise hardcoded limits throughout the codebase so clusters up to 1000 nodes work without configuration changes. Scaling limits: - max_workstreams default 10 → 50 (configurable via settings) - Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit) - MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers) - Console SSE queue 500 → 2000, server global SSE queue 500 → 1000 - httpx proxy pool: explicit max_connections on both proxy clients - PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries) - Redis pool: explicit max_connections=200 on both sync and async brokers Performance optimizations: - Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET - Collector poll: raise thread pool to 200 (matches fan-out limit) - Server SSE: dedicated ThreadPoolExecutor(200) for queue polling - Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling Bug fixes: - Settings reload notification was silently failing (called .get() on tuple) - Watch fan-out only queried 500 nodes instead of full cluster New cluster settings (configurable via admin Settings tab): - cluster.node_fan_out_limit (default 200, range 10-1000) - cluster.mcp_max_servers (default 200, range 1-2000) Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale. Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10). Updates architecture, console, docker, settings, and API reference docs. * fix: add image tag to compose anchors to avoid redundant builds All cluster/stress services inherit `build:` from the anchor, causing Docker to attempt 200+ separate builds. Adding `image: turnstone:local` means Docker builds once and all services reuse the cached image. * fix: address Copilot review feedback on scaling PR - Remove magic number in get_all_nodes (limit=None instead of 2**31) - Size httpx proxy pool from fan-out limit setting (not hardcoded 250) - Cap cluster.node_fan_out_limit max_value to 500, mark restart_required - Convert _publish_config_change from sync to async (was blocking event loop) - Use shutdown(wait=True, cancel_futures=True) for SSE executor * fix: add PostgreSQL env vars to cluster bridge anchor Bridges initialize storage for auth/migrations but the bridge anchor was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all bridges to fall back to SQLite. With 100 bridges sharing the same volume, concurrent SQLite migrations corrupt the database. * fix: address Copilot round 2 + PG connection exhaustion at startup Copilot feedback: - Raise cluster.node_fan_out_limit max_value to 1000 (matches target) - Cache fan-out limit on app.state at startup instead of re-reading DB per request (pool and semaphore now use the same value consistently) - Remove unused params from _publish_config_change Stress cluster fix: - Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS) to handle 200 processes connecting simultaneously at startup - Bump PG shared_buffers to 128MB and memory limit to 1G to match - Add DB env vars to production bridge service * fix readme * fix: startup resilience for large clusters Server no longer crashes when LLM backend is unreachable at startup. detect_model() accepts fatal=False, returning (None, None) so the server starts in degraded mode with circuit breaker open. The health monitor will detect when the backend becomes available. Migration runner retries with jittered exponential backoff (up to 10 attempts) when PostgreSQL rejects connections during startup stampedes. Collector httpx pool sized to match poll workers (was using default of 100 connections with 200 workers). Also addresses Copilot round 2: - Raise cluster.node_fan_out_limit max_value to 1000 - Cache fan-out limit on app.state at startup - Remove unused params from _publish_config_change - Add DB env vars to production bridge service * fix: replace silent error suppression with structured logging Audit and fix 30+ instances of silently swallowed exceptions across 8 files. No-raise contracts are preserved — all changes add logging while keeping the same return-value behavior. memory.py (26 changes): Every storage operation now logs on failure. Previously the entire persistence facade had zero logging — messages, workstream state, and structured memories could silently stop being saved. server.py: Usage recording failures now log at warning (was pass). Global SSE fan-out errors log at debug (was pass). console/server.py: Config reload notification logs per-node failures at warning. Settings read fallbacks log at warning with the default value used. auth.py: User existence check logs at warning (was pass). Setup rollback failures log at error (was suppress). OIDC state cleanup logs at debug (was suppress). mcp_client.py: DB-managed MCP server list failure logs at warning (was pass). collector.py: Node poll failure upgraded from debug to warning with exc_info. Health fetch failure logs at debug with exc_info (was silent). bridge.py: Best-effort plan rejection logs at warning (was suppress). Malformed SSE data logs at debug (was suppress). session.py: Tool output UI callback failure logs at debug (was suppress). * fix: stagger collector poll with deterministic per-node jitter Each node gets a stable offset within the first half of the poll interval, derived from hashing the node_id against a Mersenne prime (2^31 - 1). This spreads HTTP requests across the cycle instead of firing all 100+ at the same instant. Also raises poll interval from 10s to 15s and HTTP timeout from 5s to 30s for large-cluster resilience. * fix: add startup jitter to bridge heartbeat and health monitor probe Bridge heartbeat: deterministic per-node jitter (from node_id hash) spreads initial registration across the first quarter of the heartbeat TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead of all firing at T=0. Health monitor probe: deterministic per-process jitter (from PID hash) spreads initial LLM backend probes across half the probe interval. At 100 servers with 30s interval, probes spread across 15s instead of all hitting the LLM at T=30. Both use the same Mersenne prime hashing approach as the collector poll jitter for consistency. * fix: split collector httpx timeout and raise keepalive pool Use separate connect/read/write/pool timeouts instead of a single 30s for all phases. Raise keepalive connections from 50 to 200 so the collector reuses TCP connections across poll cycles instead of constantly tearing down and re-establishing them. * fix: narrow detect_model return type for CLI and eval callers detect_model() now returns tuple[str | None, int | None] to support fatal=False. CLI and eval always use fatal=True (the default), which guarantees a non-None model or SystemExit. Add assert to narrow the type for mypy. |
||
|
Patrick Buckley
|
50277cd4de | Add 10-node cluster profile to Docker Compose | ||
|
Patrick Buckley
|
a6e929b0a0 |
Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24) Bidirectional channel adapter framework connecting external messaging platforms to turnstone workstreams via Redis MQ. Discord ships as the first adapter; the protocol supports future Slack/Teams integrations. Channel framework: - ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping - AsyncRedisBroker with single dispatch loop and per-channel ordered workers - channel_routes table (migration 003) for persistent route storage - 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD) - Unified turnstone-channel gateway entry point, loads adapters by config - Message chunking, approval formatting, plan review formatting Discord adapter: - discord.py v2.4+ bot with thread-per-@mention model - Slash commands: /link (modal), /unlink, /ask, /status, /close - Persistent button views for tool approval and plan review - Streaming responses via edit-in-place (1.5s interval) - Stale route detection and atomic session resume via resume_session field - SessionResumedEvent confirmation back to channel - Auto-approve support (blanket + per-tool list) Atomic session resume: - resume_session field on CreateWorkstreamMessage for single-request resume - Server resumes session during POST /v1/api/workstreams/new atomically - Bridge emits SessionResumedEvent to per-workstream channel - WorkstreamCreatedEvent extended with resumed/session_id/message_count - Server UI dashboardResumeSession simplified to single request - Pruned sessions fall back gracefully to fresh start Service auth: - Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET - Bridge: approve scope (1 week). Console collector: read. Proxy: write. Console admin: - Channels tab with per-user view, force-link modal, unlink - 3 admin API endpoints for channel user management - Styled confirm modals replacing browser confirm() dialogs Bug fixes: - AsyncRedisBroker: replaced per-channel listener tasks with single dispatch loop + per-channel queue workers (fixes message stealing race) - Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates - Bridge: _active_sends tracked for initial messages (fixes missing TurnCompleteEvent and unfinalized streaming messages) - Bridge: HTTP calls moved outside lock scope in approval handlers - Bridge: _handle_send cleans up _active_sends on HTTP/server errors - Formatter: reads server SSE format (func_name/preview) with fallback Docs, SDK, tests: - docs/channels.md setup guide, architecture diagram 16 - Updated api-reference.md, architecture.md, console.md, docker.md - Python SDK: resume_session param on create_workstream (async + sync) - TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces - OpenAPI schema: resume_session request, resumed/message_count response - 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing, 18 discord, 12 resume flow) — 1120 total passing * Fix CI lint/typecheck failures and address Copilot review feedback (#24) Lint: fix import ordering, remove unused imports, use contextlib.suppress. Mypy: explicit postgresql dialect import, add discord module overrides for optional-dependency CI environments. Copilot: fix double-escaping in admin confirm modals, return resolved session_id from server resume response, fix channel_routes diagram schema, use atomic setdefault for routing locks, add post-insert race guard in admin channel create, support SSE format in auto-approve check, update identity linking note in architecture diagram. * Fix remaining mypy call-arg errors for discord.py optional dependency Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class definitions that fail when discord.py is not installed in CI. |
||
|
Patrick Buckley
|
047680d669 |
Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23) JWT-based authentication with three token types: config-file (hmac, backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs (HS256, 24h expiry). Username:password login via bcrypt. Hierarchical scopes: read < write < approve. New tables: users (username, password_hash), api_tokens (token_hash, scopes, expires), channel_users (future channel integrations). user_id column added to sessions and workstreams for attribution. Console owns admin CRUD (6 endpoints under /api/admin/). Server validates JWTs locally with shared signing secret. Public /api/auth/setup endpoint for first-time admin creation (atomic, only works with zero users). turnstone-admin CLI for user/token management. Admin console UI: Users and Tokens tabs with full CRUD modals, scope badges, token show-once with clipboard copy, keyboard accessibility (focus traps, Escape, arrow key tabs, ARIA roles). Login UI redesigned: username:password primary, token toggle for legacy, setup wizard auto-detected via /api/auth/status. Python + TypeScript SDKs updated with login(username, password), authStatus(), setup(). New docs/security.md + diagram 15-auth-architecture.puml. All existing docs updated. OpenAPI specs include all new endpoints. 64 new tests (1023 total). Dependencies: PyJWT, bcrypt. * Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review Address Copilot review feedback: escape double quotes in escapeHtml() to prevent XSS in HTML attributes, add JWT validation fallback so config tokens containing dots still work, add user_id to AuthLoginResponse schema, return created field from admin_create_user, and correct five documentation files to match actual API behavior. |
||
|
Patrick Buckley
|
f02972c11d |
Add provider-native web search with Tavily fallback (#13)
* Add provider-native web search with Tavily fallback Replace client-side Tavily web search with provider-native implementations: - Anthropic: inject web_search_20250305 server-side tool, handle server_tool_use / web_search_tool_result streaming blocks, emit info_delta for search status display - OpenAI: inject web_search_options for gpt-5-search-api, format url_citation annotations as footnote sources - Local/vLLM: preserve existing Tavily-based web_search tool as fallback Add supports_web_search to ModelCapabilities and info_delta to StreamChunk. Remove end-of-life GPT-4o model entries from capability tables. Update docs, diagrams, and README. 88 provider tests (32 new). * Fix Copilot PR #13 review: capture streaming url_citation annotations Accumulate url_citation annotations during OpenAI streaming and emit formatted citations as a final info_delta chunk after the stream ends. Previously annotations were only captured in non-streaming mode, so search model users in the interactive path never saw citation sources. |
||
|
Patrick Buckley
|
9be155b97a |
Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup - Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files - Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow - Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems - Refresh README and docs with badges, diagram links, and current descriptions - Refactor test_server_live.py with mock streaming helpers for deterministic CI testing - Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2) Console dashboard: - Move state indicators from top cards to fixed bottom status bar with cluster metrics - Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000) - Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent, LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance - Add render cache, stale indicator, active filter highlight, loading states Server web UI: - Apply matching Instrument Panel aesthetic for visual consistency with console - Fix branding (pcode → turnstone), extract inline styles to CSS classes - Rename pcode localStorage keys and history state to turnstone Legacy cleanup: - Remove persona-model-specific --persona flag and /persona slash command - Remove model_identity from chat_template_kwargs (vLLM-specific mechanism) - Refactor plan agent to use standard developer message instead of model_identity - Remove dead code (unused date/has_tools variables, noqa suppressions) * Fix CI typecheck: add mypy overrides for optional sympy/numpy imports The math sandbox optionally imports sympy and numpy at runtime (try/except ImportError). In CI these packages are not installed, so mypy raises import-not-found rather than import-untyped. Add mypy overrides to ignore missing imports for these optional dependencies. * Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity - Change #node-table from role="tree" to role="list" and group elements from role="treeitem" to role="listitem" (proper ARIA semantics) - Include currentView and currentFilter.state in renderStatusBar cache key so active pill highlight updates when switching views - Align pulse animation to 0.35 opacity (already applied in CSS) |
||
|
Patrick Buckley
|
0d6252dd7d | Initial commit — turnstone multi-node AI orchestration platform. |