* fix: harden MCP client against misbehaving servers
Misbehaving/failed/misconfigured MCP servers could peg CPU at 100% due
to anyio cancel-scope busy-loops (SDK #2147), uncancelled orphaned
futures, and missing application-layer resilience.
Five fixes:
1. Cancel orphaned futures on timeout — future.cancel() in all sync
bridge methods prevents coroutine accumulation on the event loop
2. Per-server circuit breaker — 3-failure threshold with exponential
cooldown (30s–5min), per-server jitter, auto-reconnect on half-open
probe, McpError excluded (protocol errors from healthy servers)
3. Safe transport stream pre-close — store stream refs and close them
before stack teardown in all error/shutdown paths, preventing the
anyio zero-buffer CPU busy-loop
4. Notification debounce — 5s per-server rate limit on list_changed
refresh storms from buggy servers
5. Periodic refresh backoff with auto-reconnect — disconnected servers
get reconnection attempts with exponential backoff (60s–1hr) instead
of being silently skipped forever
* docs: add MCP resilience section to architecture docs and diagram
Document the circuit breaker, future cancellation, stream pre-close,
notification debounce, and periodic refresh backoff in the architecture
guide and the MCP architecture PlantUML diagram.
* fix: address review — stack leak on transport error, half-open comment
- Widen _connect_one guard to check _per_server_stacks too, not just
_sessions. Transport errors in sync dispatch methods evict the session
but left the stack behind, leaking anyio tasks on reconnect.
- Clarify half-open design: multiple callers are intentionally allowed
through (reconnects serialize on the event loop, first failure re-trips).
* chore: release infrastructure for dual-track stable/experimental
CI/CD changes for the 1.0 release:
- Gate PyPI publish and Docker publish on CI success via workflow_run
- Add docker-publish.yml: builds and pushes to GHCR with smart tagging
(stable gets :X.Y.Z/:X.Y/:stable/:latest, pre-release gets :experimental)
- Add stable/* and v* tags to CI and docker-scan triggers
- Remove stale [mq] extra and types-redis from CI (Redis MQ deleted)
- Remove stale redis from Renovate package rules
Release tooling:
- scripts/release.sh: bump version, uv lock, commit, tag (with --push)
- docs/releasing.md: documents stable/experimental workflow
Docker:
- Add /workspace mount point (WORKSPACE_MOUNT env var, defaults to empty volume)
- Update .env.example: remove stale Redis/auth-token refs, add workspace/model/discord
README:
- Remove beta warning, add hero image and release tracks table
* fix: derive release tag from git instead of workflow_run.head_branch
Use git tag --points-at HEAD after checkout to resolve the release
tag instead of relying on workflow_run.head_branch, which may not
reliably be the tag name for tag-triggered CI runs. Both publish
and docker-publish workflows now skip cleanly when no v* tag exists
at the checked-out commit.
* fix: remove non-auth support from bootstrap wizard
Auth is now mandatory for all deployments. Remove the
TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN
required in the wizard's system prompt.
* fix: remove auth disable support from runtime and infra
Remove AuthConfig.enabled field — auth is always on. Drop
TURNSTONE_AUTH_ENABLED env var, config toggle, and the
check_request bypass. Update compose.yaml, Helm chart,
Terraform, docs, and tests to match.
* feat: deprecate config tokens, require JWT secret, prefer JWT auth
Phase 1 of config-token removal:
- load_jwt_secret() now exits with error if no secret is configured
(was: silently auto-generated ephemeral secret)
- _authenticate_token() logs deprecation warning on config token use
- CLI /cluster commands use ServiceTokenManager when JWT secret is set
- turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set
- Update bootstrap wizard, docker.md, security.md to mark
TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required
- Console test fixtures use auth token + headers (auth always enforced)
* feat: add service scope for inter-service JWT auth
Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens
bypass require_permission() RBAC checks, replacing the old
empty-user-id bypass that config tokens relied on.
All ServiceTokenManager instances that need admin access now include
"service" in their scopes (console proxy, channel gateway, CLI,
admin CLI). Read-only services (collector, notification) unchanged.
* feat: phase 2 config token deprecation
- SDK doc examples now show API tokens (ts_) instead of config tokens
- Remove _get_config_token() from admin CLI (dead code)
- Block config token exchange in handle_auth_login — only password
and API token login allowed
- Update login tests to use password-based auth instead of config
token exchange
* feat: phase 3 — remove config tokens entirely
Complete removal of config-file token authentication:
- Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch
branch, and config token loading from load_auth_config()
- Remove auth_config parameter from _authenticate_token() and
check_request() — callers updated throughout
- Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts,
Terraform, turnstone.example.toml
- Remove --auth-token CLI flags from turnstone, turnstone-admin,
and turnstone-console
- Simplify console main() — always use ServiceTokenManager
(no fallback to static tokens)
- Delete config-token-specific tests, rewrite check_request and
integration tests to use JWT auth with proper audience claims
- Remove all config token references from docs (security.md,
docker.md, sdk.md, console.md, architecture.md, bootstrap prompt)
* fix: address code review findings
- Fix 33 broken tests: add JWT auth to test_api_versioning,
test_console_routing_proxy, test_tls_admin, test_tls_manager,
test_server_live (jwt_secret + audience-scoped auth headers)
- Add TestRequirePermissionServiceScope: 4 tests covering the
service scope RBAC bypass path
- Remove stale comments referencing config tokens in auth.py and
console/server.py
- Remove dead proxy_auth_token parameter from console create_app()
and static token fallback in _proxy_auth_headers()
- Remove TURNSTONE_AUTH_TOKEN from env.py scrub list
* fix: address Copilot review — JWT audience, compose require secret
- CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager
(console validates audience, JWTs without it were rejected)
- Admin CLI tls-list: same audience fix
- compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset
- SDK console: fix default port from 8081 to 8090
* test: add auth enforcement tests for TLS admin endpoints
5 new tests: unauthenticated requests return 401 (list, renew,
delete), read-only-scoped requests return 403 (renew, delete).
Closes the TLS auth enforcement test gap noted in PROGRESS.md.
* fix: address remaining Copilot review feedback
- Fix token_source="config" → "test" in TLS test fixtures
- Fix AuthResult.token_source docstring to include service origins
- Require TURNSTONE_JWT_SECRET in cluster compose profile (:?)
- Helm: add auth.jwtSecret + auth.existingSecret values, wire
TURNSTONE_JWT_SECRET into secret.yaml and both deployments
- Terraform: replace auth_token with jwt_secret variable + secret,
remove orphaned auth_token resources and IAM reference
- Remove [[auth.tokens]] from security.md config example
* fix: address full code review — 10 findings
Critical:
- Terraform: replace concat(common_env, auth_env) with common_env
(auth_env local was removed but still referenced)
- Channel gateway: remove hmac static token auth from _check_auth(),
use JWT-only validation. Remove --auth-token CLI arg from channel
- Rebalancer: add token_manager support so migration requests carry
JWT auth (was sending unauthenticated POST to /internal/migrate)
Major:
- Guard _permissions_to_scopes() against "service" privilege
escalation from DB role permissions
- Remove dead AuthConfig class, load_auth_config(), and all
auth_config parameters from create_app() signatures
- Helm: inject JWT secret for both inline and existingSecret paths
Minor:
- Remove dead auth_token param from ClusterCollector
- Remove empty TestLoadAuthConfig class
- Short JWT secret now exits instead of warning
- Compose: add generation command comment above JWT_SECRET
- Clean stale config token references from 6 doc files
- Clean stale AUTH_TOKEN reference from bootstrap wizard prompt
* fix: remove remaining stale config token references from docs
- channels.md: remove --auth-token from options table
- oidc.md: remove "config-file tokens still work" claim
- security.md: remove config token section, fix JWT secret docs
(now required/exits, no ephemeral fallback), remove hmac from
ASCII diagram, remove --auth-token reference
* fix: MCP tools not surfacing after Sync to Nodes, update Anthropic tool search
Three fixes:
1. session_factory closure captured mcp_client=None when no --mcp-config
was passed at startup. internal_mcp_reload created a new MCPClientManager
on app.state but the factory never saw it. New workstreams got 0 MCP tools.
Fix: mutable _mcp_ref list shared between factory and reload handler.
2. Anthropic dropped the date suffix from tool_search_tool_bm25_20251119
and now requires name == type. Updated constant and tool definition.
3. Add diagnostic logging around API errors (provider, model, base_url,
message counts, full exception chain) and workstream resume (pre/post
provider state, alias resolution warnings).
Also adds Node.js 24 LTS to Dockerfile via multi-stage copy for npx-based
MCP servers.
* fix: address Copilot review — set_storage on reload, sanitize log output
- Call mcp_mgr.set_storage(storage) when internal_mcp_reload creates a
new MCPClientManager so prompt sync works for post-startup servers
- Strip query params from base_url before logging (may contain API keys
in some vLLM deployments)
- Split API error logging: concise warning (type names only) + separate
debug with exc_info=True for full traceback when needed
* chore: remove DDG MCP sidecar, web_search uses built-in ddgs client
The DuckDuckGo MCP server container is redundant — the built-in
DuckDuckGoClient (via ddgs package, included in all extras) auto-detects
when no Tavily key is configured. Removes the ddg-search service,
ddgCluster profile, and mcp-ddg.json config file.
* feat: replace console HTTP polling with persistent SSE streams
Console collector now subscribes to each server node's /v1/api/events/global
SSE stream for real-time state updates instead of polling /v1/api/dashboard
and /health every 15 seconds.
Server changes:
- Emit ws_created/ws_closed events on global queue from create/close handlers
- Add node_snapshot on SSE connect (workstreams, health, aggregate)
- Add ?expected_node_id= identity verification (409 on mismatch)
- Add health_changed callback to BackendHealthMonitor circuit breaker
- Add periodic aggregate emitter thread (10s)
Console collector changes:
- Single asyncio event loop on one thread multiplexes all SSE connections
(scales to 1000+ nodes vs thread-per-node)
- Discovery loop spawns/cancels async SSE tasks per node
- Snapshot reconciliation on connect, delta application for live events
- Fix ws_state→cluster_state event type mismatch
- Remove polling code (poll_interval, max_poll_workers, --poll-interval CLI)
SDK changes:
- Add NodeSnapshotEvent, HealthChangedEvent, AggregateEvent dataclasses
- Add stream_node_events() method (async + sync)
* fix: address review feedback on node event streams
- Fix stop() to let SSE manager exit naturally instead of force-stopping
the event loop (ensures finally cleanup runs)
- Guard against empty/invalid SSE data from ping frames
- Treat missing node_id as identity mismatch (409) when expected_node_id
is provided
- Fix stale docstring on _update_metrics
- 404 retry: use blocking lock acquire so retry waits for cache refresh
to complete instead of skipping on contention
- 404 retry: surface httpx.HTTPError as 502 instead of suppressing it
and returning the original 404
- channel router: pass auto_approve_tools to create_workstream calls
(was silently dropped for console-routed creates)
- api-reference.md: document all /v1/api/route/* console routing proxy
endpoints and console /metrics
- router.route(): validate ws_id length and hex format before bucket
extraction, raise NoAvailableNodeError instead of ValueError
- router: expose version as public property, collector uses it instead
of accessing _version directly
- memory.py: deduplicate _bucket_of with canonical bucket_of from
hash_ring module
- architecture SVG: reroute direct/SSE lines below console to avoid
crossing over the console box
Move the consistent hash ring implementation (FNV-1a, virtual nodes,
bisect lookup) from code to docs/design/consistent-hash-ring.md as a
forward-looking reference for future scalability work.
The current rebalancer uses weight-proportional distribution (simpler,
exact splits, no hash variance). The ring algorithm is documented with
test vectors, stability properties, and a comparison table for when
the ring approach becomes advantageous (large clusters, decentralized
routing, cross-language determinism).
hash_ring.py retains: RING_SIZE, bucket_of(), RingNode, NoAvailableNodeError
(all actively used by router and rebalancer).
Delete the entire turnstone/mq/ package (broker, bridge, protocol,
client) and turnstone/sim/ package. Remove Redis as a dependency.
Channel gateway and console now communicate with server nodes via
direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues.
Single-node deployments work with zero infrastructure beyond the
database.
Key changes:
- Channel adapters use httpx POST for create/send/approve/close
and httpx-sse for per-workstream event streaming
- Console collector discovers nodes via services table instead of
Redis SCAN
- Console scheduler dispatches tasks via HTTP POST with DB-based
leader election
- Server registers in services table with 30s heartbeat
- Server accepts optional ws_id in create request (for Phase 2
console-generated routing)
- SDK events gain IntentVerdictEvent and OutputWarningEvent types
- All docs, examples, bootstrap wizard updated
63 files changed, -5968 net lines (Redis transport fully removed)
* fix: scope-filter memory list/search to current workstream and user
Unscoped memory(action='list') and memory(action='search') returned all
memories across all workstreams. Now applies the same 3-query pattern
(global + current workstream + current user) used by system prompt
injection.
* fix: validate user scope on memory search/list for unauthenticated sessions
Adds _validate_scope guard to search and list prepare paths, matching
save/get/delete. Prevents explicit scope='user' from returning all
user-scoped memories when session is unauthenticated.
* fix: update _get_visible_memories references to _list_visible_memories
* fix: defense-in-depth guard for empty scope_id on search/list
Copilot review: if scope is 'user' or 'workstream' with empty
scope_id, the storage query returns all memories in that scope
across all users/workstreams. The prepare step already validates
via _validate_scope, but add exec-level guard to reject scoped
queries with empty scope_id as defense-in-depth.
* refactor: add is_error to on_tool_result protocol, remove text heuristics
Add is_error keyword arg to SessionUI.on_tool_result() so tools
report errors structurally. Server and JS client no longer guess
from output text prefixes — each tool sets the flag at the source.
Bash tool: exit code >= 2 is error, exit code 1 is ambiguous (grep
no-match). History reconstruction keeps text heuristic as fallback
for pre-migration data.
Update SDKs (Python + TypeScript), test mocks, docs, and diagrams.
* fix: infinite recursion in _report_tool_result, signal exits, stale docs
* fix: add _tool_error_flags to test_load_skill ChatSession stubs
The cancel endpoint emitted a 'cancelled' SSE event before the worker
thread terminated. The frontend transitioned to "send" mode prematurely,
so the next send got rejected with "Already processing a request."
Backend:
- Providers expose SDK stream handle via cancel_ref parameter so
cancel() can close the HTTP connection and unblock iteration
- Generation counter prevents orphaned threads from mutating messages
or clearing cancel state after force cancel
- _check_cancelled() added between retry attempts in _try_stream
- Server polls (async, non-blocking) for cancelled worker to exit
- Force cancel (force:true) abandons stuck worker, keeps cancel event
set so subprocesses are killed, guards against spurious SSE events
Frontend:
- 'cancelled' shows "Cancelling..." then escalates to "Force Stop"
after 2s for a harder cancel that abandons the worker immediately
- 10s safety timeout auto-recovers if stream_end never arrives
- busy_error re-enables stop button instead of showing send
- Timeout cleanup in disconnectSSE, stream_end, and force .then()
- Layout shift prevention (min-width, white-space: nowrap)
- aria-label updates for accessibility
Tests:
- 7 new tests: stream close, error suppression, cancel_ref population,
transport error conversion, non-cancel exception propagation, retry
cancellation check
Update security.md with trusted service user_id forwarding. Update
MQ protocol diagram to include user_id field on CreateWorkstreamMessage.
Update console data flow diagram to show user_id in message and bridge
forwarding.
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
Replace linear optimization chain with UCB1 evolution tree. Each
iteration selects the most promising node to extend, preventing
irrecoverable collapse from bad edits. Also adds improvement-based
delta feedback to the optimizer and optional holdout set separation.
Inspired by "Learning to Self-Evolve" (arXiv:2603.18620).
The script detected the old version from pyproject.toml, which Renovate
had already updated. This caused OLD_DIR == NEW_DIR, so the script
downloaded files then immediately deleted them.
Fix: detect old version from the actual directory on disk. Add a guard
that errors if old == new version to prevent silent data loss.
Also: run the fixed script to vendor katex 0.16.40 (fonts + css + js).
* fix: align ConfigStore implementation with spec
- Add cluster + skills sections to admin UI settings order and labels
- Return default value in DELETE /v1/api/admin/settings response per spec
- Document 4 missing settings in docs/settings.md (trusted_proxies,
output_guard, redact_secrets, discovery_url) and correct count to 48
- Wire ConfigStore into console server replacing 4 raw
get_system_setting() calls with validated/cached config_store.get()
- Reload console ConfigStore on settings mutations via
_publish_config_change()
- Update registry URL tests for ConfigStore-based resolution
* fix: address Copilot review feedback on ConfigStore PR
- Move config_store.reload() before collector guard in
_publish_config_change() so cache refreshes even without collector
- Add DeleteSettingResponse schema and update OpenAPI spec to match
the actual delete response (status + key + default)
- Add test asserting default field in delete response
- Fix stale docstring in test helper
* feat: raise scaling limits for 1000-node clusters
Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.
Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers
Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling
Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster
New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)
Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.
* fix: add image tag to compose anchors to avoid redundant builds
All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.
* fix: address Copilot review feedback on scaling PR
- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor
* fix: add PostgreSQL env vars to cluster bridge anchor
Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.
* fix: address Copilot round 2 + PG connection exhaustion at startup
Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change
Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service
* fix readme
* fix: startup resilience for large clusters
Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.
Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.
Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).
Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service
* fix: replace silent error suppression with structured logging
Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.
memory.py (26 changes):
Every storage operation now logs on failure. Previously the entire
persistence facade had zero logging — messages, workstream state,
and structured memories could silently stop being saved.
server.py:
Usage recording failures now log at warning (was pass).
Global SSE fan-out errors log at debug (was pass).
console/server.py:
Config reload notification logs per-node failures at warning.
Settings read fallbacks log at warning with the default value used.
auth.py:
User existence check logs at warning (was pass).
Setup rollback failures log at error (was suppress).
OIDC state cleanup logs at debug (was suppress).
mcp_client.py:
DB-managed MCP server list failure logs at warning (was pass).
collector.py:
Node poll failure upgraded from debug to warning with exc_info.
Health fetch failure logs at debug with exc_info (was silent).
bridge.py:
Best-effort plan rejection logs at warning (was suppress).
Malformed SSE data logs at debug (was suppress).
session.py:
Tool output UI callback failure logs at debug (was suppress).
* fix: stagger collector poll with deterministic per-node jitter
Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.
Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.
* fix: add startup jitter to bridge heartbeat and health monitor probe
Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.
Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.
Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.
* fix: split collector httpx timeout and raise keepalive pool
Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.
* fix: narrow detect_model return type for CLI and eval callers
detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
* feat: split-pane layout for chat UI
Refactor the server UI from a single-pane global-state design to a
multi-pane architecture with per-workstream Pane instances and a binary
layout tree. Each pane has its own SSE connection, message area, input,
and state (busy, approval, streaming).
Phase 1 — Pane class with 25 prototype methods encapsulating all
per-workstream state. Phase 2 — binary split tree (leaf/split nodes)
with recursive flexbox rendering and drag-to-resize handles. Phase 3 —
keyboard shortcuts (Ctrl+\, Ctrl+Shift+\, Ctrl+Shift+W, Ctrl+Alt+Arrow)
and right-click context menu. Phase 4 — layout persistence via
localStorage.
Key design decisions:
- No duplicate workstreams across panes (split refused if no unused ws,
auto-close redundant pane on ws deletion)
- Max 6 panes to avoid exhausting browser SSE connections
- Viewport guard prevents splitting below min-width/min-height
- Only focused pane refreshes workstream list on SSE reconnect (prevents
race when multiple panes disconnect simultaneously)
- Tab click focuses existing pane showing that ws in multi-pane mode
- Pointer events on drag handles for mouse + touch support
- Full a11y: ARIA roles/labels, keyboard nav in context menu, focus
restoration, prefers-reduced-motion coverage
* fix: address PR #127 review feedback
- Add focusin handler so keyboard focus (Tab) updates focusedPaneId
- Context menu skips interactive elements (textarea, input, links,
buttons) so native copy/paste and link context menus work
- Split handles get ARIA role=separator, aria-orientation, aria-valuenow,
keyboard resizing (arrow keys, Home/End), and tabindex=0
- Enforce MAX_PANES limit in deserializeLayout to prevent corrupted
localStorage from creating too many panes/SSE connections
- Update architecture.md to document split-pane layout
- Null-safe extraction for description, license, and compatibility in
skill_parser.py — YAML bare keys (e.g. `description:`) no longer
produce the literal string "None"
- Log warning on skill catalog storage failure instead of silent swallow
- Use `enabled == 1` in list_skills_by_activation for consistency with
other prompt_templates queries in both storage backends
- Add parser tests for YAML null description, license, and compatibility
- Update governance.md: document runtime config editing on installed
skills, two-column modal layout, SPDX license dropdown, origin badge
* feat: load_skill built-in tool — model-driven skill discovery and activation
Two-action tool: 'search' finds skills by multi-word query with substring
matching on name/description/tags/category (auto-approved, read-only);
'load' activates a skill by name via set_skill() (requires approval).
Guards: filters disabled skills from search + load; short-circuits when
skill is already active; approval_label includes skill name for granular
tool policies (load_skill__<name>); main session only (excluded from
sub-agents). Logs storage errors in search path.
25 tests covering registration, preparer validation, executor logic,
disabled/already-active edge cases, multi-word queries, approval labels.
* refactor: use BM25 relevance ranking for load_skill search
Replace substring matching with BM25Index from turnstone/core/bm25.py,
matching the pattern used by memory relevance and tool search. Handles
multi-word queries, term frequency, and document length normalization.
* fix: address copilot review — BM25 tags parsing, primary_key, test cleanup
- Parse JSON tags into space-separated text before BM25 indexing so
individual tag terms match queries (was passing raw '["foo","bar"]')
- Add primary_key: "name" to load_skill.json for PRIMARY_KEY_MAP
- Remove dead resolve_workstream patch from test helper
- Update diagram: "substring match" → "BM25 ranking"
* feat: skill discovery — search and install skills from external sources
Add discovery UI and API for finding and installing skills from
skills.sh registries and GitHub repositories with one-click install,
SKILL.md frontmatter parsing, and security scan integration.
Core modules:
- skill_parser.py: ParsedSkill dataclass, parse_skill_md() with YAML
frontmatter support (Anthropic + Hermes tag formats), name validation
- skill_sources.py: SkillsShClient (async search + resolve),
fetch_skill_from_github (SKILL.md + bundled resource fetching with
256KB cap, text extension filter, GitHub API tree traversal)
API:
- GET /v1/api/admin/skills/discover — search with installed annotation
and scan_status for installed skills
- POST /v1/api/admin/skills/install — fetch, parse, duplicate check,
create with origin="source" readonly=true, store resources, audit
Also fixes pre-existing bug where _skill_to_response omitted scan_status,
scan_report, scan_version fields — scan tier badges in the installed
skills table were silently empty despite data existing in storage.
Admin UI: pill toggle (Installed/Discover), discovery cards with scan
tier badges, GitHub import modal with proper focus trap/Escape/backdrop,
scoped selectors preventing MCP↔Skills cross-tab state corruption.
SDK: discover_skills() + install_skill() on Python (async+sync) and
TypeScript console clients.
48 new tests across 3 test files. All 2632 tests pass.
* fix: address copilot review — 404 vs 502, O(n) lookups, branch fallback
- SkillNotFoundError subclass: install returns 404 when SKILL.md is
missing, 502 only for connectivity/upstream errors
- get_skill_by_source_url() + list_installed_skill_urls(): indexed
storage lookups replace O(n) full-table scans with content blobs
- Default branch fallback: tries main then master when URL doesn't
specify a branch
- Path normalization: strip trailing slash once, remove redundant
candidate
- SDK install_skill() returns typed SkillInfo with response_model
- Tree size guard: skip resource tree if response >2MB
* feat: output guard data pipeline — persist assessments, SSE events, admin UI
Complete the output guard pipeline: persist assessments for v2 calibration,
surface warnings in every UI layer, and add scan badges to admin skills tab.
Storage: migration 022 adds output_assessments table (flags, risk_level,
annotations, output_length, redacted — raw output never stored) and
scan_version column on prompt_templates. Three new protocol methods with
SQLite + PostgreSQL implementations.
Server/CLI: on_output_warning now persists assessments fire-and-forget.
CLI shows flags, annotations, and redaction notice. Session emits on_info
warning when high/critical scan_status skill is loaded.
MQ: OutputWarningEvent dataclass + bridge SSE forwarding.
Web UI: output_warning SSE handler with inline warning rendering
(role="alert" for accessibility), semantic risk colors.
Console admin: scan badges on skills list (dedicated scope-scan-* CSS with
green/yellow/red risk vocabulary), scan report breakdown in edit modal with
4-axis scores, POST /admin/skills/{id}/rescan endpoint, GET
/admin/output-assessments endpoint with date-filtered pagination.
Security fixes: ReDoS in connection string regex ([^@\s]+ → [^:@\s]+),
negative limit bypass in all admin endpoints (max(1, ...)), to_dict()
excludes sanitized output by default.
False-positive fixes: credentials pattern anchored to path context,
env secret key check restricted to key portion only.
* fix: address PR #110 review — list redaction, test fixture, OpenAPI snapshot
Per-part redaction: evaluate each text part independently in structured
output instead of joining all parts and replacing only the first one.
Fix test annotations default from "{}" to "[]" matching schema.
Regenerate TypeScript OpenAPI snapshots for new admin endpoints.
Add turnstone/core/skill_scanner.py — a production content scanner
that evaluates skill risk across four axes:
1. Content risk: command execution, external downloads, credential
handling, data exfiltration, eval/exec, sudo, browser automation
2. Supply chain risk: pipe-to-shell, transitive installs, obfuscation,
download-exec chains, executable URLs from untrusted domains
3. Vulnerability risk: prompt injection (E004), insecure credential
handling (W007), third-party content exposure (W011)
4. Declared capability risk: parsed from allowed_tools field —
Bash(*) is high, Bash(git:*) is low, read-only tools are safe
Composite score with equal 25% weights per axis. Floor rule: any
single axis at critical forces composite to at least medium tier.
Wired into both SQLite and PostgreSQL storage backends:
- scan_skill() runs at create_prompt_template time
- Re-scan triggers on update when content or allowed_tools change
- Results populate the existing scan_status and scan_report columns
- Silent failure on scanner errors (never blocks skill creation)
Scanner helper factored into _utils.py (shared across backends).
23 unit tests covering tier classification, capability scoring,
negation filtering, floor rule, serialization, and trusted domains.
* feat(judge): enrich heuristic rules from 23 to 36
Add 13 new pattern-based rules to the intent validation heuristic,
calibrated from analysis of 25K public agent skill security audits
across three independent auditors.
New critical: download-then-execute chains.
New high: browser+data export, transitive installs from untrusted
sources, control plane mutations (crontab, systemctl).
New medium: content ingestion pipelines (curl|python3), interpreter
execution (python3 script.py), cloud CLI mutations (az/gcloud/aws/
kubectl/terraform create/delete/destroy).
New low: tool_search, read_resource, web_search.
Fixes: crontab -l no longer false-positives, systemctl stop/disable
now flagged, az/gcloud subcommand patterns work correctly.
* fix(judge): address PR #107 review feedback
- content-ingestion: narrow second pattern to specific interpreters/
processors (python3, node, ruby, perl, php, jq) instead of any word.
Prevents false positives on read-only downstream (wget -O - | head).
- cloud-infra-mutation: split kubectl into its own pattern with specific
verbs (apply, create, delete, scale, rollout, drain, cordon) to avoid
false positive on resource types (kubectl get deploy).
- cloud-infra-mutation: split terraform/pulumi to specific verbs only
(apply, destroy, import) — terraform plan no longer matches.
- control-plane-mutation: exclude -h and -V flags from crontab pattern
alongside existing -l exclusion.
- Add 35 heuristic rule tests covering all 13 new rules with positive
matches and negative (false-positive prevention) cases.
* feat: unified skills system — merge prompt templates + workstream templates
Evolves prompt_templates into a first-class skills entity and merges
workstream templates into the same model, collapsing two concepts into
one.
Migration 021: 21 new columns on prompt_templates (skills metadata,
security scan fields, session config from WS templates), skill_resources
table for bundled files, skill_versions table for auto-snapshot version
history. Data migration converts existing WS templates into skills with
name collision handling, migrates version history, renames workstreams
and scheduled_tasks columns, cleans orphaned permissions, drops old
tables.
Key changes:
- All public interfaces renamed: templates → skills (API, CLI, SDK, UI)
- Session config (model, temperature, token_budget, auto_approve, etc.)
now lives on the skill and is applied at workstream creation
- /skill slash command, set_skill() API, --skill CLI flag
- BM25 skill search via SkillSearchManager for activation="search" skills
- Admin UI: Skills tab with collapsible Session Config section,
description subtitles, activation/origin/MCP badges, pagination
- Shared validation helper (_parse_skill_session_config) for DRY CRUD
- Version history with auto-snapshot on every edit + API endpoint
- Cascade delete (resources + versions) on skill removal
- Security: range validation, activation allowlist, fail-closed enabled
check, duplicate name 409, readonly guard, JSON validation
- 77 new tests across storage, runtime, search, API integration, and
migration behavior verification (2521 total)
* fix: address Copilot review + rename admin.templates → admin.skills
- Skip skill lookup when resume_ws is set (avoids spurious 400)
- Fix _applied_skill_version mismatch (1 in both workstreams table and session)
- Remove stale template field from MQ protocol diagram
- Rename admin.templates permission to admin.skills everywhere (runtime,
frontend, tests, docs) with migration step for persisted role data
- Fix stale /api/templates references in docs and diagrams
- Update docstrings/comments for skills terminology
* fix: address Copilot round 2 — skill version lineage + stale doc refs
- Compute actual skill version from skill_versions count (not hardcoded 1)
- Use same version in both workstreams table and session metadata
- Fix response payload example: "templates" → "skills" key
- Fix "Each template summary" → "Each skill summary"
Review fixes:
- Rename query param from `q` to `search` across endpoint, frontend,
SDKs, OpenAPI spec, docs, and tests to match upstream registry API
- Validate variables/env/headers are dicts in install endpoint (400 on
malformed input instead of 500)
- Block javascript: and unsafe URL schemes on repo and website links
rendered from registry data (XSS prevention)
- Add roving tabindex to Servers/Registry pill toggle for correct
keyboard focus behavior
- Add noreferrer to website link in detail modal
Sync-pending indicator:
- "Sync to Nodes" button pulses yellow after create/edit/delete/import
to alert admin that nodes have unseen changes
- Clears after successful sync
- Reduced-motion safe
Backend: standalone MCPRegistryClient (httpx async) queries the official MCP
Registry API (registry.modelcontextprotocol.io, v0.1). Two new console admin
endpoints: GET /v1/api/admin/mcp-registry/search (proxy with installed-status
annotation, dedup, uninstallable server filtering) and POST
/v1/api/admin/mcp-registry/install (auto-reloads all cluster nodes). Migration
019 adds registry_name/version/meta columns to mcp_servers with partial unique
index. Configurable registry URL via mcp.registry_url setting for
enterprise/private registries. resolve_install_config() handles both remote
(streamable-http) and package (npm→npx, pypi→uvx) installs. Pydantic models,
OpenAPI spec, Python + TypeScript SDK methods.
Frontend: unified MCP admin tab with Servers/Registry pill toggle (ARIA
tablist). Servers view: tri-state source badges (CONFIG/MANUAL/REGISTRY).
Registry view: search bar with type filter (remote/npm/pypi), auto-browse on
tab switch, result cards with source-type badges and repo links, one-click
install for zero-config remotes, install modal with dynamic form for servers
needing env vars/headers/URL variables. Package install warning banner.
Post-install status polling with connection/error feedback toasts. Trust
notice banner linking to the official registry.
Safety: 30s connect timeout on streamablehttp_client and session.initialize()
prevents hung connections from blocking the MCP event loop indefinitely.
Required-only headers in install config prevents empty auth headers from
causing silent 401s.
71 new tests (registry client, API endpoints, storage columns). Docs:
dedicated docs/mcp-registry.md, updated api-reference, architecture, console,
sdk, settings docs. Updated MCP architecture diagram.
* feat: per-tool "Always" approve instead of blanket auto-approve
Interactive "Always" button now adds specific tool names to
auto_approve_tools instead of setting blanket auto_approve=True.
Only the tool types in the current batch are auto-approved going
forward — new tool types still prompt for approval.
Server uses approval_label (with func_name fallback) matching the
existing approve_tools() lookup. CLI and bridge use func_name.
Budget override excluded from all paths.
UI: dashed border on Always button signals persistent action,
dynamic tooltip/badge show tool names, aria-label for screen
readers, focus-visible outline fix, overflow-wrap on badge.
Bridge: seeds with DEFAULT_SAFE_TOOLS on first "always" to avoid
losing existing safe-tool auto-approvals.
16 new tests (10 unit + 6 TestClient integration). Updated tool
pipeline diagram and docs.
* fix: address copilot review — filter errored items, hide Always on budget-only
- Server/bridge/JS: add `not it.get("error")` filter so policy-denied
items aren't added to auto_approve_tools
- Hide Always button when no eligible tools (budget-override-only batch)
- Docs: clarify CLI/bridge use func_name (coarser MCP granularity)
Eliminate dual accumulation by piggybacking assistant response text on
the server's ws_state:idle SSE event. The bridge no longer maintains
its own _ws_content_buffer — it reads content directly from the idle
event and passes it through to TurnCompleteEvent unchanged.
Server-side: WebUI accumulates tokens in on_content_token(), joins and
includes in the idle broadcast, then resets (with 256 KB cap).
Downstream consumers (Discord bidi DM forwarding, catch-up) are
unaffected — TurnCompleteEvent.content is still populated.
* fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var
OIDC redirect_uri was derived from the request Host header, which is
unreliable behind reverse proxies. Add TURNSTONE_OIDC_REDIRECT_BASE
(env var / config.toml) to pin the externally-reachable origin.
Extract _build_oidc_redirect_uri() helper to deduplicate the authorize
and callback handlers. Validate redirect_base at load time (must be
scheme://host[:port], rejects paths/query strings/invalid schemes).
* fix(oidc): reject redirect_base with missing hostname
Addresses Copilot review: values like `https://` or `https://:443`
passed validation but would produce invalid redirect URIs.
* fix(oidc): reject redirect_base with userinfo or invalid port
Addresses Copilot round 2: urlparse silently accepts user:pass@host
and non-numeric ports. Now explicitly rejects both.
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal
Close test coverage gaps for prompt templates:
- Resume with deleted template: verifies graceful degradation (template_content=None, warning logged)
- Threading safety: concurrent set_template/init_system_messages with no race conditions
- Factory passthrough: template kwarg propagation through WorkstreamManager.create()
Add read-only template listing endpoints (read scope, no content exposed):
- GET /v1/api/templates — prompt template summaries (name, category, is_default, origin)
- GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model)
- Available on both server and console; Python + TypeScript SDK methods added
- Console creation modal switched from admin endpoint to read-scope endpoint
Eliminate double-load inefficiency in workstream creation:
- Template validation moved before mgr.create() (no create-then-rollback on invalid template)
- template kwarg plumbed through WorkstreamManager.create() and session factory
- _SessionFactory Protocol added for proper mypy typing
Add workstream creation modal to server web UI:
- Name, model, template dropdown, ws_template/profile dropdown
- Instrument panel aesthetic: gradient top border, blur backdrop, amber accent
- Focus trap, Escape/Enter keyboard handling, loading state, error display
- WCAG AA contrast compliance, reduced-motion support
* fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots
Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates()
to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint.
Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types.
Regenerate openapi-server.json and openapi-console.json snapshots.
Addresses Copilot review feedback on PR #67.
* fix: skip template pre-validation when resuming a workstream
When resume_ws is set, the request's template field is irrelevant —
resume() restores the template from workstream_config. Pre-validating
a stale template name would incorrectly return 400 before the resume
even runs.
Addresses Copilot review feedback on PR #67.
* feat: rich markdown renderer with LaTeX support for server web UI
Extract markdown rendering from app.js into dedicated renderer.js with
full GFM support: tables (alignment, hover, striping), nested lists,
task list checkboxes, nested blockquotes, images (click-to-load for
privacy), and inline/display LaTeX math via self-hosted KaTeX 0.16.38.
Security: escape image/link URLs to prevent attribute injection, block
javascript: scheme in links, add rel="noopener noreferrer", images
require explicit click to load (no automatic external requests).
Accessibility: scope="col" on table headers, tabindex on scrollable
table containers, aria-labels on task checkboxes and image placeholders,
KaTeX error color override for WCAG AA contrast, reduced-motion support.
* fix: address code review — XSS hardening and list type splitting
- Escape all text through escapeHtml() at start of inlineMarkdown()
so only renderer-generated tags appear in innerHTML (prevents raw
HTML/script injection from LLM output)
- Replace inline onclick handler on image placeholders with data-*
attributes and delegated DOM event listeners (prevents entity
decoding XSS in event handler attributes)
- Split list blocks into separate <ul>/<ol> when marker type changes
at the same indent level (mixed ordered/unordered sequences)
renovate[bot]